no unicode for csv #99

m-rieke commented 2018年06月28日 16:25:52 +02:00 (Migrated from github.com)

There is currently no way on using unicode in csv, this was working before 0.6.0:

import overpass
query_adminlevel = "\
area['int_name'='Nederland']->.searchArea;\
(\
 relation[boundary=administrative][admin_level=4]['ISO3166-2'~'NL'](area.searchArea);\
 );\
out center;"
api = overpass.API(timeout=1000)
overpass_adminlevel = api.Get(query_adminlevel, responseformat='csv(::"id", "name:en", "name", "int_name")')
print overpass_adminlevel

With 0.6.0 there is an error:

Traceback (most recent call last):
File "test.py", line 14, in
overpass_adminlevel = api.Get(query_adminlevel, responseformat='csv(::"id", "name:en", "name", "int_name")')
File "/home/localuser/.local/lib/python2.7/site-packages/overpass/api.py", line 74, in get
for row in reader:
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe2' in position 21: ordinal not in range(128)

I think it is because the default csv package does not support utf-8

There is currently no way on using unicode in csv, this was working before 0.6.0: ``` import overpass query_adminlevel = "\ area['int_name'='Nederland']->.searchArea;\ (\ relation[boundary=administrative][admin_level=4]['ISO3166-2'~'NL'](area.searchArea);\ );\ out center;" api = overpass.API(timeout=1000) overpass_adminlevel = api.Get(query_adminlevel, responseformat='csv(::"id", "name:en", "name", "int_name")') print overpass_adminlevel ``` With 0.6.0 there is an error: > Traceback (most recent call last): > File "test.py", line 14, in <module> > overpass_adminlevel = api.Get(query_adminlevel, responseformat='csv(::"id", "name:en", "name", "int_name")') > File "/home/localuser/.local/lib/python2.7/site-packages/overpass/api.py", line 74, in get > for row in reader: > UnicodeEncodeError: 'ascii' codec can't encode character u'\xe2' in position 21: ordinal not in range(128) I think it is because the default csv package does not support utf-8

mvexel commented 2018年09月05日 16:57:54 +02:00

Hmm yes, good point. We could use some more unit testing for unicode related pitfalls existing in Python 2. I don't want to deprecate Python 2 support just yet, since many people still use it. Consider updating to Python 3 if you can (the code snippet works in Py3). I will keep this open as a bug, but unless someone wants to come up with a PR to fix I won't prioritize this.

Hmm yes, good point. We could use some more unit testing for unicode related pitfalls existing in Python 2. I don't want to deprecate Python 2 support just yet, since many people still use it. Consider updating to Python 3 if you can (the code snippet [works in Py3](https://repl.it/@mvexel/VioletPushyUtilities)). I will keep this open as a bug, but unless someone wants to come up with a PR to fix I won't prioritize this.

mvexel commented 2018年09月05日 16:59:44 +02:00

(The csv module documentation has an example using a generator function to make this work (however this is probably lossy).

(The `csv` module documentation has [an example](https://docs.python.org/2/library/csv.html#csv-examples) using a generator function to make this work (however this is probably lossy).

mvexel commented 2026年01月05日 21:03:59 +01:00

Can you confirm this still reproduces on current versions? Please include the minimal query and CSV output that shows the unicode issue.

Can you confirm this still reproduces on current versions? Please include the minimal query and CSV output that shows the unicode issue.