Python: Unicode problems

Question

I am getting an error at this line

logger.info(u"Data: {}".format(data))

I'm getting this error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 4: ordinal not in range(128)

Before that line, I tried adding data = data.decode('utf8') and I still get the same error. I tried data = data.encode('utf8') and it says UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4: ordinal not in range(128)

How do I fix this? I don't know if I should encode or decode but neither works.

Answer 1

Use a string literal:

if isinstance(data, unicode):
 data = data.encode('utf8')
logger.info("Data: {}".format(data))

The logging module needs you to pass in string values as these values are passed on unaltered to formatters and the handlers. Writing log messages to a file means that unicode values are encoded with the default (ASCII) codec otherwise. But you also need to pass in a bytestring value when formatting.

Passing in a str value into a unicode .format() template leads to decoding errors, passing in a unicode value into a str .format() template leads to encoding errors, and passing a formatted unicode value to logger.info() leads to encoding errors too.

Better not mix and encode explicitly beforehand.

Answer 2

You could do something such as

data.decode('utf-8').encode("ascii",errors="ignore")

This will "ignore" the unicode characters

edit: data.encode('ascii',error='ignore') may be enough but i'm not in a position to test this currently.