Convert Unicode to UTF-8 Python

Question

I am working with a database that has throughout it scattered characters like this: â€TM. I need to take this from the database, convert it to UTF-8, and then import it into a different database, using python. When printed to the Windows Command Prompt, these characters look like this: \xe2\u20ac\u2122. I have tried various combinations of .decode(), .encode(), and unicode() to convert the data, but Im really stuck.

• Yann Ramin: Well, what is the database encoding? UTF-16?
• Jonathan Lerner: The database encoding is UTF_8_bin
• Cameron: Did you try blah.decode('utf-8-sig')? What happens when you do?
• Jim Garrison: The Windows command-prompt cannot display UTF-8 as it uses Win-1252 (a variant of ISO-8859-1). What you see is the result of trying to display UTF-8 in a non-UTF-8 environment. Why is the display of the data in a command-prompt important?
• Edgar Velasquez Lim: To display properly at the prompt you might want to try putty and connect to localhost, putty supports UTF-8 under the Windows->Translation option. Although I've personally had little luck with this feature.

Answer 1

Always decode on input, and encode on output. (There ought to be handy mnemonic for this: perhaps "take your code [coat] off when you come indoors".)

Decode on input: You say that the database encoding is "UTF_8_bin". Are you using MySQL-Python? If so, then you can set the use_unicode option when you connect to the database. Then all strings are fetched from the database in Unicode, so you don't have worry about decoding them.

Encode on output: You can find out the current character encoding (or "code page" as they call it in Windows) with the chcp command. Let's suppose it's code page 1252. Then you can write

print text.encode('windows-1252')

to produce something that you can read from the Windows command line.

If you're writing the strings back to another MySQL database using MySQL-Python, you shouldn't need to do anything special: MySQL-Python claims that "you can always write Unicode strings" (regardless of whether you specified use_unicode when you opened the connection).

• Jonathan Lerner: Where exactly would I specify the use_unicode option? I am running a standalone script on a django app; Do I add this to the settings.py file, or somewhere else?
• Gareth Rees: It's an option to MySQLdb.connect. I don't know anything about Django, but the documentation suggests that you could put it under DATABASES.default.OPTIONS in your configuration.
• jpmc26: DIEO (read as "die-oh")? Despise incompetent, evil offenders? (I'm not very creative. lol)