Working with UTF-8 encoding in Python source [duplicate]

Question

This question already has answers here:

"SyntaxError: Non-ASCII character ..." or "SyntaxError: Non-UTF-8 code starting with ..." trying to use non-ASCII text in a Python script (7 answers)

Closed 10 years ago.

Consider:

$ cat bla.py 
u = unicode('d...')
s = u.encode('utf-8')
print s
$ python bla.py 
 File "bla.py", line 1
SyntaxError: Non-ASCII character '\xe2' in file bla.py on line 1, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details

How can I declare UTF-8 strings in source code?

• Lennart Regebro: "See python.org/peps/pep-0263.html for details" seems clear to me.

Answer 1

In Python 3, UTF-8 is the default source encoding (see PEP 3120), so Unicode characters can be used anywhere.

In Python 2, you can declare in the source code header:

# -*- coding: utf-8 -*-
....

This is described in PEP 0263.

Then you can use UTF-8 in strings:

# -*- coding: utf-8 -*-
u = 'idzie wąż wąską dróżką'
uu = u.decode('utf8')
s = uu.encode('cp1250')
print(s)

• Nullpoet: now it gives """UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1: ordinal not in range(128)"""
• Michał Niklas: You need not use unicode(), simply write string in UTF-8 encoding.
• Anton Strogonoff: In Python versions older than 3, you also need to prefix unicode string literals with "u": some_string = u'idzie wąż wąską dróżką'.
• Nullpoet: on a diffrent string I am getting """UnicodeEncodeError: 'charmap' codec can't encode characters in position 1845-1846: character maps to <undefined>"""... does that mean a different encoding is required?
• warvariuc: or #!/usr/bin/env python # coding: utf-8

Answer 2

Do not forget to verify if your text editor encodes properly your code in UTF-8.

Otherwise, you may have invisible characters that are not interpreted as UTF-8.

• Ricardo Cruz: Is this needed for python3? I know python3 assumes all literals within the code are unicode. But does it assume the source files are also written in utf8?
• Jonathan Hartley: @RicardoCruz Yes I believe utf-8 is the default for Python 3. See python.org/dev/peps/pep-3120
• noobninja: @ricardo-cruz With Python 3, all strings will be Unicode strings, so the original encoding of the source will have no impact at run-time. 1. PEP 3120 -- Using UTF-8 as the default source encoding 2. PEP 263 -- Defining Python Source Code Encodings
• Ricardo Cruz: @noobninja thanks for the links: PEP 3120 confirms that the source code itself is now assumed to be UTF-8, not just strings.
• show0k: Use # coding: utf8 instead of # -*- coding: utf-8 -*-which is far easier to remember.