How to read binary strings in Python

Question

I am wondering how is binary encoding for a string is given in Python.

For example,

>>> b'\x25'
b'%'

or

>>>b'\xe2\x82\xac'.decode()
'€'

but

>>> b'\xy9'
File "<stdin>", line 1
SyntaxError: (value error) invalid \x escape at position 0

Please, could you explain what \xe2 stand for and how this binary encoding works.

• zondo: That's hexadecimal which uses 0-9 and a-f. It complains because y is not valid.

Answer 1

\x is used to introduce a hexadecimal value, and must be followed by exactly two hexadecimal digits. For example, \xe2 represents the byte (in decimal) 226 (= 14 * 16 + 2).

In the first case, the two strings b'\x25' and b'%' are identical; Python displays values using ASCII equivalents where possible.

• Sean McCarthy: Why does it have to be followed by exactly two hexadecimal digits? Why not just one?
• chepner: Because then \x25 would be ambiguous: is it the single character % or is it a two-character sequence consisting of the ASCII control character START TRANSMISSION followed by the character 5?
• Sean McCarthy: Right, thanks. I guess it's also because each hexadecimal digit is 4 bits, so two hex digits is 8 bits or one byte
• chepner: Yes. In a bytes value, \x is used to specify a single byte by its hexadecimal value. In a str value, you can use \x (primarily for historical reasons) to specify a Unicode code point using a single value between 0 and 255, \u to specify one using a 16-bit value, or \U using a 32-bit value.

Answer 2

I assume that you use a Python 3 version. In Python 3 the default encoding is UTF-8, so b'\xe2\x82\xac'.decode() is in fact b'\xe2\x82\xac'.decode('UTF-8).

It gives the character '€' which is U+20AC in unicode and the UTF8 encoding of U+20AC is indeed `b'\xe2\x82\xac' in 3 bytes.

So all ascii characters (code below 128) are encoded into one single byte with same value as the unicode code. Non ascii characters corresponding to one single 16 bit unicode value are utf8 encoded into 2 or 3 bytes (this is known as the Basic Multilingual Plane).