4. How to Deal With Strings

4.2.1. Python 2.x’s Unicode Support

Python 2 comes with two different kinds of objects that can be used to represent strings, str and unicode. Instances of the latter are used to express Unicode strings, whereas instances of the str type are byte representations (the encoded string). Under the hood, Python represents Unicode strings as either 16- or 32-bit integers, depending on how the Python interpreter was compiled. Unicode strings can be converted to 8-bit strings with unicode.encode():

>>> unicode_string = u"Fu\u00dfb\u00e4lle"
>>> print unicode_string
Fußbälle
>>> type(unicode_string)
<type 'unicode'>
>>> unicode_string.encode("utf-8")
'Fu\xc3\x9fb\xc3\xa4lle'

Python’s 8-bit strings have a str.decode() method that interprets the string using the given encoding:

>>> utf8_string = unicode_string.encode("utf-8")
>>> type(utf8_string)
<type 'str'>
>>> u2 = utf8_string.decode("utf-8")
>>> unicode_string == u2
True

Unfortunately, Python 2.x allows you to mix unicode and str if the 8-bit string happened to contain only 7-bit (ASCII) bytes, but would get UnicodeDecodeError if it contained non-ASCII values:

>>> utf8_string = " sind rund"
>>> unicode_string + utf8_string
u'Fu\xdfb\xe4lle sind rund'
>>> utf8_string = " k\xc3\xb6nnten rund sein"
>>> print utf8_string
 könnten rund sein
>>> unicode_string + utf8_string
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2:
ordinal not in range(128)

4.2.2. Unicode in GTK+

GTK+ uses UTF-8 encoded strings for all text. This means that if you call a method that returns a string you will always obtain an instance of the str type. The same applies to methods that expect one or more strings as parameter, they must be UTF-8 encoded. However, for convenience PyGObject will automatically convert any unicode instance to str if supplied as argument:

>>> fromgi.repositoryimport Gtk
>>> label = Gtk.Label()
>>> unicode_string = u"Fu\u00dfb\u00e4lle"
>>> label.set_text(unicode_string)
>>> txt = label.get_text()
>>> type(txt), txt
(<type 'str'>, 'Fu\xc3\x9fb\xc3\xa4lle')
>>> txt == unicode_string
__main__:1: UnicodeWarning: Unicode equal comparison failed to convert
both arguments to Unicode - interpreting them as being unequal
False

Note the warning at the end. Although we called Gtk.Label.set_text() with a unicode instance as argument, Gtk.Label.get_text() will always return a str instance. Accordingly, txt and unicode_string are not equal.

This is especially important if you want to internationalize your program using gettext. You have to make sure that gettext will return UTF-8 encoded 8-bit strings for all languages. In general it is recommended to not use unicode objects in GTK+ applications at all and only use UTF-8 encoded str objects since GTK+ does not fully integrate with unicode objects. Otherwise, you would have to decode the return values to Unicode strings each time you call a GTK+ method:

>>> txt = label.get_text().decode("utf-8")
>>> txt == unicode_string
True

4.3.1. Python 3.x’s Unicode support

Since Python 3.0, all strings are stored as Unicode in an instance of the str type. Encoded strings on the other hand are represented as binary data in the form of instances of the bytes type. Conceptually, str refers to text, whereas bytes refers to data. Use str.encode() to go from str to bytes, and bytes.decode() to go from bytes to str.

In addition, it is no longer possible to mix Unicode strings with encoded strings, because it will result in a TypeError:

>>> text = "Fu\u00dfb\u00e4lle"
>>> data = b" sind rund"
>>> text + data
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
TypeError: Can't convert 'bytes' object to str implicitly
>>> text + data.decode("utf-8")
'Fußbälle sind rund'
>>> text.encode("utf-8") + data
b'Fu\xc3\x9fb\xc3\xa4lle sind rund'

4.3.2. Unicode in GTK+

As a consequence, things are much cleaner and consistent with Python 3.x, because PyGObject will automatically encode/decode to/from UTF-8 if you pass a string to a method or a method returns a string. Strings, or text, will always be represented as instances of str only:

>>> fromgi.repositoryimport Gtk
>>> label = Gtk.Label()
>>> text = "Fu\u00dfb\u00e4lle"
>>> label.set_text(text)
>>> txt = label.get_text()
>>> type(txt), txt
(<class 'str'>, 'Fußbälle')
>>> txt == text
True