[Python-Dev] bytes / unicode

Tue Jun 22 20:07:18 CEST 2010

On Jun 22, 2010, at 1:03 PM, Ian Bicking wrote:
> Similarly I'd expect (from experience) that a programmer using 
> Python to want to take the same approach, sticking with unencoded 
> data in nearly all situations.

Yeah. This is a real issue I have with the direction Python3 went: it 
pushes you into decoding everything to unicode early, even when you 
don't care -- all you really wanted to do is pass it from one API to 
another, with some well-defined transformations, which don't actually 
depend on it having being decoded properly. (For example, extracting 
the path from the URL and attempting to open it as a file on the 
filesystem.)
This means that Python3 programs can become *more* fragile in the face 
of random data you encounter out in the real world, rather than less 
fragile, which was the goal of the whole exercise.
The surrogateescape method is a nice workaround for this, but I can't 
help thinking that it might've been better to just treat stuff as 
possibly-invalid-but-probably-utf8 byte-strings from input, through 
processing, to output. It seems kinda too late for that, though: next 
time someone designs a language, they can try that. :)
James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100622/7f537c1e/attachment-0001.html>