homepage

In previous versions of python, when there was a
unicode decode error, StreamReader.readline() would
advance to the next line. In the current version(2.4.2
and trunk), it doesn't. Testing under Linux AMD64
(Ubuntu 5.10)
Attaching an example script. In python2.3 it prints:
hi~
hi
error: 'utf8' codec can't decode byte 0x80 in position
2: unexpected code byte
error: 'utf8' codec can't decode byte 0x81 in position
2: unexpected code byte
all done
In python2.4 and trunk it prints:
hi~
hi
error: 'utf8' codec can't decode byte 0x80 in position
0: unexpected code byte
error: 'utf8' codec can't decode byte 0x80 in position
0: unexpected code byte
error: 'utf8' codec can't decode byte 0x80 in position
0: unexpected code byte
[repeats forever]
Maybe this isn't actually supposed to work (the docs
don't mention what should happen with strict error
checking..), but it would be nice, given the alternatives:
1. use errors='replace' and then search the result for
the replacement character. (ick)
2. define a custom error handler similar to ignore or
replace, that also sets some flag. (slightly less ick,
but more work.)

Logged In: YES 
user_id=1188172
I don't know what should be correct. Walter?

Logged In: YES 
user_id=89016
IMHO the current behaviour is more consistent. To read the
broken utf-8 stream from the test script the appropriate
error handler should be used. What is the desired outcome?
If only the broken byte sequence should be skipped
errors="replace" is appropriate. To skip a complete line
that contains a broken byte sequence do something like in
the attached skipbadlines.py. The StreamReader can't know
which behaviour is wanted.

Logged In: YES 
user_id=1188172
Closing as Won't Fix, then.