homepage

Copy/paste of message79330 from the issue #4862:
--------------
>>> f = open('utf16.txt', 'w', encoding='utf-16')
>>> f.write('abc')
3
>>> f.close()
>>> f = open('utf16.txt', 'a', encoding='utf-16')
>>> f.write('def')
3
>>> f.close()
>>> open('utf16.txt', 'r', encoding='utf-16').read()
'abc\ufeffdef'
--------------

Bug is reproductible with:
 * Python 2.5 : charset utf-8-sig and utf-16 for codecs.open()
 * trunk : charset utf-8-sig, utf-16 and utf-32 for codecs.open()
 * py3k : charset utf-8-sig, utf-16 and utf-32 for open()
With utf-7 or utf-8, no BOM is written.
Note: with UTF-32, the BOM is 4 bytes long (0xff 0xfe 0x00 0x00 on 
little endian) but it's still the character (BOM) \ufeff (little 
endian).

See also issues #5008 (f.tell()) and #5016 (f.seekable()), not 
directly related to this issue.

One possible solution would be to call tell() in the TextIOWrapper
constructor, and then decoder.setstate((b"", 0)) if the current pos is >0.

@pitrou: You're right, but the state have to be changed for the 
encoder, not the decoder. I added the following code to TextIOWrapper 
constructor (for the C and the Python version of io library):
 if self._seekable and self.writable():
 position = self.buffer.tell()
 if position != 0:
 self._encoder = self._get_encoder()
 self._encoder.setstate(0)

As for the C version of the patch:
- I'm not sure why you make self->ok handling more complicated than it was
- encodefunc should not be forced to NULL, otherwise it will yield a big
decrease in write() performance

> - I'm not sure why you make self->ok handling more complicated than it was
tell() requires self->ok=1. I choosed to reset self->ok to zero on error, but 
it's maybe useless.
> - encodefunc should not be forced to NULL, otherwise it will yield a big
> decrease in write() performance
self>encodefunc value have to be changed because <encoder>.setstate() changes 
the encoding function: UTF-16 => UTF-16-LE or UTF-16-BE.
I don't know how to get the new value of self>encodefunc. I will try to write 
another patch.

Le jeudi 26 mars 2009 à 17:26 +0000, STINNER Victor a écrit :
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
> > - I'm not sure why you make self->ok handling more complicated than it was
> 
> tell() requires self->ok=1. I choosed to reset self->ok to zero on error, but 
> it's maybe useless.
You are calling tell() on the underlying binary buffer, not on the
TextIOWrapper object. So self->ok should have no impact. Am I missing
something?
> self>encodefunc value have to be changed because <encoder>.setstate() changes 
> the encoding function: UTF-16 => UTF-16-LE or UTF-16-BE.
I know, but then the logic should probably be changed and use an
additional struct member to signal that the start of file has been
skipped.

Faster patch!
 - add fast encoder for UTF-32, UTF-32-LE and UTF-32-BE (copy/paste of 
utf16 functions)
 - move utf-8 before utf-16-* because utf8 more popular than utf16 :-p
 - don't set self->encodefunc=NULL (loose all the encoder 
optimisation), but only fix self->encodefunc for two special cases: 
utf16=>utf16le or utf16be and utf32=>utf32le or utf32be
 - remove self->ok: it was may be usefull for an older version of my 
patch, but it's not more needed

Hum, it's a detail, but is it a good idea to keep the reference the int(0)? Or 
would it be better to release it at exit?

It's better now, although I think it's not good to duplicate the
encoding switch logic. It would be better to have a separate flag
indicate whether it's the start of stream or not. I'm gonna produce a
new patch, unless you beat me to it.
Also, I'm adding Amaury to the nosy list so that he tells us whether he
thinks the approach is sound.

Here is a new patch catering to more cases (seek()) in addition to just
opening in append mode.

seek() has also the problem? It's really hard to encode UTF-16/32
correctly...

Updated patch against py3k.

If no-one objects, I'm going to commit this in a couple of days.

As I said on IRC a few days ago, I think the patch is ready to go.

Ok, committed in r72635.

Shouldn't this fix be back-ported to the 2.6 branch too?

I fixed #6213 in 2.6 and 2.7, and so it's now possible to backport this fix to 2.6 => r83200