homepage

Turns out that PyTokenizer_FindEncoding() never properly succeeds
because the tok_state used by it does not have tok->filename set, which
is an error condition in the tokenizer. This error has been masked by
the one place the function is used, imp.find_module() because a NULL
return is never checked for an error, but instead just assumes the
default source encoding suffices.

I have not bothered to check if this exists in 2.6, but I don't see why
it would be any different.

Turns out that the NULL return value can signal an error that manifests
itself as SyntaxError("encoding problem: with BOM") thanks to the lack
of tok->filename being set in Parser/tokenizer.c:fp_setreadl() which is
called by check_coding_spec() and assumes that since tok->encoding was
never set (because fp_setreadl() returned an error value) that it had
something to do with the BOM.
The only reason this was found is because my bootstrapping of importlib
into Py3K, at some point, triggers a PyErr_Occurred() which finally
notices the error.

Attached is a patch that fixes where the error occurs. By opening the
file by either file name or file descriptor, the problem goes away. Once
this patch is accepted then PyErr_Occurred() should be added to all uses
of PyTokenizer_FindEncoding().

I don't understand the whole decoding machinery in the tokenizer, but
the patch looks ok to me. (tested in debug mode under Linux and Windows)

The patch also looks pretty harmless to me. :)

pitrou, that's because Python source code can't be correctly tokenized 
when it's encoded in few odd encodings like iso-2022 or shift-jis which 
utilizes ,円 (, ) and " as second byte of two-byte character sequence.
For example, '\x81\\' is HORIZONTAL BAR in shift-jis,
exec('print "\x81\\"')
fails. because of " is ignored by second byte of '\x81\\'.

Committed in r66209.