homepage

I've hit this issue while playing with tokenize for the pep8.py module.
The tokenize detect_encoding() should report SyntaxError when the encoding is improperly declared.
However it raises a LookupError in some cases.
$ ./python -m tokenize Lib/test/bad_coding2.py 
unexpected error: unknown encoding: utf8-sig
Traceback (most recent call last):
 File "./Lib/runpy.py", line 162, in _run_module_as_main
 "__main__", fname, loader, pkg_name)
 File "./Lib/runpy.py", line 75, in _run_code
 exec(code, run_globals)
 File "./Lib/tokenize.py", line 686, in <module>
 main()
 File "./Lib/tokenize.py", line 656, in main
 tokens = list(tokenize(f.readline))
 File "./Lib/tokenize.py", line 489, in _tokenize
 line = line.decode(encoding)
LookupError: unknown encoding: utf8-sig

This patch seems to fix the issue.

The patch is correct according to the PEP 263:
 If a source file uses both the UTF-8 BOM mark signature and a
 magic encoding comment, the only allowed encoding for the comment
 is 'utf-8'. Any other encoding will cause an error.
The fix should also be applied to 3.2.
(Note: Python 3.1 doesn't accept bugfixes anymore.)

It should raise a SyntaxError, if coding is 'utf8'.
I don't agree with the last patch proposed.
If the import report a SyntaxError, 'tokenize' should do the same.
$ ./python Lib/test/bad_coding2.py
 File "Lib/test/bad_coding2.py", line 1
SyntaxError: encoding problem: utf-8
and it complies strictly with PEP263.

Oops, I didn't want to attach my patch to the issue. Mine is wrong, whereas yours is the right fix :-)

New changeset 5020afc0b7c9 by Florent Xicluna in branch '3.2':
Issue #14990: tokenize: correctly fail with SyntaxError on invalid encoding declaration.
http://hg.python.org/cpython/rev/5020afc0b7c9 

Thanks. Fixed in trunk too, changeset b4322ad1fec4