homepage

Exemple of the problem: exec('#header\n# encoding:
ISO-8859-1\nprint("h\xe9 h\xe9")\n')
exec(unicode) calls source_as_string() which converts unicode to bytes
using _PyUnicode_AsDefaultEncodedString() (UTF-8 charset). Then
PyRun_StringFlags() is called with the UTF-8 byte string with
PyCF_SOURCE_IS_UTF8 flag. But in the parser, get_coding_spec() recognize
the "#coding:" header and convert bytes to unicode using the specified
charset (which may be different than UTF-8).
The problem is in the function PyAST_FromNode(): the flag in not used in
the tokenizer but only in the AST parser. I also see:
 if (flags && flags->cf_flags & PyCF_SOURCE_IS_UTF8) {
 c.c_encoding = "utf-8";
 if (TYPE(n) == encoding_decl) {
#if 0
 ast_error(n, "encoding declaration in Unicode string");
 goto error;
#endif
 n = CHILD(n, 0);
 }
 } else if (TYPE(n) == encoding_decl) {
 c.c_encoding = STR(n);
 n = CHILD(n, 0);
 } else {
	/* PEP 3120 */
 c.c_encoding = "utf-8";
 }
The ast_error() may be uncommented.