homepage

open() uses the locale encoding in Python 3 when opening text file if the encoding argument is not specified (implicit). Some functions use locale encoding, but it's not the right encoding. I see at least three cases where the encoding should be changed:
 - UTF-8 should be used instead for portability: it's a bug in the module
 - ASCII must be used instead: the module doesn't support non-ASCII characters (old file formats, old network protocols, some fields of a document, etc.)
 - ASCII can be used instead: it's just a micro-optimization, the ASCII encoding is a little bit faster
To detect the usage of the implicit locale encoding, some functions can be monkeypatched:
 - builtins.open, io.open, _pyio.open
 - io.TextIOWrapper, _pyio.TextIOWrapper
 - more functions using directly or indirectly open/TextIOWrapper may be patched to emit the warning earlier
Attached open_hook.patch implements these hooks (hacks?) in the site module: it emits a ResourceWarning. Use python -Werror to raise an error if the locale encoding is used implicitly. If you really want to use the locale encoding, use encoding='locale' to make quiet the warning.
Quite all functions in Python uses the implicit locale encoding. For example, Python doesn't start with the patch and -Werror. If you use -Werror, you have to patch *all* calls to open()/TextIOWrapper to be able to locate real bugs, or the program will stop before hitting the real problems. Each time you have to check what is the real expected encoding, it takes a lot of time.
I started this huge project. I'm using ASCII most of the time (especially in Python tests), I don't know if it's correct. It will require a second step to ensure that the function really don't use/support non-ASCII characters.
I will use this issue for my commits, attach patches, and more generally discuss this topic.

New changeset bd73edea78dc by Victor Stinner in branch '3.2':
Issue #12451: distutils now opens the setup script in binary mode to read the
http://hg.python.org/cpython/rev/bd73edea78dc
New changeset 8a7fd54cba01 by Victor Stinner in branch 'default':
(merge 3.2) Issue #12451: distutils now opens the setup script in binary mode
http://hg.python.org/cpython/rev/8a7fd54cba01 

New changeset 1942f7c8f51c by Victor Stinner in branch '3.2':
Issue #12451: pydoc.synopsis() now reads the encoding cookie if available, to
http://hg.python.org/cpython/rev/1942f7c8f51c
New changeset 3e627877b5a9 by Victor Stinner in branch 'default':
(merge 3.2) Issue #12451: pydoc.synopsis() now reads the encoding cookie if
http://hg.python.org/cpython/rev/3e627877b5a9 

New changeset bafc5c7d24b2 by Victor Stinner in branch '3.2':
Issue #12451: doctest.debug_script() doesn't create a temporary file anymore to
http://hg.python.org/cpython/rev/bafc5c7d24b2
New changeset 77c589b27e90 by Victor Stinner in branch 'default':
(merge 3.2) Issue #12451: doctest.debug_script() doesn't create a temporary
http://hg.python.org/cpython/rev/77c589b27e90 

See also issue #9561 for distutils: I just attached a new patch for PKG-INFO / .egg-info files.

New changeset 45e3dafb3dbe by Victor Stinner in branch '3.2':
Issue #12451: The XInclude default loader of xml.etree now decodes files from
http://hg.python.org/cpython/rev/45e3dafb3dbe
New changeset e8eea84a90dc by Victor Stinner in branch 'default':
(merge 3.2) Issue #12451: The XInclude default loader of xml.etree now decodes
http://hg.python.org/cpython/rev/e8eea84a90dc 

New changeset 68bc1a29ba5a by Victor Stinner in branch '3.2':
Issue #12451: Open files in binary mode in some tests when the text file is not
http://hg.python.org/cpython/rev/68bc1a29ba5a
New changeset 3969b6377f52 by Victor Stinner in branch 'default':
(merge 3.2) Issue #12451: Open files in binary mode in some tests when the text
http://hg.python.org/cpython/rev/3969b6377f52 

New changeset c4388478f9b2 by Victor Stinner in branch 'default':
Issue #12451: Open the test file in binary mode in test_bz2, the text file is
http://hg.python.org/cpython/rev/c4388478f9b2 

See also issue #12454 for the mailbox module (.mh_sequences).

New changeset 0c49260e85a0 by Victor Stinner in branch 'default':
Issue #12451: Add support.create_empty_file()
http://hg.python.org/cpython/rev/0c49260e85a0 

New changeset 81424281ee59 by Victor Stinner in branch '3.2':
Issue #12451: xml.dom.pulldom: parse() now opens files in binary mode instead
http://hg.python.org/cpython/rev/81424281ee59
New changeset c039c6b58907 by Victor Stinner in branch 'default':
(merge 3.2) Issue #12451: xml.dom.pulldom: parse() now opens files in binary
http://hg.python.org/cpython/rev/c039c6b58907 

New changeset cd1759711357 by Victor Stinner in branch '3.2':
Issue #12451: runpy: run_path() now opens the Python script in binary mode,
http://hg.python.org/cpython/rev/cd1759711357
New changeset e240af1f0ae1 by Victor Stinner in branch 'default':
(merge 3.2) Issue #12451: runpy: run_path() now opens the Python script in
http://hg.python.org/cpython/rev/e240af1f0ae1 

New changeset a1b4f1716b73 by Victor Stinner in branch '3.2':
Issue #12451: pydoc: importfile() now opens the Python script in binary mode,
http://hg.python.org/cpython/rev/a1b4f1716b73
New changeset 5ca136dccbf7 by Victor Stinner in branch 'default':
(merge 3.2) Issue #12451: pydoc: importfile() now opens the Python script in
http://hg.python.org/cpython/rev/5ca136dccbf7 

New changeset 8b62f5d722f4 by Victor Stinner in branch '3.2':
Issue #12451: pydoc: html_getfile() now uses tokenize.open() to support Python
http://hg.python.org/cpython/rev/8b62f5d722f4
New changeset 2fbfb7ea362f by Victor Stinner in branch 'default':
(merge 3.2) Issue #12451: pydoc: html_getfile() now uses tokenize.open() to
http://hg.python.org/cpython/rev/2fbfb7ea362f 

Ok, it should be enough :-)