homepage

Since the changeset 45079ad1e260 (issue #4388), command line arguments are decoded from UTF-8 instead of the locale encoding. Functions of Python/fileutils.c are still using the locale encoding.
It does not work: see issue #16218. On Mac OS X, in the command line "python script.py", the filename "script.py" is decoded from UTF-8 (by _Py_DecodeUTF8_surrogateescape) but then it is passed to _Py_fopen() which encodes the filename to the locale encoding (ex: ISO-8859-1 if $LANG, $LC_CTYPE and $LC_ALL environment variables are not set). The result is mojibake and Python fails to open the script.
Attached patch modifies function of Python/fileutils.c to use UTF-8 to encode and decode filenames, instead of the locale encoding on Mac OS X.
I don't know yet if Module/getpath.c should also decode paths from UTF-8 instead of the locale encoding on Mac OS X. We may expose _Py_decode_filename().

macosx-2.patch patches _Py_wchar2char() and _Py_char2wchar() functions to
use UTF-8/surrogateescape for any function using the locale encoding, not
only file related functions of fileutils.h. The patch does also simplify
the code, no more specific #ifdef __APPLE__ in python.c:
-#ifdef __APPLE__
- argv_copy[i] = _Py_DecodeUTF8_surrogateescape(argv[i],
strlen(argv[i]));
-#else
 argv_copy[i] = _Py_char2wchar(argv[i], NULL);
-#endif
2012年11月7日 Andrew Svetlov <report@bugs.python.org>
>
> Changes by Andrew Svetlov <andrew.svetlov@gmail.com>:
>
>
> ----------
> nosy: +asvetlov
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue16416>
> _______________________________________
>

New changeset 48fbdaf3a849 by Victor Stinner in branch 'default':
Issue #16416: OS data are now always encoded/decoded to/from
http://hg.python.org/cpython/rev/48fbdaf3a849 

New changeset f3e512b5ffb3 by Victor Stinner in branch 'default':
Issue #16416: Fix error handling in _Py_wchar2char() _Py_char2wchar() functions
http://hg.python.org/cpython/rev/f3e512b5ffb3 

New changeset 1b97cc71a05e by Victor Stinner in branch 'default':
Issue #16416: Fix Misc/NEWS entry, mention Mac OS X
http://hg.python.org/cpython/rev/1b97cc71a05e 

@Serhiy: Thanks for your review, I missed it before my first commit.

Victor, are you going to backport this to 3.3?

> Victor, are you going to backport this to 3.3?
I'm waiting for the result of the buildbots, and maybe also the fix for the issue #16455 (which has an impact on tests on undecodable bytes).

Victor, could you please backport to 3.3?

Ping.

New changeset c838c9b117f1 by Victor Stinner in branch '3.2':
Issue #16416: On Mac OS X, operating system data are now always
http://hg.python.org/cpython/rev/c838c9b117f1
New changeset 26c4748351cb by Victor Stinner in branch '3.3':
(Merge 3.2) Issue #16416: On Mac OS X, operating system data are now always
http://hg.python.org/cpython/rev/26c4748351cb 

New changeset af6fd3ca6de9 by Victor Stinner in branch '3.2':
Issue #16416: Fix compilation error
http://hg.python.org/cpython/rev/af6fd3ca6de9 

The issue should now be fixed in Python 3.2, 3.3 and 3.4.