homepage

On Windows, if an OS error fails, the filename type is bytes and the filename cannot be decoded: Python raises an UnicodeDecodeError instead of an OSError. The problem is that Python decodes the filename to fill OSError.filename field. See the issue #15441 for the initial report.
There are different options to solve this issue:
 - always keep the filename parameter unchanged, so OSError.filename can be a str or a bytes string, depending on the input parameter
 - try to decode the filename from the filesystem encoding, or keep the filename unchanged: OSError.filename is only a bytes string if the filename cannot be decoded
 - don't fill OSError.filename (= None) if the filename cannot be decoded
 - use "surrogateescape", "replace" or "backslashreplace" error handler to decode the filename
This issue is specific to Windows: on other plaforms, the filename is decoded using the "surrogateescape" error handler and so decoding the filename cannot fail.
I don't know if OSError.filename is only used to display more information to the user, or if it is used to do another operation on the file (ex: os.chmod).
I like solutions keeping the filename unchanged, because it does not loose information, and the user can decide how to handle the undecodable filename.
I don't like the option trying to decode the filename or keeping it unchanged it decoding fails, because applications will work in most cases, but "crash" when someone comes with an unusual code page, a special USB key, or a filename with a non-ASCII character.
So the best option is maybe to always keep the bytes filename unchanged.
Such change cannot be done anymore in Python 3.3, it's too late to test it correctly.

In Python 2, it looks like open(arg) does pass its filename argument unchanged to OSError constructor (so it can be bytes or unicode). OSError.filename is always bytes for os.chdir() on UNIX, but OSError.filename can be bytes or unicode for os.chdir() on Windows.

+1 for keeping the file name unchanged. This solution is not very 
compatible with prior versions, but simple and least-surprise.
I prefer other platforms than Windows to use same method to build OSError.

Attached patch modifies all functions of the os module taking filenames to keep the filename unmodified in OSError.filename.
The patch changes also os.link(), os.rename() and os.replace() to use the source, not the destination, in the error message. It is maybe a mistake because these functions can also fail in the directory of the destination does not exist.

New changeset 67d69f943b7f by Victor Stinner in branch 'default':
Issue #15478: Raising an OSError doesn't decode or encode the filename anymore
http://hg.python.org/cpython/rev/67d69f943b7f 

New changeset 27a3b19ee792 by Victor Stinner in branch 'default':
Issue #15478: Fix compilation on Windows
http://hg.python.org/cpython/rev/27a3b19ee792 

The commit is incomplete, there are some remaining functions that need to be patched: here is a new (untested) patch for more Windows functions.

See also issue16074.
> The patch changes also os.link(), os.rename() and os.replace() to use the source, not the destination, in the error message. It is maybe a mistake because these functions can also fail in the directory of the destination does not exist.
Yes, in different cases it can be the source, the destination, both, unknown or none of them.

New changeset 01cc9fb52887 by Victor Stinner in branch 'default':
Issue #15478: Fix test_os on Windows (os.chown is missing)
http://hg.python.org/cpython/rev/01cc9fb52887 

New changeset ef87bd0797de by Victor Stinner in branch 'default':
Issue #15478: Fix test_os on FreeBSD
http://hg.python.org/cpython/rev/ef87bd0797de 

New changeset 13ebaa36d87d by Victor Stinner in branch 'default':
Issue #15478: Use path_error() in more posix functions, especially in Windows
http://hg.python.org/cpython/rev/13ebaa36d87d
New changeset 9f696742dbda by Victor Stinner in branch 'default':
Issue #15478: Fix again to fix test_os on Windows
http://hg.python.org/cpython/rev/9f696742dbda 

New changeset 6903f5214e99 by Victor Stinner in branch 'default':
Issue #15478: Use source filename in OSError, not destination filename
http://hg.python.org/cpython/rev/6903f5214e99 

All issues should now be fixed.

New changeset b3434c1ae503 by Victor Stinner in branch 'default':
Issue #15441, #15478: Reenable test_nonascii_abspath() on Windows
http://hg.python.org/cpython/rev/b3434c1ae503 

One of 13ebaa36d87d, 9f696742dbda or 6903f5214e99 causes test failures in test_pep277:
======================================================================
FAIL: test_failures (test.test_pep277.UnicodeFileTests)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "C:\buildbot.python.org3円.x.kloth-win64\build\lib\test\test_pep277.py", line 120, in test_failures
 self._apply_failure(os.listdir, name)
 File "C:\buildbot.python.org3円.x.kloth-win64\build\lib\test\test_pep277.py", line 105, in _apply_failure
 self.assertEqual(wildcard, '*.*')
AssertionError: '7_\u05d4\u05e9\u05e7\u05e6\u05e5\u05e1' != '*.*'
- 7_\u05d4\u05e9\u05e7\u05e6\u05e5\u05e1
+ *.*

Additionally, some of the changes cause a failure in test_subprocess:
====================================================================== 
ERROR: test_no_leaking (test.test_subprocess.ProcessTestCase) 
---------------------------------------------------------------------- 
Traceback (most recent call last): 
 File "C:\Users\stefan\pydev\cpython\lib\test\test_subprocess.py", line 823, in test_no_leaking 
 handles.append(os.open(tmpfile, os.O_WRONLY|os.O_CREAT)) 
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'c:\\users\\stefan\\appdata\\l
ocal\\temp\\tmpa41o4x\\@test_2236_tmp' 
 
----------------------------------------------------------------------

New changeset 817a90752470 by Victor Stinner in branch 'default':
Issue #15478: Oops, fix regression in os.open() on Windows
http://hg.python.org/cpython/rev/817a90752470 

New changeset 11ea4eb79e9d by Victor Stinner in branch 'default':
Issue #15478: Fix test_pep277 on Windows
http://hg.python.org/cpython/rev/11ea4eb79e9d 

New changeset ee7b713fec71 by Victor Stinner in branch 'default':
Issue #15478: os.lchflags() is not always available when os.chflags() is available
http://hg.python.org/cpython/rev/ee7b713fec71