homepage

Bytes paths are deprecated on Windows and their support was not added in os.scandir() for purpose. But this makes harder converting filesystem walking functions to use os.scandir(). We have to add a special case for bytes paths on Windows in every such function (os.walk(), glob.iglob()). This is cumbersome inefficient and errorprone code that slows down common case due to additional checks for special case. It would be better to add a support of bytes paths directly in os.scandir().

In the light of Steve Dower's work to "un-deprecate" bytes paths, I agree this should be added.

This is implemented in Steve's latest patch in issue 27781. For example:
 >>> sys.platform
 'win32'
 >>> os.mkdir('test')
 >>> f = open('test/\U00010000.txt', 'w')
 >>> next(os.scandir(b'test')).name
 b'\xf0\x90\x80\x80.txt'
 >>> next(os.scandir(b'test')).path
 b'test\\\xf0\x90\x80\x80.txt'

Heh, totally by accident as well.

Guido just said that we should make sure that os.scandir() continues to not support bytes, even after the patch for issue27781 goes in, so I'm reopening this.

But why? The code of os and glob modules could be simpler if add the support.

I'm ok to support os.scandir(bytes) on Windows.
My long term goal was to drop bytes support on Windows, but it seems like the new trend is more to keep this support and even enhance it.
It's quite easy to support os.scandir(bytes) on Windows, whereas it seems tricky and annoying to have to emulate the feature in pure Python.
Note: I was the BDFL-delegate for the PEP 471 ;-)

There are only two references to bytes in the pep:
> Like the other functions in the os module, scandir() accepts either a 
> bytes or str object for the path parameter, and returns the 
> DirEntry.name and DirEntry.path attributes with the same type as
> path . However, it is strongly recommended to use the str type, as
> this ensures cross-platform support for Unicode filenames. (On
> Windows, bytes filenames have been deprecated since Python 3.3). 
I'm not concerned about supporting bytes here (with PEP 529, obviously), though the strong recommendation to use str should certainly stand.

I don't know what is the benefit of removing support of bytes paths in os.scandir(), but supporting bytes paths allows to remove ugly workarounds from os and glob modules.
 Lib/glob.py | 16 --------!!!!!!
 Lib/os.py | 67 ---------------------------------------------------------!!!
 2 files changed, 73 deletions(-), 10 modifications(!)

Nice!

I'm confused by the current title of this issue. Should the support of bytes paths in os.scandir() be removed or preserved? In latter case we can remove workaround code from os and glob.

It went back and forth. The current feeling is to *keep* that support,
especially since their deprecation on Windows has been *undone* in
3.6. Feel free to update the title (though the initial comments may be
harder to understand without the context of the original title).

I suggest to modify posixmodule.c to support bytes on Windows at the C
level.
Since Windows uses utf8, there is no more real reason to drop bytes support
only on Windows.

New changeset 7c36e6fd0232 by Serhiy Storchaka in branch '3.6':
Issue #27998: Removed workarounds for supporting bytes paths on Windows in
https://hg.python.org/cpython/rev/7c36e6fd0232
New changeset bcee710c42fe by Serhiy Storchaka in branch 'default':
Issue #27998: Removed workarounds for supporting bytes paths on Windows in
https://hg.python.org/cpython/rev/bcee710c42fe 

Tests are passed on Windows.
Now we need to document that bytes paths are supported in os.scandir() on Windows since 3.6.

Hmm, tests are passed on some Windows buildbots, but failed on others.
http://buildbot.python.org/all/builders/AMD64%20Windows10%203.x/builds/1631/steps/test/logs/stdio
======================================================================
ERROR: test_walk_topdown (test.test_os.BytesWalkTests)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "D:\buildarea3円.x.bolen-windows10\build\lib\test\test_os.py", line 890, in test_walk_topdown
 all = list(self.walk(self.walk_path))
 File "D:\buildarea3円.x.bolen-windows10\build\lib\test\test_os.py", line 1055, in walk
 for broot, bdirs, bfiles in os.walk(os.fsencode(top), **kwargs):
 File "D:\buildarea3円.x.bolen-windows10\build\lib\os.py", line 409, in walk
 yield from walk(new_path, topdown, onerror, followlinks)
 File "D:\buildarea3円.x.bolen-windows10\build\lib\os.py", line 367, in walk
 is_dir = entry.is_dir()
TypeError: bad argument type for built-in operation
----------------------------------------------------------------------
The error message is not very informative. Seems like some C API function takes an argument of wrong type (e.g. bytes instead of unicode).

I suppose the following patch should fix the issue. Could anybody please test it on Windows?

Here's an alternative patch, using PyUnicode_FSDecoder. It also adds path_object_error and path_object_error2 helper functions.

Thank you Eryk. The patch LGTM. Did you tested that it fixes tests?

With the patch I uploaded, test_glob and test_os BytesWalkTests both pass, in both Windows 10 and Linux. Without it those tests fail for me in Windows 10.

New changeset 4ed634870a9a by Serhiy Storchaka in branch '3.6':
Issue #27998: Fixed bytes path support in os.scandir() on Windows.
https://hg.python.org/cpython/rev/4ed634870a9a
New changeset 837114dea493 by Serhiy Storchaka in branch 'default':
Issue #27998: Fixed bytes path support in os.scandir() on Windows.
https://hg.python.org/cpython/rev/837114dea493 

Now it is documentation issue again.

Proposed patch documents bytes paths support on Windows.

Doc change looks good to me.

New changeset ac63c70635db by Serhiy Storchaka in branch '3.6':
Issue #27998: Documented bytes paths support on Windows.
https://hg.python.org/cpython/rev/ac63c70635db
New changeset 26195e07fcc5 by Serhiy Storchaka in branch 'default':
Issue #27998: Documented bytes paths support on Windows.
https://hg.python.org/cpython/rev/26195e07fcc5 

Thanks Steve.