homepage

If 'top' is an unicode directory name, os.listdir can still return non-unicode filenames if they can't be decoded. This case is not handled in the Python 2.x standard library version of os.walk and will cause join(top, name) to fail on such filenames with an UnicodeDecodeError.

What is your OS?

This problem only affects Linux as far as I know (in my case I'm using Fedora 21 Beta).

Your problem has two solutions.
1) Upgrade to Python 3 which handles correctly your use case (thanks to the PEP 383, surrogateescape error handler)
2) Only process filenames as bytes, and encode/decode manually (so you can decide how to handle undecodable filenames)

1) Is not yet possible for me unfortunately, some libraries I require are not yet available for Python 3 (but in the long run, this would be my preferred solution)
2) Would necessitate too many changes in a carefully crafted, unicode-only application. I think I'll just override os.listdir and filter out filenames that are not decodable, or override os.walk and do something equivalent.

> 1) Is not yet possible for me unfortunately, some libraries I require are not yet available for Python 3 (but in the long run, this would be my preferred solution)
I'm curious, which libraries?
Oh, I forgot to say that it's not possible to fix this issue in Python 2. Backporting the PEP 383 in Python 2 requires deep changes in the Unicode machinery, starting by the UTF-8 codec. Currently, the UTF-8 encoder encodes surrogates which violates Unicode standard and makes impossible to use this codec with the surrogateescape error handler.

> I'm curious, which libraries?
wxPython and wexpect (wexpect I could probably port myself, so the problem is mainly with wx)
> Oh, I forgot to say that it's not possible to fix this issue in Python 2. Backporting the PEP 383 in Python 2 requires deep changes in the Unicode machinery, starting by the UTF-8 codec.
Ok, that's understandable of course.