homepage

Attached patch adds support.NONASCII to have a "portable" non-ASCII character that can be used to test non-ASCII strings. The patch uses it in some existing functions.
I wrote the patch on the default branch, we may start to use it since Python 3.2.

I think you should ensure that os.fsdecode(os.fsencode(character)) == character.

If NONASCII is None I suggest the followed fallback code
 for i in range(0x80, 0xFFFF):
 character = chr(i)
 if character.isprintable():
 try:
 if os.fsdecode(os.fsencode(character)) == character:
 NONASCII = character
 break
 except UnicodeError:
 pass

+# NONASCII: non-ASCII character encodable by os.fsencode(),
+# or None if there is no such character.
+NONASCII = None
Can you use a name that reflects that this is a specific type of non-ASCII character having a special property (e.g. FS_NONASCII)? I think "ASCII" should be reserved for a generic non-ASCII character. Moreover, there may be other types of non-ASCII we can add in the future.

> I think you should ensure that os.fsdecode(os.fsencode(character)) == character.
Chosen characters respect this property, but it doesn't hurt to add such check.
> Can you use a name that reflects that this is a specific type
> of non-ASCII character having a special property (e.g. FS_NONASCII)?
Done.

> If NONASCII is None I suggest the followed fallback code
I prefer to not "brute force" Unicode because it would slow down any test, even tests not using FS_NONASCII. I wrote attached brute.py script to compute an exhaustive list of non-ASCII characters encodable to "any" locale encoding. My locale encoding list is not complete, but it should be enough for our purpose. The list can be completed later.

I tested support_non_ascii-2.patch on Windows with cp932 ANSI code page (FS encoding), and on Linux with ASCII, ISO-8859-1, ISO-8859-15 and UTF-8 locale encodings.

I tested brute.py for all supported in Python encodings:
No character for encoding cp1006:surrogateescape :-(
No character for encoding cp720:surrogateescape :-(
No character for encoding cp864:surrogateescape :-(
No character for encoding iso8859_3:surrogateescape :-(
No character for encoding iso8859_6:surrogateescape :-(
No character for encoding mac_arabic:surrogateescape :-(
No character for encoding mac_farsi:surrogateescape :-(

> I tested brute.py for all supported in Python encodings:
Oh thanks, interesting result. I completed the encoding list and the character list: see brute2.py. I added "joker" characters: U+00A0 and U+20AC which match requierements for most locale encodings.

New changeset de8cf1ece068 by Victor Stinner in branch 'default':
Issue #16414: Add support.FS_NONASCII and support.TESTFN_NONASCII
http://hg.python.org/cpython/rev/de8cf1ece068 

New changeset 0e9fbdda3c92 by Victor Stinner in branch 'default':
Issue #16414: Fix support.TESTFN_UNDECODABLE and test_genericpath.test_nonascii_abspath()
http://hg.python.org/cpython/rev/0e9fbdda3c92 

Why were you add '- ' suffix to TESTFN_NONASCII?

I don't see U+00A0 and U+20AC in the changeset.

New changeset 55710b8c6670 by Victor Stinner in branch 'default':
Issue #16414: Fix typo in support.TESTFN_NONASCII (useless space)
http://hg.python.org/cpython/rev/55710b8c6670 

New changeset 7f90305d9f23 by Victor Stinner in branch 'default':
Issue #16414: Test more characters for support.FS_NONASCII
http://hg.python.org/cpython/rev/7f90305d9f23 

New changeset fce9e892c65d by Victor Stinner in branch 'default':
Issue #16414: Fix test_os on Windows, don't test os.listdir() with undecodable
http://hg.python.org/cpython/rev/fce9e892c65d 

> Why were you add '- ' suffix to TESTFN_NONASCII?
Oops, the space was a mistake. I add "-" just for the readability of the generated filename.
> I don't see U+00A0 and U+20AC in the changeset.
Oh, I forgot to update the patch with the latest results of "brute2.py". It is now fixed.
Thanks for the review!

Handling non-ASCII paths is always a pain. I don't plan to backport support.FS_NONASCII to Python 3.3 right now, but I may backport it later.

New changeset 41658a4fb3cc by Victor Stinner in branch '3.2':
Issue #16218, #16414, #16444: Backport FS_NONASCII, TESTFN_UNDECODABLE,
http://hg.python.org/cpython/rev/41658a4fb3cc
New changeset 4d40c1ce8566 by Victor Stinner in branch '3.3':
(Merge 3.2) Issue #16218, #16414, #16444: Backport FS_NONASCII,
http://hg.python.org/cpython/rev/4d40c1ce8566