[Python-Dev] PEP 383: Non-decodable Bytes in System Character Interfaces

Thu Apr 30 22:20:31 CEST 2009

On Apr 30, 2009, at 5:42 AM, Martin v. Löwis wrote:
> I think you are right. I have now excluded ASCII bytes from being
> mapped, effectively not supporting any encodings that are not ASCII
> compatible. Does that sound ok?

Yes. The practical upshot of this is that users who brokenly use 
"ja_JP.SJIS" as their locale (which, note, first requires editing some 
files in /var/lib/locales manually to enable its use..) may still have 
python not work with invalid-in-shift-jis filenames. Since that locale 
is widely recognized as a bad idea to use, and is not supported by any 
distros, it certainly doesn't bother me that it isn't 100% supported 
in python. It seems like the most common reason why people want to use 
SJIS is to make old pre-unicode apps work right in WINE -- in which 
case it doesn't actually affect unix python at all.
I'd personally be fine with python just declaring that the filesystem- 
encoding will *always* be utf-8b and ignore the locale...but I expect 
some other people might complain about that. Of course, application 
authors can decide to do that themselves by calling 
sys.setfilesystemencoding('utf-8b') at the start of their program.
James