homepage

This issue tracker has been migrated to GitHub , and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: sys.stdout.errors is set to "surrogateescape"
Type: behavior Stage: resolved
Components: IO Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: ncoghlan, python-dev, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2015年10月08日 06:51 by serhiy.storchaka, last changed 2022年04月11日 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
default_io_error_handler.patch serhiy.storchaka, 2015年11月10日 20:50 review
Messages (12)
msg252515 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015年10月08日 06:51
The error handler of sys.stdout and sys.stdin is set to "surrogateescape" even for non-ASCII encoding.
$ LANG= PYTHONIOENCODING=UTF-8 ./python -c 'import sys; print(sys.stdout.encoding, sys.stdout.errors)'
UTF-8 surrogateescape
msg253009 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015年10月14日 16:35
Sorry, I don't understand the issue. Do you consider that using surrogateescape is a bug?
Which behaviour do you expect?
Python 3.5 now uses surrogateescape by default for stdout and stderr when the locale is POSIX. I guess that you got the POSIX locale using "LANG=".
msg253011 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015年10月14日 17:02
I'm not sure this is a bug, but it looks at least unexpected, that surrogateescape is used with non-ASCII encoding. For example my last test for issue19058 fails on POSIX locale in 3.5+, and it is not so easy to make it working.
May be change error handler to surrogateescape only if PYTHONIOENCODING is not specified?
msg253018 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015年10月14日 20:12
"it looks at least unexpected, that surrogateescape is used with non-ASCII encoding"
What do you mean by non-ASCII encoding? surrogateescape is used by all encodings for all OS operations on Python 3, like os.listdir(), even for UTF-8.
msg254463 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015年11月10日 20:50
The default encoding of sys.stdio and sys.stdout is determined by (in order of increasing precedence):
1. locale
2. PYTHONIOENCODING
3. Py_SetStandardStreamEncoding()
The default error handler before 3.5 was determined by:
1. 'strict'
2. PYTHONIOENCODING
3. Py_SetStandardStreamEncoding()
The default error handler since 3.5 (issue19977) is determined by:
1. PYTHONIOENCODING
2. locale
3. Py_SetStandardStreamEncoding()
Even if you explicitly specified the error handler by PYTHONIOENCODING, it doesn't have effect in POSIX locale. This doesn't look right to me. I think the order should be the same as for encoding.
Proposed patch makes PYTHONIOENCODING to override locale default for error handler.
msg262991 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016年04月07日 15:35
What do you think about this Victor?
msg263012 - (view) Author: Alyssa Coghlan (ncoghlan) * (Python committer) Date: 2016年04月08日 09:37
I believe the problem may be that we can't readily tell the difference between "PYTHONIOENCODING=ascii" and "PYTHONIOENCODING=ascii:strict", and in the former case we'd ideally still end up using "surrogateescape" by default.
That said, the real intent of the change was "If the detected encoding is ASCII, enable surrogateescape automatically", and detecting the POSIX locale was a proxy for that. We didn't account for PYTHONIOENCODING being used to select a more sensible encoding.
msg263015 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016年04月08日 10:46
Making "PYTHONIOENCODING=ascii" to mean "PYTHONIOENCODING=ascii:surrogateescape" is different (and may be more complex) issue. What error handler should use open(name, encoding='ascii')? open(name) in POSIX locale?
This issue is about incorrect working of PYTHONIOENCODING in POSIX locale.
msg263090 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016年04月09日 12:53
Ok, I now understand the issue. Your change looks good to me.
I agree that strict error handler is good choice for PYTHONIOENCODING=ascii.
msg263091 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016年04月09日 12:53
The patch looks good to me.
msg263130 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016年04月10日 11:45
New changeset 56eca1c08738 by Serhiy Storchaka in branch '3.5':
Issue #25339: PYTHONIOENCODING now has priority over locale in setting the
https://hg.python.org/cpython/rev/56eca1c08738
New changeset 9c6623099da1 by Serhiy Storchaka in branch 'default':
Issue #25339: PYTHONIOENCODING now has priority over locale in setting the
https://hg.python.org/cpython/rev/9c6623099da1 
msg263133 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016年04月10日 11:50
Thank you for your review Victor. I have added yet one minor change in tests because -I doesn't suppress PYTHONIOENCODING.
History
Date User Action Args
2022年04月11日 14:58:22adminsetgithub: 69526
2016年04月10日 11:50:50serhiy.storchakasetstatus: open -> closed
messages: + msg263133

assignee: serhiy.storchaka
resolution: fixed
stage: patch review -> resolved
2016年04月10日 11:45:44python-devsetnosy: + python-dev
messages: + msg263130
2016年04月09日 12:53:45vstinnersetmessages: + msg263091
2016年04月09日 12:53:00vstinnersetmessages: + msg263090
2016年04月08日 10:46:09serhiy.storchakasetmessages: + msg263015
2016年04月08日 09:37:07ncoghlansetmessages: + msg263012
2016年04月07日 15:35:30serhiy.storchakasetmessages: + msg262991
2015年11月10日 20:50:50serhiy.storchakasetfiles: + default_io_error_handler.patch
keywords: + patch
messages: + msg254463

stage: patch review
2015年10月14日 20:12:38vstinnersetmessages: + msg253018
2015年10月14日 17:02:57serhiy.storchakasetmessages: + msg253011
2015年10月14日 16:35:15vstinnersetmessages: + msg253009
2015年10月08日 06:51:27serhiy.storchakacreate

AltStyle によって変換されたページ (->オリジナル) /