homepage

Currently, the code to select the encoding and error handler of sys.stdin, sys.stdout and sys.stderr is run later in Py_Initialize(). I propose to move most of this code into _PyCoreConfig_Read() and add stdio_encoding and stdio_errors to _PyCoreConfig to easily let Python embedders to select the encoding.
Attached PR implements this idea. During Py_Initialize(), the code still reads the LC_CTYPE locale to choose the error handler if stdio_errors is NULL. This check cannot be done earlier, since the LC_CTYPE locale is only set to the user locale in Py_Initialize(): after _PyCoreConfig_Read().
The Py_SetStandardStreamEncoding() function has been added by Nick Coghlan in bpo-16129.
 This new pre-initialization API allows embedding
 applications like Blender to force a particular
 encoding and error handler for the standard IO streams.

New changeset 9e4994d410970fb4e75168401d159ba47a8f7108 by Victor Stinner in branch 'master':
bpo-34485: Enhance init_sys_streams() (GH-8978)
https://github.com/python/cpython/commit/9e4994d410970fb4e75168401d159ba47a8f7108

New changeset 2c8ddcf4f14f3e4c87a6fe6678ab5ad09130c6ab by Victor Stinner in branch 'master':
bpo-34485: Fix _Py_InitializeCore() for C locale coercion (GH-8979)
https://github.com/python/cpython/commit/2c8ddcf4f14f3e4c87a6fe6678ab5ad09130c6ab

New changeset 98c49c6ab239875e35a3c271bc8fabde6c9be804 by Victor Stinner in branch '3.7':
bpo-34485: Fix _Py_InitializeCore() for C locale coercion (GH-8979) (GH-8981)
https://github.com/python/cpython/commit/98c49c6ab239875e35a3c271bc8fabde6c9be804

New changeset 124b9eb4e41ba1676dad954eec9a1fb3480794e4 by Victor Stinner in branch 'master':
bpo-34485: Add _Py_ClearStandardStreamEncoding() (GH-8982)
https://github.com/python/cpython/commit/124b9eb4e41ba1676dad954eec9a1fb3480794e4

New changeset 315877dc361d554bec34b4b62c270479ad36a1be by Victor Stinner in branch 'master':
bpo-34485: stdout uses surrogateescape on POSIX locale (GH-8986)
https://github.com/python/cpython/commit/315877dc361d554bec34b4b62c270479ad36a1be

New changeset 0b9ea4b211b24464c7d38f63e45e51c275c52dcd by Victor Stinner in branch '3.7':
[3.7] bpo-34485: stdout uses surrogateescape on POSIX locale (GH-8986) (GH-8987)
https://github.com/python/cpython/commit/0b9ea4b211b24464c7d38f63e45e51c275c52dcd

New changeset 177d921c8c03d30daa32994362023f777624b10d by Victor Stinner in branch 'master':
bpo-34485, Windows: LC_CTYPE set to user preference (GH-8988)
https://github.com/python/cpython/commit/177d921c8c03d30daa32994362023f777624b10d

New changeset dfe0dc74536dfb6f331131d9b2b49557675bb6b7 by Victor Stinner in branch 'master':
bpo-34485: Add _PyCoreConfig.stdio_encoding (GH-8881)
https://github.com/python/cpython/commit/dfe0dc74536dfb6f331131d9b2b49557675bb6b7

Oh wow, I didn't expect that it would be so complicated to "just" get the encoding of sys.stdout... I had to fix many bugs to be able to do that!

New changeset cf21504194927b2f22132f48effea69eb8ade751 by Victor Stinner in branch 'master':
bpo-34485: Emit C locale coercion warning later (GH-9002)
https://github.com/python/cpython/commit/cf21504194927b2f22132f48effea69eb8ade751

Yeah, there were some good reasons I went with the relatively brute force option of provideding Blender with a new config API call back in bpo-16129 - as we've seen, actually fixing it properly has been a multi-year multi-person effort :)

> Yeah, there were some good reasons I went with the relatively brute force option of provideding Blender with a new config API call back in bpo-16129 - as we've seen, actually fixing it properly has been a multi-year multi-person effort :)
Yeah, I see ;-) Thanks for Py_SetStandardStreamEncoding(), it fixed Blender use case!

New changeset f01b2a1b84ee08df73a78cf1017eecf15e3cb995 by Victor Stinner in branch 'master':
bpo-34544: Fix setlocale() in pymain_read_conf() (GH-9041)
https://github.com/python/cpython/commit/f01b2a1b84ee08df73a78cf1017eecf15e3cb995

New changeset 73b00becbdd40f6a80cfa00abf1ae650a2b5e454 by Victor Stinner in branch 'master':
bpo-34544: pymain_read_conf() don't change LC_ALL (GH-9045)
https://github.com/python/cpython/commit/73b00becbdd40f6a80cfa00abf1ae650a2b5e454