homepage

When Python path configuration is not properly configured, Python fails with an error looking like:
---
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
Current thread 0x00007ff0065b2740 (most recent call first):
<no Python frame>
---
This error is very common and hard to debug. It can happen in various cases for various reasons:
* Python is embedded in an application
* PYTHONPATH and/or PYTHONHOME environment variable is set
* pyenv.cfg configuration for virtual environment
* Local .py files which override the standard library
A few examples:
* bpo-38234: Py_SetPath() regression in Python 3.8 which impacts PyInstaller
* https://bugs.python.org/issue35978#msg335363 virtual env
* https://bugs.python.org/issue35969#msg335249 "encodings.py" file in the current directory
* https://stackoverflow.com/questions/55657593/python-3-8-0a3-failed-to-get-the-python-codec-of-the-filesystem-encoding embedded Python
* https://askubuntu.com/questions/640010/fatal-python-error-py-initialize-unable-to-get-the-locale-encoding-syntaxe Python 3 installed by Anaconda which uses Python 2.7 stdlib because of PYTHONPATH
* https://github.com/GNS3/gns3-gui/issues/2267 Python embedded in "gns3"
* https://github.com/JuliaPy/PyCall.jl/issues/410 venv
* https://techglimpse.com/error-executing-python3-5-command-solution/ another random error
Attached PR dumps the Python path configuration if importing the Python codec of the filesystem encoding fails. In the Python initialization, it's the first import which is done.

Without my change:
---
vstinner@apu$ PYTHONHOME=/xxx ./python 
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
Current thread 0x00007fa9c32d6740 (most recent call first):
<no Python frame>
---
With my PR 16300:
---
$ PYTHONHOME=/xxx ./python 
Python path configuration:
 PYTHONHOME = '/xxx'
 PYTHONPATH = '/usr/share/qa-tools/python-modules'
 isolated = 0
 environment = 1
 user site = 1
 import site = 1
 sys.executable = '/home/vstinner/python/master/python'
 sys.prefix = '/xxx'
 sys.exec_prefix = '/xxx'
 sys.path = [
 '/usr/share/qa-tools/python-modules',
 '/xxx/lib/python39.zip',
 '/xxx/lib/python3.9',
 '/xxx/lib/python3.9/lib-dynload',
 ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
Current thread 0x00007f9cef949740 (most recent call first):
<no Python frame>
---

> user site = 1
> import site = 1
I'm not sure that dumping these configuration variables are useful to debug this very specific error:
"Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding"
They only have an effect on sys.path later.

+1

New changeset fcdb027234566c4d506d6d753c7d5638490fb088 by Victor Stinner in branch 'master':
bpo-38236: Dump path config at first import error (GH-16300)
https://github.com/python/cpython/commit/fcdb027234566c4d506d6d753c7d5638490fb088

New changeset 88e6447451fb5525e83e802c66c3e51b4a45bf86 by Victor Stinner in branch 'master':
bpo-38236: Fix init_dump_ascii_wstr() (GH-16333)
https://github.com/python/cpython/commit/88e6447451fb5525e83e802c66c3e51b4a45bf86

New changeset c5c642565e260477ae2fb29d0c86a91e19702ae3 by Victor Stinner in branch '3.8':
bpo-38236: Dump path config at first import error (GH-16300) (GH-16332)
https://github.com/python/cpython/commit/c5c642565e260477ae2fb29d0c86a91e19702ae3

I backported my change to Python 3.8.
Example with PYTHONPATH set to Python 2.7 with Python 3.8:
$ PYTHONPATH=/usr/lib64/python2.7/ ./python -c pass
Python path configuration:
 PYTHONHOME = (not set)
 PYTHONPATH = '/usr/lib64/python2.7/'
 program name = './python'
 isolated = 0
 environment = 1
 user site = 1
 import site = 1
 sys._base_executable = '/home/vstinner/python/3.8/python'
 sys.base_prefix = '/usr/local'
 sys.base_exec_prefix = '/usr/local'
 sys.executable = '/home/vstinner/python/3.8/python'
 sys.prefix = '/usr/local'
 sys.exec_prefix = '/usr/local'
 sys.path = [
 '/usr/lib64/python2.7/',
 '/usr/local/lib/python38.zip',
 '/home/vstinner/python/3.8/Lib',
 '/home/vstinner/python/3.8/build/lib.linux-x86_64-3.8-pydebug',
 ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
 File "/usr/lib64/python2.7/encodings/__init__.py", line 123
 raise CodecRegistryError,\
 ^
SyntaxError: invalid syntax
Current thread 0x00007fe3ade4d740 (most recent call first):
<no Python frame>