[Python-checkins] bpo-45413: Define "posix_venv", "nt_venv" and "venv" sysconfig installation schemes (GH-31034)

Fri Mar 18 05:53:36 EDT 2022

https://github.com/python/cpython/commit/48d926269963cfe7a49c0a4f34af4fe9b832399b
commit: 48d926269963cfe7a49c0a4f34af4fe9b832399b
branch: main
author: Miro Hrončok <miro at hroncok.cz>
committer: encukou <encukou at gmail.com>
date: 2022年03月18日T10:53:29+01:00
summary:
bpo-45413: Define "posix_venv", "nt_venv" and "venv" sysconfig installation schemes (GH-31034)
Define *posix_venv* and *nt_venv* sysconfig installation schemes
to be used for bootstrapping new virtual environments.
Add *venv* sysconfig installation scheme to get the appropriate one of the above.
The schemes are identical to the pre-existing
*posix_prefix* and *nt* install schemes.
The venv module now uses the *venv* scheme to create new virtual environments
instead of hardcoding the paths depending only on the platform. Downstream
Python distributors customizing the *posix_prefix* or *nt* install
scheme in a way that is not compatible with the install scheme used in
virtual environments are encouraged not to customize the *venv* schemes.
When Python itself runs in a virtual environment,
sysconfig.get_default_scheme and
sysconfig.get_preferred_scheme with `key="prefix"` returns
*venv*.
files:
A Misc/NEWS.d/next/Library/2022-01-31-15-19-38.bpo-45413.1vaS0V.rst
M Doc/library/sysconfig.rst
M Doc/library/venv.rst
M Doc/whatsnew/3.11.rst
M Lib/sysconfig.py
M Lib/test/test_sysconfig.py
M Lib/test/test_venv.py
M Lib/venv/__init__.py

diff --git a/Doc/library/sysconfig.rst b/Doc/library/sysconfig.rst
index 713be1e02cea6..fa18d62d22af5 100644
--- a/Doc/library/sysconfig.rst
+++ b/Doc/library/sysconfig.rst
@@ -73,7 +73,7 @@ Every new component that is installed using :mod:`distutils` or a
 Distutils-based system will follow the same scheme to copy its file in the right
 places.
 
-Python currently supports six schemes:
+Python currently supports nine schemes:
 
 - *posix_prefix*: scheme for POSIX platforms like Linux or macOS. This is
 the default scheme used when Python or a component is installed.
@@ -83,8 +83,14 @@ Python currently supports six schemes:
 - *posix_user*: scheme for POSIX platforms used when a component is installed
 through Distutils and the *user* option is used. This scheme defines paths
 located under the user home directory.
+- *posix_venv*: scheme for :mod:`Python virtual environments <venv>` on POSIX
+ platforms; by default it is the same as *posix_prefix* .
 - *nt*: scheme for NT platforms like Windows.
 - *nt_user*: scheme for NT platforms, when the *user* option is used.
+- *nt_venv*: scheme for :mod:`Python virtual environments <venv>` on NT
+ platforms; by default it is the same as *nt* .
+- *venv*: a scheme with values from ether *posix_venv* or *nt_venv* depending
+ on the platform Python runs on
 - *osx_framework_user*: scheme for macOS, when the *user* option is used.
 
 Each scheme is itself composed of a series of paths and each path has a unique
@@ -119,6 +125,9 @@ identifier. Python currently uses eight paths:
 This function was previously named ``_get_default_scheme()`` and
 considered an implementation detail.
 
+ .. versionchanged:: 3.11
+ When Python runs from a virtual environment,
+ the *venv* scheme is returned.
 
 .. function:: get_preferred_scheme(key)
 
@@ -132,6 +141,10 @@ identifier. Python currently uses eight paths:
 
 .. versionadded:: 3.10
 
+ .. versionchanged:: 3.11
+ When Python runs from a virtual environment and ``key="prefix"``,
+ the *venv* scheme is returned.
+
 
 .. function:: _get_preferred_schemes()
 
diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst
index 092781b5ff1c4..b40bd4102c259 100644
--- a/Doc/library/venv.rst
+++ b/Doc/library/venv.rst
@@ -177,6 +177,11 @@ creation according to their needs, the :class:`EnvBuilder` class.
 ``clear=True``, contents of the environment directory will be cleared
 and then all necessary subdirectories will be recreated.
 
+ .. versionchanged:: 3.11
+ The *venv*
+ :ref:`sysconfig installation scheme <installation_paths>`
+ is used to construct the paths of the created directories.
+
 .. method:: create_configuration(context)
 
 Creates the ``pyvenv.cfg`` configuration file in the environment.
diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst
index 391423407eecd..2af663809a448 100644
--- a/Doc/whatsnew/3.11.rst
+++ b/Doc/whatsnew/3.11.rst
@@ -361,6 +361,24 @@ sys
 (equivalent to ``sys.exc_info()[1]``).
 (Contributed by Irit Katriel in :issue:`46328`.)
 
+
+sysconfig
+---------
+
+* Two new :ref:`installation schemes <installation_paths>`
+ (*posix_venv*, *nt_venv* and *venv*) were added and are used when Python
+ creates new virtual environments or when it is running from a virtual
+ environment.
+ The first two schemes (*posix_venv* and *nt_venv*) are OS-specific
+ for non-Windows and Windows, the *venv* is essentially an alias to one of
+ them according to the OS Python runs on.
+ This is useful for downstream distributors who modify
+ :func:`sysconfig.get_preferred_scheme`.
+ Third party code that creates new virtual environments should use the new
+ *venv* installation scheme to determine the paths, as does :mod:`venv`.
+ (Contributed by Miro Hrončok in :issue:`45413`.)
+
+
 threading
 ---------
 
@@ -395,6 +413,20 @@ unicodedata
 * The Unicode database has been updated to version 14.0.0. (:issue:`45190`).
 
 
+venv
+----
+
+* When new Python virtual environments are created, the *venv*
+ :ref:`sysconfig installation scheme <installation_paths>` is used
+ to determine the paths inside the environment.
+ When Python runs in a virtual environment, the same installation scheme
+ is the default.
+ That means that downstream distributors can change the default sysconfig install
+ scheme without changing behavior of virtual environments.
+ Third party code that also creates new virtual environments should do the same.
+ (Contributed by Miro Hrončok in :issue:`45413`.)
+
+
 fcntl
 -----
 
diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py
index d4a8a680286c1..2a01342eda8d6 100644
--- a/Lib/sysconfig.py
+++ b/Lib/sysconfig.py
@@ -56,8 +56,53 @@
 'scripts': '{base}/Scripts',
 'data': '{base}',
 },
+ # Downstream distributors can overwrite the default install scheme.
+ # This is done to support downstream modifications where distributors change
+ # the installation layout (eg. different site-packages directory).
+ # So, distributors will change the default scheme to one that correctly
+ # represents their layout.
+ # This presents an issue for projects/people that need to bootstrap virtual
+ # environments, like virtualenv. As distributors might now be customizing
+ # the default install scheme, there is no guarantee that the information
+ # returned by sysconfig.get_default_scheme/get_paths is correct for
+ # a virtual environment, the only guarantee we have is that it is correct
+ # for the *current* environment. When bootstrapping a virtual environment,
+ # we need to know its layout, so that we can place the files in the
+ # correct locations.
+ # The "*_venv" install scheme is a scheme to bootstrap virtual environments,
+ # essentially identical to the default posix_prefix/nt schemes.
+ # Downstream distributors who patch posix_prefix/nt scheme are encouraged to
+ # leave the following schemes unchanged
+ 'posix_venv': {
+ 'stdlib': '{installed_base}/{platlibdir}/python{py_version_short}',
+ 'platstdlib': '{platbase}/{platlibdir}/python{py_version_short}',
+ 'purelib': '{base}/lib/python{py_version_short}/site-packages',
+ 'platlib': '{platbase}/{platlibdir}/python{py_version_short}/site-packages',
+ 'include':
+ '{installed_base}/include/python{py_version_short}{abiflags}',
+ 'platinclude':
+ '{installed_platbase}/include/python{py_version_short}{abiflags}',
+ 'scripts': '{base}/bin',
+ 'data': '{base}',
+ },
+ 'nt_venv': {
+ 'stdlib': '{installed_base}/Lib',
+ 'platstdlib': '{base}/Lib',
+ 'purelib': '{base}/Lib/site-packages',
+ 'platlib': '{base}/Lib/site-packages',
+ 'include': '{installed_base}/Include',
+ 'platinclude': '{installed_base}/Include',
+ 'scripts': '{base}/Scripts',
+ 'data': '{base}',
+ },
 }
 
+# For the OS-native venv scheme, we essentially provide an alias:
+if os.name == 'nt':
+ _INSTALL_SCHEMES['venv'] = _INSTALL_SCHEMES['nt_venv']
+else:
+ _INSTALL_SCHEMES['venv'] = _INSTALL_SCHEMES['posix_venv']
+
 
 # NOTE: site.py has copy of this function.
 # Sync it when modify this function.
@@ -251,6 +296,8 @@ def _get_preferred_schemes():
 
 
 def get_preferred_scheme(key):
+ if key == 'prefix' and sys.prefix != sys.base_prefix:
+ return 'venv'
 scheme = _get_preferred_schemes()[key]
 if scheme not in _INSTALL_SCHEMES:
 raise ValueError(
diff --git a/Lib/test/test_sysconfig.py b/Lib/test/test_sysconfig.py
index 2c4120979d9a2..c7ec78fa4dc81 100644
--- a/Lib/test/test_sysconfig.py
+++ b/Lib/test/test_sysconfig.py
@@ -139,6 +139,72 @@ def test_get_preferred_schemes(self):
 self.assertIsInstance(schemes, dict)
 self.assertEqual(set(schemes), expected_schemes)
 
+ def test_posix_venv_scheme(self):
+ # The following directories were hardcoded in the venv module
+ # before bpo-45413, here we assert the posix_venv scheme does not regress
+ binpath = 'bin'
+ incpath = 'include'
+ libpath = os.path.join('lib',
+ 'python%d.%d' % sys.version_info[:2],
+ 'site-packages')
+
+ # Resolve the paths in prefix
+ binpath = os.path.join(sys.prefix, binpath)
+ incpath = os.path.join(sys.prefix, incpath)
+ libpath = os.path.join(sys.prefix, libpath)
+
+ self.assertEqual(binpath, sysconfig.get_path('scripts', scheme='posix_venv'))
+ self.assertEqual(libpath, sysconfig.get_path('purelib', scheme='posix_venv'))
+
+ # The include directory on POSIX isn't exactly the same as before,
+ # but it is "within"
+ sysconfig_includedir = sysconfig.get_path('include', scheme='posix_venv')
+ self.assertTrue(sysconfig_includedir.startswith(incpath + os.sep))
+
+ def test_nt_venv_scheme(self):
+ # The following directories were hardcoded in the venv module
+ # before bpo-45413, here we assert the posix_venv scheme does not regress
+ binpath = 'Scripts'
+ incpath = 'Include'
+ libpath = os.path.join('Lib', 'site-packages')
+
+ # Resolve the paths in prefix
+ binpath = os.path.join(sys.prefix, binpath)
+ incpath = os.path.join(sys.prefix, incpath)
+ libpath = os.path.join(sys.prefix, libpath)
+
+ self.assertEqual(binpath, sysconfig.get_path('scripts', scheme='nt_venv'))
+ self.assertEqual(incpath, sysconfig.get_path('include', scheme='nt_venv'))
+ self.assertEqual(libpath, sysconfig.get_path('purelib', scheme='nt_venv'))
+
+ def test_venv_scheme(self):
+ if sys.platform == 'win32':
+ self.assertEqual(
+ sysconfig.get_path('scripts', scheme='venv'),
+ sysconfig.get_path('scripts', scheme='nt_venv')
+ )
+ self.assertEqual(
+ sysconfig.get_path('include', scheme='venv'),
+ sysconfig.get_path('include', scheme='nt_venv')
+ )
+ self.assertEqual(
+ sysconfig.get_path('purelib', scheme='venv'),
+ sysconfig.get_path('purelib', scheme='nt_venv')
+ )
+ else:
+ self.assertEqual(
+ sysconfig.get_path('scripts', scheme='venv'),
+ sysconfig.get_path('scripts', scheme='posix_venv')
+ )
+ self.assertEqual(
+ sysconfig.get_path('include', scheme='venv'),
+ sysconfig.get_path('include', scheme='posix_venv')
+ )
+ self.assertEqual(
+ sysconfig.get_path('purelib', scheme='venv'),
+ sysconfig.get_path('purelib', scheme='posix_venv')
+ )
+
 def test_get_config_vars(self):
 cvars = get_config_vars()
 self.assertIsInstance(cvars, dict)
@@ -267,7 +333,7 @@ def test_get_config_h_filename(self):
 self.assertTrue(os.path.isfile(config_h), config_h)
 
 def test_get_scheme_names(self):
- wanted = ['nt', 'posix_home', 'posix_prefix']
+ wanted = ['nt', 'posix_home', 'posix_prefix', 'posix_venv', 'nt_venv', 'venv']
 if HAS_USER_BASE:
 wanted.extend(['nt_user', 'osx_framework_user', 'posix_user'])
 self.assertEqual(get_scheme_names(), tuple(sorted(wanted)))
diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py
index 043158c79214b..db812f21dbc56 100644
--- a/Lib/test/test_venv.py
+++ b/Lib/test/test_venv.py
@@ -236,6 +236,20 @@ def test_prefixes(self):
 out, err = check_output(cmd)
 self.assertEqual(out.strip(), expected.encode(), prefix)
 
+ @requireVenvCreate
+ def test_sysconfig_preferred_and_default_scheme(self):
+ """
+ Test that the sysconfig preferred(prefix) and default scheme is venv.
+ """
+ rmtree(self.env_dir)
+ self.run_with_capture(venv.create, self.env_dir)
+ envpy = os.path.join(self.env_dir, self.bindir, self.exe)
+ cmd = [envpy, '-c', None]
+ for call in ('get_preferred_scheme("prefix")', 'get_default_scheme()'):
+ cmd[2] = 'import sysconfig; print(sysconfig.%s)' % call
+ out, err = check_output(cmd)
+ self.assertEqual(out.strip(), b'venv', err)
+
 if sys.platform == 'win32':
 ENV_SUBDIRS = (
 ('Scripts',),
diff --git a/Lib/venv/__init__.py b/Lib/venv/__init__.py
index b90765074c36d..a8640d9163fbe 100644
--- a/Lib/venv/__init__.py
+++ b/Lib/venv/__init__.py
@@ -93,6 +93,15 @@ def clear_directory(self, path):
 elif os.path.isdir(fn):
 shutil.rmtree(fn)
 
+ def _venv_path(self, env_dir, name):
+ vars = {
+ 'base': env_dir,
+ 'platbase': env_dir,
+ 'installed_base': env_dir,
+ 'installed_platbase': env_dir,
+ }
+ return sysconfig.get_path(name, scheme='venv', vars=vars)
+
 def ensure_directories(self, env_dir):
 """
 Create the directories for the environment.
@@ -120,18 +129,12 @@ def create_if_needed(d):
 context.executable = executable
 context.python_dir = dirname
 context.python_exe = exename
- if sys.platform == 'win32':
- binname = 'Scripts'
- incpath = 'Include'
- libpath = os.path.join(env_dir, 'Lib', 'site-packages')
- else:
- binname = 'bin'
- incpath = 'include'
- libpath = os.path.join(env_dir, 'lib',
- 'python%d.%d' % sys.version_info[:2],
- 'site-packages')
- context.inc_path = path = os.path.join(env_dir, incpath)
- create_if_needed(path)
+ binpath = self._venv_path(env_dir, 'scripts')
+ incpath = self._venv_path(env_dir, 'include')
+ libpath = self._venv_path(env_dir, 'purelib')
+
+ context.inc_path = incpath
+ create_if_needed(incpath)
 create_if_needed(libpath)
 # Issue 21197: create lib64 as a symlink to lib on 64-bit non-OS X POSIX
 if ((sys.maxsize > 2**32) and (os.name == 'posix') and
@@ -139,8 +142,8 @@ def create_if_needed(d):
 link_path = os.path.join(env_dir, 'lib64')
 if not os.path.exists(link_path): # Issue #21643
 os.symlink('lib', link_path)
- context.bin_path = binpath = os.path.join(env_dir, binname)
- context.bin_name = binname
+ context.bin_path = binpath
+ context.bin_name = os.path.relpath(binpath, env_dir)
 context.env_exe = os.path.join(binpath, exename)
 create_if_needed(binpath)
 # Assign and update the command to use when launching the newly created
diff --git a/Misc/NEWS.d/next/Library/2022-01-31-15-19-38.bpo-45413.1vaS0V.rst b/Misc/NEWS.d/next/Library/2022-01-31-15-19-38.bpo-45413.1vaS0V.rst
new file mode 100644
index 0000000000000..6daff85781a5d
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2022-01-31-15-19-38.bpo-45413.1vaS0V.rst
@@ -0,0 +1,15 @@
+Define *posix_venv* and *nt_venv*
+:ref:`sysconfig installation schemes <installation_paths>`
+to be used for bootstrapping new virtual environments.
+Add *venv* sysconfig installation scheme to get the appropriate one of the above.
+The schemes are identical to the pre-existing
+*posix_prefix* and *nt* install schemes.
+The :mod:`venv` module now uses the *venv* scheme to create new virtual environments
+instead of hardcoding the paths depending only on the platform. Downstream
+Python distributors customizing the *posix_prefix* or *nt* install
+scheme in a way that is not compatible with the install scheme used in
+virtual environments are encouraged not to customize the *venv* schemes.
+When Python itself runs in a virtual environment,
+:func:`sysconfig.get_default_scheme` and
+:func:`sysconfig.get_preferred_scheme` with ``key="prefix"`` returns
+*venv*.
