[Python-checkins] cpython: Changes pyvenv.cfg trick into an actual sys.path file.

steve.dower python-checkins at python.org
Fri Sep 9 12:23:11 EDT 2016

• Previous message (by thread): [Python-checkins] cpython: Fixes expected error when getting encoding while shutting down.
• Next message (by thread): [Python-checkins] cpython: credit Raymond
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

---

https://hg.python.org/cpython/rev/03517dd54977
changeset: 103408:03517dd54977
parent: 103406:fdb3537e633a
user: Steve Dower <steve.dower at microsoft.com>
date: Fri Sep 09 09:17:35 2016 -0700
summary:
 Changes pyvenv.cfg trick into an actual sys.path file.
files:
 Doc/using/windows.rst | 53 ++++---
 Lib/site.py | 6 -
 PC/getpathp.c | 181 +++++++++++++++---------
 PCbuild/pythoncore.vcxproj | 2 +-
 Tools/msi/make_zip.py | 11 +-
 5 files changed, 154 insertions(+), 99 deletions(-)
diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst
--- a/Doc/using/windows.rst
+++ b/Doc/using/windows.rst
@@ -29,13 +29,13 @@
 
 As specified in :pep:`11`, a Python release only supports a Windows platform
 while Microsoft considers the platform under extended support. This means that
-Python 3.5 supports Windows Vista and newer. If you require Windows XP support
+Python 3.6 supports Windows Vista and newer. If you require Windows XP support
 then please install Python 3.4.
 
 Installation Steps
 ------------------
 
-Four Python 3.5 installers are available for download - two each for the 32-bit
+Four Python 3.6 installers are available for download - two each for the 32-bit
 and 64-bit versions of the interpreter. The *web installer* is a small initial
 download, and it will automatically download the required components as
 necessary. The *offline installer* includes the components necessary for a
@@ -193,13 +193,13 @@
 For example, to silently install a default, system-wide Python installation,
 you could use the following command (from an elevated command prompt)::
 
- python-3.5.0.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0
+ python-3.6.0.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0
 
 To allow users to easily install a personal copy of Python without the test
 suite, you could provide a shortcut with the following command. This will
 display a simplified initial page and disallow customization::
 
- python-3.5.0.exe InstallAllUsers=0 Include_launcher=0 Include_test=0
+ python-3.6.0.exe InstallAllUsers=0 Include_launcher=0 Include_test=0
 SimpleInstall=1 SimpleInstallDescription="Just for me, no test suite."
 
 (Note that omitting the launcher also omits file associations, and is only
@@ -234,13 +234,13 @@
 useful to have a locally cached copy.
 
 Execute the following command from Command Prompt to download all possible
-required files. Remember to substitute ``python-3.5.0.exe`` for the actual
+required files. Remember to substitute ``python-3.6.0.exe`` for the actual
 name of your installer, and to create layouts in their own directories to
 avoid collisions between files with the same name.
 
 ::
 
- python-3.5.0.exe /layout [optional target directory]
+ python-3.6.0.exe /layout [optional target directory]
 
 You may also specify the ``/quiet`` option to hide the progress display.
 
@@ -345,7 +345,7 @@
 To temporarily set environment variables, open Command Prompt and use the
 :command:`set` command::
 
- C:\>set PATH=C:\Program Files\Python 3.5;%PATH%
+ C:\>set PATH=C:\Program Files\Python 3.6;%PATH%
 C:\>set PYTHONPATH=%PYTHONPATH%;C:\My_python_lib
 C:\>python
 
@@ -401,10 +401,10 @@
 
 Besides using the automatically created start menu entry for the Python
 interpreter, you might want to start Python in the command prompt. The
-installer for Python 3.5 and later has an option to set that up for you.
+installer for Python 3.6 has an option to set that up for you.
 
-On the first page of the installer, an option labelled "Add Python 3.5 to
-PATH" can be selected to have the installer add the install location into the
+On the first page of the installer, an option labelled "Add Python to PATH"
+may be selected to have the installer add the install location into the
 :envvar:`PATH`. The location of the :file:`Scripts\\` folder is also added.
 This allows you to type :command:`python` to run the interpreter, and
 :command:`pip` for the package installer. Thus, you can also execute your
@@ -418,7 +418,7 @@
 example variable could look like this (assuming the first two entries already
 existed)::
 
- C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\Python 3.5
+ C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\Python 3.6
 
 .. _launcher:
 
@@ -720,7 +720,15 @@
 :file:`C:\\Python\\Lib\\` and third-party modules should be stored in
 :file:`C:\\Python\\Lib\\site-packages\\`.
 
-This is how :data:`sys.path` is populated on Windows:
+To completely override :data:`sys.path`, create a text file named ``'sys.path'``
+containing a list of paths alongside the Python executable. This will ignore all
+registry settings and environment variables, enable isolated mode, disable
+importing :mod:`site`, and fill :data:`sys.path` with exactly the paths listed
+in the file. Paths may be absolute or relative to the directory containing the
+file.
+
+When the ``'sys.path'`` file is missing, this is how :data:`sys.path` is
+populated on Windows:
 
 * An empty entry is added at the start, which corresponds to the current
 directory.
@@ -755,10 +763,6 @@
 path is used instead of the path to the main executable when deducing the
 home location.
 
-* If ``applocal`` is set to true, the ``home`` property or the main executable
- is always used as the home path, and all environment variables or registry
- values affecting the path are ignored. The landmark file is not checked.
-
 The end result of all this is:
 
 * When running :file:`python.exe`, or any other .exe in the main Python
@@ -777,13 +781,11 @@
 For those who want to bundle Python into their application or distribution, the
 following advice will prevent conflicts with other installations:
 
-* Include a ``pyvenv.cfg`` file alongside your executable containing
- ``applocal = true``. This will ensure that your own directory will be used to
- resolve paths even if you have included the standard library in a ZIP file.
- It will also ignore user site-packages and other paths listed in the
- registry.
+* Include a ``sys.path`` file alongside your executable containing the
+ directories to include. This will ignore user site-packages and other paths
+ listed in the registry or in environment variables.
 
-* If you are loading :file:`python3.dll` or :file:`python35.dll` in your own
+* If you are loading :file:`python3.dll` or :file:`python36.dll` in your own
 executable, explicitly call :c:func:`Py_SetPath` or (at least)
 :c:func:`Py_SetProgramName` before :c:func:`Py_Initialize`.
 
@@ -801,6 +803,11 @@
 the first suggestion is the best, as the other may still be susceptible to
 non-standard paths in the registry and user site-packages.
 
+.. versionchanged:: 3.6
+
+ Adds ``sys.path`` file support and removes ``applocal`` option from
+ ``pyvenv.cfg``.
+
 Additional modules
 ==================
 
@@ -900,7 +907,7 @@
 When extracted, the embedded distribution is (almost) fully isolated from the
 user's system, including environment variables, system registry settings, and
 installed packages. The standard library is included as pre-compiled and
-optimized ``.pyc`` files in a ZIP, and ``python3.dll``, ``python35.dll``,
+optimized ``.pyc`` files in a ZIP, and ``python3.dll``, ``python36.dll``,
 ``python.exe`` and ``pythonw.exe`` are all provided. Tcl/tk (including all
 dependants, such as Idle), pip and the Python documentation are not included.
 
diff --git a/Lib/site.py b/Lib/site.py
--- a/Lib/site.py
+++ b/Lib/site.py
@@ -463,12 +463,6 @@
 system_site = value.lower()
 elif key == 'home':
 sys._home = value
- elif key == 'applocal' and value.lower() == 'true':
- # App-local installs use the exe_dir as prefix,
- # not one level higher, and do not use system
- # site packages.
- site_prefix = exe_dir
- system_site = 'false'
 
 sys.prefix = sys.exec_prefix = site_prefix
 
diff --git a/PC/getpathp.c b/PC/getpathp.c
--- a/PC/getpathp.c
+++ b/PC/getpathp.c
@@ -6,7 +6,8 @@
 PATH RULES FOR WINDOWS:
 This describes how sys.path is formed on Windows. It describes the
 functionality, not the implementation (ie, the order in which these
- are actually fetched is different)
+ are actually fetched is different). The presence of a sys.path file
+ alongside the program overrides these rules - see below.
 
 * Python always adds an empty entry at the start, which corresponds
 to the current directory.
@@ -36,6 +37,12 @@
 used (eg. .\Lib;.\plat-win, etc)
 
 
+ If a sys.path file exists adjacent to python.exe, it must contain a
+ list of paths to add to sys.path, one per line (like a .pth file but without
+ the ability to execute arbitrary code). Each path is relative to the
+ directory containing the file. No other paths are added to the search path,
+ and the registry finder is not enabled.
+
 The end result of all this is:
 * When running python.exe, or any other .exe in the main Python directory
 (either an installed version, or directly from the PCbuild directory),
@@ -52,7 +59,10 @@
 some default, but relative, paths.
 
 * An embedding application can use Py_SetPath() to override all of
- these authomatic path computations.
+ these automatic path computations.
+
+ * An isolation install of Python can disable all implicit paths by
+ providing a sys.path file.
 
 ---------------------------------------------------------------- */
 
@@ -61,9 +71,12 @@
 #include "osdefs.h"
 #include <wchar.h>
 
-#ifdef MS_WINDOWS
+#ifndef MS_WINDOWS
+#error getpathp.c should only be built on Windows
+#endif
+
 #include <windows.h>
-#endif
+#include <Shlwapi.h>
 
 #ifdef HAVE_SYS_TYPES_H
 #include <sys/types.h>
@@ -163,24 +176,30 @@
 than MAXPATHLEN characters at exit. If stuff is too long, only as much of
 stuff as fits will be appended.
 */
+
+static int _PathCchCombineEx_Initialized = 0;
+typedef HRESULT(__stdcall *PPathCchCombineEx)(PWSTR pszPathOut, size_t cchPathOut, PCWSTR pszPathIn, PCWSTR pszMore, unsigned long dwFlags);
+static PPathCchCombineEx _PathCchCombineEx;
+
 static void
 join(wchar_t *buffer, const wchar_t *stuff)
 {
- size_t n;
- if (is_sep(stuff[0]) ||
- (wcsnlen_s(stuff, 4) >= 3 && stuff[1] == ':' && is_sep(stuff[2]))) {
- if (wcscpy_s(buffer, MAXPATHLEN+1, stuff) != 0)
- Py_FatalError("buffer overflow in getpathp.c's join()");
- return;
+ if (_PathCchCombineEx_Initialized == 0) {
+ HMODULE pathapi = LoadLibraryW(L"api-ms-win-core-path-l1-1-0.dll");
+ if (pathapi)
+ _PathCchCombineEx = (PPathCchCombineEx)GetProcAddress(pathapi, "PathCchCombineEx");
+ else
+ _PathCchCombineEx = NULL;
+ _PathCchCombineEx_Initialized = 1;
 }
 
- n = wcsnlen_s(buffer, MAXPATHLEN+1);
- if (n > 0 && !is_sep(buffer[n - 1]) && n < MAXPATHLEN) {
- buffer[n] = SEP;
- buffer[n + 1] = '0円';
+ if (_PathCchCombineEx) {
+ if (FAILED(_PathCchCombineEx(buffer, MAXPATHLEN+1, buffer, stuff, 0)))
+ Py_FatalError("buffer overflow in getpathp.c's join()");
+ } else {
+ if (!PathCombineW(buffer, NULL, stuff))
+ Py_FatalError("buffer overflow in getpathp.c's join()");
 }
- if (wcscat_s(buffer, MAXPATHLEN+1, stuff) != 0)
- Py_FatalError("buffer overflow in getpathp.c's join()");
 }
 
 /* gotlandmark only called by search_for_prefix, which ensures
@@ -214,7 +233,6 @@
 return 0;
 }
 
-#ifdef MS_WINDOWS
 #ifdef Py_ENABLE_SHARED
 
 /* a string loaded from the DLL at startup.*/
@@ -369,7 +387,6 @@
 return retval;
 }
 #endif /* Py_ENABLE_SHARED */
-#endif /* MS_WINDOWS */
 
 static void
 get_progpath(void)
@@ -378,7 +395,6 @@
 wchar_t *path = _wgetenv(L"PATH");
 wchar_t *prog = Py_GetProgramName();
 
-#ifdef MS_WINDOWS
 #ifdef Py_ENABLE_SHARED
 extern HANDLE PyWin_DLLhModule;
 /* static init of progpath ensures final char remains 0円 */
@@ -390,7 +406,6 @@
 #endif
 if (GetModuleFileNameW(NULL, progpath, MAXPATHLEN))
 return;
-#endif
 if (prog == NULL || *prog == '0円')
 prog = L"python";
 
@@ -483,6 +498,67 @@
 return result;
 }
 
+static int
+read_sys_path_file(const wchar_t *path, const wchar_t *prefix)
+{
+ FILE *sp_file = _Py_wfopen(path, L"r");
+ if (sp_file == NULL)
+ return -1;
+
+ size_t bufsiz = MAXPATHLEN;
+ size_t prefixlen = wcslen(prefix);
+
+ wchar_t *buf = (wchar_t*)PyMem_RawMalloc(bufsiz * sizeof(wchar_t));
+ buf[0] = '0円';
+
+ while (!feof(sp_file)) {
+ char line[MAXPATHLEN + 1];
+ char *p = fgets(line, MAXPATHLEN + 1, sp_file);
+ if (!p)
+ break;
+
+ DWORD n = strlen(line);
+ if (n == 0 || p[n - 1] != '\n')
+ break;
+ if (n > 2 && p[n - 1] == '\r')
+ --n;
+
+ DWORD wn = MultiByteToWideChar(CP_UTF8, 0, line, n - 1, NULL, 0);
+ wchar_t *wline = (wchar_t*)PyMem_RawMalloc((wn + 1) * sizeof(wchar_t));
+ wn = MultiByteToWideChar(CP_UTF8, 0, line, n - 1, wline, wn);
+ wline[wn] = '0円';
+
+ while (wn + prefixlen + 4 > bufsiz) {
+ bufsiz += MAXPATHLEN;
+ buf = (wchar_t*)PyMem_RawRealloc(buf, (bufsiz + 1) * sizeof(wchar_t));
+ if (!buf) {
+ PyMem_RawFree(wline);
+ goto error;
+ }
+ }
+
+ if (buf[0])
+ wcscat_s(buf, bufsiz, L";");
+ wchar_t *b = &buf[wcslen(buf)];
+ 
+ wcscat_s(buf, bufsiz, prefix);
+ join(b, wline);
+
+ PyMem_RawFree(wline);
+ }
+
+ module_search_path = buf;
+
+ fclose(sp_file);
+ return 0;
+
+error:
+ PyMem_RawFree(buf);
+ fclose(sp_file);
+ return -1;
+}
+
+
 static void
 calculate_path(void)
 {
@@ -492,32 +568,34 @@
 wchar_t *pythonhome = Py_GetPythonHome();
 wchar_t *envpath = NULL;
 
-#ifdef MS_WINDOWS
 int skiphome, skipdefault;
 wchar_t *machinepath = NULL;
 wchar_t *userpath = NULL;
 wchar_t zip_path[MAXPATHLEN+1];
- int applocal = 0;
 
 if (!Py_IgnoreEnvironmentFlag) {
 envpath = _wgetenv(L"PYTHONPATH");
 }
-#else
- char *_envpath = Py_GETENV("PYTHONPATH");
- wchar_t wenvpath[MAXPATHLEN+1];
- if (_envpath) {
- size_t r = mbstowcs(wenvpath, _envpath, MAXPATHLEN+1);
- envpath = wenvpath;
- if (r == (size_t)-1 || r >= MAXPATHLEN)
- envpath = NULL;
- }
-#endif
 
 get_progpath();
 /* progpath guaranteed 0円 terminated in MAXPATH+1 bytes. */
 wcscpy_s(argv0_path, MAXPATHLEN+1, progpath);
 reduce(argv0_path);
 
+ /* Search for a sys.path file */
+ {
+ wchar_t spbuffer[MAXPATHLEN+1];
+
+ wcscpy_s(spbuffer, MAXPATHLEN+1, argv0_path);
+ join(spbuffer, L"sys.path");
+ if (exists(spbuffer) && read_sys_path_file(spbuffer, argv0_path) == 0) {
+ wcscpy_s(prefix, MAXPATHLEN + 1, argv0_path);
+ Py_IsolatedFlag = 1;
+ Py_NoSiteFlag = 1;
+ return;
+ }
+ }
+
 /* Search for an environment configuration file, first in the
 executable's directory and then in the parent directory.
 If found, open it for use when searching for prefixes.
@@ -543,17 +621,6 @@
 }
 }
 if (env_file != NULL) {
- /* Look for an 'applocal' variable and, if true, ignore all registry
- * keys and environment variables, but retain the default paths
- * (DLLs, Lib) and the zip file. Setting pythonhome here suppresses
- * the search for LANDMARK below and overrides %PYTHONHOME%.
- */
- if (find_env_config_value(env_file, L"applocal", tmpbuffer) &&
- (applocal = (wcsicmp(tmpbuffer, L"true") == 0))) {
- envpath = NULL;
- pythonhome = argv0_path;
- }
-
 /* Look for a 'home' variable and set argv0_path to it, if found */
 if (find_env_config_value(env_file, L"home", tmpbuffer)) {
 wcscpy_s(argv0_path, MAXPATHLEN+1, tmpbuffer);
@@ -576,7 +643,6 @@
 envpath = NULL;
 
 
-#ifdef MS_WINDOWS
 /* Calculate zip archive path from DLL or exe path */
 if (wcscpy_s(zip_path, MAXPATHLEN+1, dllpath[0] ? dllpath : progpath))
 /* exceeded buffer length - ignore zip_path */
@@ -590,16 +656,13 @@
 
 skiphome = pythonhome==NULL ? 0 : 1;
 #ifdef Py_ENABLE_SHARED
- if (!applocal) {
- machinepath = getpythonregpath(HKEY_LOCAL_MACHINE, skiphome);
- userpath = getpythonregpath(HKEY_CURRENT_USER, skiphome);
- }
+ machinepath = getpythonregpath(HKEY_LOCAL_MACHINE, skiphome);
+ userpath = getpythonregpath(HKEY_CURRENT_USER, skiphome);
 #endif
 /* We only use the default relative PYTHONPATH if we havent
 anything better to use! */
 skipdefault = envpath!=NULL || pythonhome!=NULL || \
 machinepath!=NULL || userpath!=NULL;
-#endif
 
 /* We need to construct a path from the following parts.
 (1) the PYTHONPATH environment variable, if set;
@@ -612,7 +675,6 @@
 Extra rules:
 - If PYTHONHOME is set (in any way) item (3) is ignored.
 - If registry values are used, (4) and (5) are ignored.
- - If applocal is set, (1), (3), and registry values are ignored
 */
 
 /* Calculate size of return buffer */
@@ -629,13 +691,11 @@
 bufsz = 0;
 bufsz += wcslen(PYTHONPATH) + 1;
 bufsz += wcslen(argv0_path) + 1;
-#ifdef MS_WINDOWS
- if (!applocal && userpath)
+ if (userpath)
 bufsz += wcslen(userpath) + 1;
- if (!applocal && machinepath)
+ if (machinepath)
 bufsz += wcslen(machinepath) + 1;
 bufsz += wcslen(zip_path) + 1;
-#endif
 if (envpath != NULL)
 bufsz += wcslen(envpath) + 1;
 
@@ -651,10 +711,8 @@
 fprintf(stderr, "Using default static path.\n");
 module_search_path = PYTHONPATH;
 }
-#ifdef MS_WINDOWS
 PyMem_RawFree(machinepath);
 PyMem_RawFree(userpath);
-#endif /* MS_WINDOWS */
 return;
 }
 
@@ -664,7 +722,6 @@
 buf = wcschr(buf, L'0円');
 *buf++ = DELIM;
 }
-#ifdef MS_WINDOWS
 if (zip_path[0]) {
 if (wcscpy_s(buf, bufsz - (buf - module_search_path), zip_path))
 Py_FatalError("buffer overflow in getpathp.c's calculate_path()");
@@ -692,15 +749,7 @@
 buf = wcschr(buf, L'0円');
 *buf++ = DELIM;
 }
- }
-#else
- if (pythonhome == NULL) {
- wcscpy(buf, PYTHONPATH);
- buf = wcschr(buf, L'0円');
- *buf++ = DELIM;
- }
-#endif /* MS_WINDOWS */
- else {
+ } else {
 wchar_t *p = PYTHONPATH;
 wchar_t *q;
 size_t n;
diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj
--- a/PCbuild/pythoncore.vcxproj
+++ b/PCbuild/pythoncore.vcxproj
@@ -69,7 +69,7 @@
 <PreprocessorDefinitions>_USRDLL;Py_BUILD_CORE;Py_ENABLE_SHARED;MS_DLL_ID="$(SysWinVer)";%(PreprocessorDefinitions)</PreprocessorDefinitions>
 </ClCompile>
 <Link>
- <AdditionalDependencies>ws2_32.lib;%(AdditionalDependencies)</AdditionalDependencies>
+ <AdditionalDependencies>shlwapi.lib;ws2_32.lib;%(AdditionalDependencies)</AdditionalDependencies>
 <BaseAddress>0x1e000000</BaseAddress>
 </Link>
 </ItemDefinitionGroup>
diff --git a/Tools/msi/make_zip.py b/Tools/msi/make_zip.py
--- a/Tools/msi/make_zip.py
+++ b/Tools/msi/make_zip.py
@@ -46,6 +46,10 @@
 'python3stub',
 }
 
+EXCLUDED_FILES = {
+ 'pyshellext',
+}
+
 def is_not_debug(p):
 if DEBUG_RE.search(p.name):
 return False
@@ -53,7 +57,7 @@
 if TKTCL_RE.search(p.name):
 return False
 
- return p.stem.lower() not in DEBUG_FILES
+ return p.stem.lower() not in DEBUG_FILES and p.stem.lower() not in EXCLUDED_FILES
 
 def is_not_debug_or_python(p):
 return is_not_debug(p) and not PYTHON_DLL_RE.search(p.name)
@@ -209,8 +213,9 @@
 copied = copy_to_layout(temp / t.rstrip('/'), rglob(s, p, c))
 print('Copied {} files'.format(copied))
 
- with open(str(temp / 'pyvenv.cfg'), 'w') as f:
- print('applocal = true', file=f)
+ with open(str(temp / 'sys.path'), 'w') as f:
+ print('python{0.major}{0.minor}.zip'.format(sys.version_info), file=f)
+ print('.', file=f)
 
 if out:
 total = copy_to_layout(out, rglob(temp, '**/*', None))
-- 
Repository URL: https://hg.python.org/cpython

---

• Previous message (by thread): [Python-checkins] cpython: Fixes expected error when getting encoding while shutting down.
• Next message (by thread): [Python-checkins] cpython: credit Raymond
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

---

More information about the Python-checkins mailing list