Why does 'python -m pip' runs the wrong pip when executed using subprocess.run() from a virtual environment?

Question

I'm tracking back a bug in our test scripts, where for some reason wrong pip is executed. I made a minimal script to reproduce it to make sure it's not the test script to blame. So my minimal script is:

import os
import subprocess
import sys
print(sys.executable)
print('system pip:')
os.system('pip -V')
print()
print('system python pip:')
os.system('python -m pip -V')
print()
print('system where python:')
os.system('where python')
print()
print('system VIRTUAL_ENV:')
os.system('cmd /c echo %VIRTUAL_ENV%')
print()
print('system PYTHONPATH:')
os.system('cmd /c echo %PYTHONPATH%')
print()
print('subprocess pip:')
subprocess.run(['pip', '-V'])
print()
print('subprocess python pip:')
subprocess.run(['python', '-m', 'pip', '-V'])
print()
print('subprocess where python:')
subprocess.run(['cmd', '/c', 'where', 'python'])
print()
print('subprocess sys.executable pip:')
subprocess.run([sys.executable, '-m', 'pip', '-V'])
print()
print('subprocess VIRTUAL_ENV:')
subprocess.run(['cmd', '/c', 'echo', '%VIRTUAL_ENV%'])
print()
print('subprocess PYTHONPATH:')
subprocess.run(['cmd', '/c', 'echo', '%PYTHONPATH%'])
print()

And when I run it with my venv active, it prints:

C:\Users\Calmarius\pythonvenv\myvenv\Scripts\python.exe
system pip:
pip 20.0.2 from c:\users\calmarius\pythonvenv\myvenv\lib\site-packages\pip (python 3.7)
system python pip:
pip 20.0.2 from C:\Users\Calmarius\pythonvenv\myvenv\lib\site-packages\pip (python 3.7)
system where python:
C:\Users\Calmarius\pythonvenv\myvenv\Scripts\python.exe
C:\Program Files\Python37\python.exe
C:\Users\Calmarius\AppData\Local\Microsoft\WindowsApps\python.exe
system VIRTUAL_ENV:
C:\Users\Calmarius\pythonvenv\myvenv
system PYTHONPATH:
%PYTHONPATH%
subprocess pip:
pip 20.0.2 from c:\users\calmarius\pythonvenv\myvenv\lib\site-packages\pip (python 3.7)
subprocess python pip:
pip 20.0.2 from C:\Program Files\Python37\lib\site-packages\pip (python 3.7)
subprocess where python:
C:\Users\Calmarius\pythonvenv\myvenv\Scripts\python.exe
C:\Program Files\Python37\python.exe
C:\Users\Calmarius\AppData\Local\Microsoft\WindowsApps\python.exe
subprocess sys.executable pip:
pip 20.0.2 from C:\Users\Calmarius\pythonvenv\myvenv\lib\site-packages\pip (python 3.7)
subprocess VIRTUAL_ENV:
C:\Users\Calmarius\pythonvenv\myvenv
subprocess PYTHONPATH:
%PYTHONPATH%

Basically it works as expected except the only one case when python -m pip is executed using subprocess, for some reason it executes the system's pip not venv one. Exactly the combination the test scripts do. Also my peers don't have problems running it. So there is something happened that is specific to my machine, but I don't where start.

Anyone know what can cause this?

• Data_Is_Everything: it has to be virtual environment path first once the virtual venv is activated then run the pip install command, I presume it's for module installation to run against your test scripts? you could try python -m pip install -r .\requirements.txt something on those lines.
• Calmarius: @FishingCode Yes the test script does exactly that, it even checks if VIRTUAL_ENV variable exists to force the user to run it from a venv. then automatically ensures all the dependencies and stuff are installed by issuing a python -m pip install -r .\requirements.txt using subprocess.run. Then the whole thing fails there because it doesn't have permission to write "Program Files". And I'm wonder why this happens.
• Data_Is_Everything: ok, I see, I think the issue is the virtual env is not activated so something like this: .\Scripts\activate after initiating that, then run the pip install cmd... otherwise you may require privileges to run program files i.e. trying sudo
• Calmarius: @FishingCode My virtual environment is active, as you can see from the printouts. Also the OS is Windows so no sudo there, and I don't want to elevate privileges, because the script must not escape the venv.
• Data_Is_Everything: actually that's just installing the venv you have to give command to activate also you can tell it's not activate as if it were the console should show something like this: (WindowsVenvServer) C: \home\projects\programmingprojects\files it's missing the myvenv which is your virtualenv at the beginning. Here WindowsVenvServer is the venv as an example.

Answer 1

It looks like this is a bug introduced in 3.7.3.

Possible workaround is using sys.executable when invoking another python process instead of relying on the OS to find the right path.