venv python reports wrong sys.executable in a subprocess on Windows

[uranusjr]

BPO	38905
Nosy	@pfmoore, @ericvsmith, @tjguk, @zware, @eryksun, @zooba, @uranusjr, @nsmcan, @nirvana-msu

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-11-24.08:17:40.212>
labels = ['3.10', 'OS-windows']
title = 'venv python reports wrong sys.executable in a subprocess on Windows'
updated_at = <Date 2021-03-29.22:53:56.136>
user = 'https://github.com/uranusjr'

bugs.python.org fields:

activity = <Date 2021-03-29.22:53:56.136>
actor = 'steve.dower'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2019-11-24.08:17:40.212>
creator = 'uranusjr'
dependencies = []
files = []
hgrepos = []
issue_num = 38905
keywords = []
message_count = 30.0
messages = ['357392', '357393', '357394', '357421', '357425', '357428', '357429', '357430', '357437', '357439', '357440', '357453', '357456', '357457', '357459', '357460', '357479', '357519', '366755', '366824', '370656', '371357', '378284', '385923', '389321', '389345', '389741', '389760', '389761', '389768']
nosy_count = 12.0
nosy_names = ['paul.moore', 'eric.smith', 'tim.golden', 'Jurko.Gospodneti\xc4\x87', 'zach.ware', 'eryksun', 'steve.dower', 'uranusjr', 'Benedek R\xc3\xa1cz', 'nsmcan', 'nirvana-msu', 'awaizman']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue38905'
versions = ['Python 3.10']

[uranusjr]

To reproduce:

> py -m venv fooenv
> fooenv\Scripts\activate.bat
(fooenv) > python -c "import sys; print(sys.executable)"  % This is correct
C:\Users\uranusjr\Downloads\venvtest\Scripts\python.exe
(fooenv) > python -q
>>> import subprocess
>>> subprocess.check_output(['python', '-c', 'import sys; print(sys.executable)'])
b'C:\\Users\\uranusjr\\AppData\\Local\\Programs\\Python\\Python37\\python.exe\r\n'

The output shows the base interpreter, not the interpreter in venv. Not sure whether this is a venv or subprocess problem.

[uranusjr]

Linux works correctly (Ubuntu with self-compiled Python 3.7.5)

$ python3.7 -m venv fooenv
$ . fooenv/bin/activate
(fooenv) $ python -c "import sys; print(sys.executable)"
/home/uranusjr/fooenv/bin/python
(fooenv) $ python -q
>>> import subprocess
>>> subprocess.check_output(['python', '-c', 'import sys; print(sys.executable)'])
b'/home/uranusjr/fooenv/bin/python\n'

[uranusjr]

3.6 works correctly on Windows:

> py -3.6 -m venv test36
> test36\Scripts\activate.bat
>>> import subprocess
>>> print(subprocess.check_output(['python', '-c', 'import sys; print(sys.executable)']))
b'C:\\Users\\uranusjr\\Downloads\\test36\\Scripts\\python.exe\r\n'

So it seems the problem is introduced sometime after.

[ericvsmith]

Your failing test case with 3.7 works for me.

If you don't use activate.bat, but just run the venv's python directly, what do you see? I get:

py -m venv fooenv

fooenv\Scripts\python -V
Python 3.7.0

>fooenv\Scripts\python -q
>>> import subprocess
>>> subprocess.check_output(['python', '-c', 'import sys; print(sys.executable)'])
b'C:\\Users\\XXXXXXX\\fooenv\\Scripts\\python.exe\r\n'

What shell are you using? Above is with cmd.exe.

If you "echo %PATH%" after activate.bat, what do you see?

Before running activate.bat, do you have a python.exe in your path? If so, is it the one that subprocess is reporting?

[uranusjr]

If you don't use activate.bat, but just run the venv's python directly, what do you see? I get:
What shell are you using? Above is with cmd.exe.

I get the same result as activating (i.e. shows the base interpeter). All results in cmd.exe as well.

If you "echo %PATH%" after activate.bat, what do you see?
Before running activate.bat, do you have a python.exe in your path? If so, is it the one that subprocess is reporting?

PATH is as expected, the venv’s Scripts directory at the front after activation. I (only) have a python.exe from Windows Store in PATH. The one reported by subprocess is not in PATH.

I’ll try to find a clean machine (maybe a VM) and try whether I can replicate this. BTW the problematic versions for me was 3.7.5 and 3.8.0.

[pfmoore]

The behaviour in this area is different between 3.7.0, 3.7.2, and 3.7.3 (at least). I have reproduced the issue with 3.7.3. Steve Dower made changes to the way the python executable works in venvs in the point releases of 3.7 - see pypa/virtualenv#1380 and pypa/virtualenv#1339 for some discussion of how this affected virtualenv.

I suspect this issue is related - from 3.7.2 onwards, the python.exe in a venv is a redirector which runs the "base" python.exe, but with sys.executable showing the redirector rather than the actual running exe.

[uranusjr]

I tested the following in various versions (all 64-bit) in a VM. All installations are 64-bit per-user.

py -m venv testenv
testenv\Scripts\python.exe -c "import subprocess; print(subprocess.check_output(['python', '-c', 'import sys; print(sys.executable)']))"

3.8.0: Incorrect
3.7.5: Incorrect
3.7.4: Incorrect
3.7.3: Incorrect
3.7.2: Correct
3.6.8: Correct
3.7.1: Correct
3.7.0: Correct

So the change seems to have happened somewhere between 3.7.2 and 3.7.3. Does this timeline line up with the venv redirector change?

[pfmoore]

Yes, it does.

I think we'd need input from Steve Dower here, as these changes were made (I believe) in support of the Windows Store build of Python, so any changes would need to be considered in the light of how they would affect that. I do, however, consider this to be a regression that should be fixed.

BTW, just for completeness,

>> subprocess.check_output([sys.executable, '-c', 'import sys; print(sys.executable)'])

works as I'd expect, and that's the idiom that is often used. So relying on a path search to find the correct Python can be considered an unusual case (but nevertheless one I'd expect to be fixed).

I assume that the issue here is that the code is being run by the python.dll in the base environment, and whens searching for executables, Windows gives "exes that are in the same directory as the currently executing code" priority over PATH. See https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessw, specifically

"""
If the file name does not contain a directory path, the system searches for the executable file in the following sequence:

1. The directory from which the application loaded.
2. The current directory for the parent process.
3. The 32-bit Windows system directory. Use the GetSystemDirectory function to get the path of this directory.
4. The 16-bit Windows system directory. There is no function that obtains the path of this directory, but it is searched. The name of this directory is System.
5. The Windows directory. Use the GetWindowsDirectory function to get the path of this directory.
6. The directories that are listed in the PATH environment variable. Note that this function does not search the per-application path specified by the App Paths registry key. To include this per-application path in the search sequence, use the ShellExecute function.
   """

[eryksun]

whens searching for executables, Windows gives "exes that are in the
same directory as the currently executing code" priority over PATH.

That subprocess lets CreateProcessW use a platform-dependent search that prioritizes the application directory has come up in previous issues. To avoid this, we'd have to implement our own search for the given or parsed executable name. Then pass the fully-qualified executable path as the lpApplicationName name of CreateProcessW. This is how CMD works, since it has its own search routine that incorporates the PATHEXT environment variable.

Because the application directory is searched before the working directory (if the working directory is searched at all, depending on context), this issue also affects searching for executable paths that contain a path separator. In Unix a relative path that contains a path separator is always relative to the working directory, but Windows CreateProcessW uses a normal search for a relative name unless it explicitly references the working directory as "." (e.g. ".\Scripts\pip.exe" instead of "Scripts\pip.exe").

[pfmoore]

I presume there's also the option of setting up the environment (or however it's done now - I know the details changed as the feature was developed) so that the "base" python.exe pretends to be the venv one, exactly as the wrapper does.

However, that may well have other difficult-to-fix implications, not least that calling the base Python using an explicit full path should act like the base Python, and *not* like the venv one.

IMO, the key thing here is that either the various limitations/quirks of redirecting to the base Python should either be treated as bugs, or they should be documented (even if only in the form of explicitly saying not to rely on any specific behaviour - e.g. "running an unqualified python and expecting a PATH search to pick up the same executable as the parent shell would is not supported and may produce unexpected results").

Virtual environments are a key part of most Python development workflows, and virtualenv is in the process of switching to use the core venv module internally. When that happens, there will be a lot more visibility for unexpected behaviours like this one.

[eryksun]

"running an unqualified python and expecting a PATH search to pick up
the same executable as the parent shell would is not supported and may
produce unexpected results"

CreateProcessW finds "python.exe" in __APPDIR__ before it even searches PATH. I expect that some scripts depend on this when python.exe isn't in PATH, or when a different version is in PATH. If subprocess implements its own search, it can continue to prioritize the *effective* application directory, from dirname(sys.executable).