venv python reports wrong sys.executable in a subprocess on Windows #83086
Comments
|
To reproduce: > py -m venv fooenv
> fooenv\Scripts\activate.bat
(fooenv) > python -c "import sys; print(sys.executable)" % This is correct
C:\Users\uranusjr\Downloads\venvtest\Scripts\python.exe
(fooenv) > python -q
>>> import subprocess
>>> subprocess.check_output(['python', '-c', 'import sys; print(sys.executable)'])
b'C:\\Users\\uranusjr\\AppData\\Local\\Programs\\Python\\Python37\\python.exe\r\n'The output shows the base interpreter, not the interpreter in venv. Not sure whether this is a venv or subprocess problem. |
uranusjr
mannequin
added
stdlib
|
Linux works correctly (Ubuntu with self-compiled Python 3.7.5) $ python3.7 -m venv fooenv
$ . fooenv/bin/activate
(fooenv) $ python -c "import sys; print(sys.executable)"
/home/uranusjr/fooenv/bin/python
(fooenv) $ python -q
>>> import subprocess
>>> subprocess.check_output(['python', '-c', 'import sys; print(sys.executable)'])
b'/home/uranusjr/fooenv/bin/python\n' |
|
3.6 works correctly on Windows: > py -3.6 -m venv test36
> test36\Scripts\activate.bat
>>> import subprocess
>>> print(subprocess.check_output(['python', '-c', 'import sys; print(sys.executable)']))
b'C:\\Users\\uranusjr\\Downloads\\test36\\Scripts\\python.exe\r\n'So it seems the problem is introduced sometime after. |
|
Your failing test case with 3.7 works for me. If you don't use activate.bat, but just run the venv's python directly, what do you see? I get:
>fooenv\Scripts\python -q
>>> import subprocess
>>> subprocess.check_output(['python', '-c', 'import sys; print(sys.executable)'])
b'C:\\Users\\XXXXXXX\\fooenv\\Scripts\\python.exe\r\n'What shell are you using? Above is with cmd.exe. If you "echo %PATH%" after activate.bat, what do you see? Before running activate.bat, do you have a python.exe in your path? If so, is it the one that subprocess is reporting? |
I get the same result as activating (i.e. shows the base interpeter). All results in cmd.exe as well.
PATH is as expected, the venv’s Scripts directory at the front after activation. I (only) have a python.exe from Windows Store in PATH. The one reported by subprocess is not in PATH. I’ll try to find a clean machine (maybe a VM) and try whether I can replicate this. BTW the problematic versions for me was 3.7.5 and 3.8.0. |
|
The behaviour in this area is different between 3.7.0, 3.7.2, and 3.7.3 (at least). I have reproduced the issue with 3.7.3. Steve Dower made changes to the way the python executable works in venvs in the point releases of 3.7 - see pypa/virtualenv#1380 and pypa/virtualenv#1339 for some discussion of how this affected virtualenv. I suspect this issue is related - from 3.7.2 onwards, the python.exe in a venv is a redirector which runs the "base" python.exe, but with sys.executable showing the redirector rather than the actual running exe. |
|
I tested the following in various versions (all 64-bit) in a VM. All installations are 64-bit per-user.
3.8.0: Incorrect So the change seems to have happened somewhere between 3.7.2 and 3.7.3. Does this timeline line up with the venv redirector change? |
|
Yes, it does. I think we'd need input from Steve Dower here, as these changes were made (I believe) in support of the Windows Store build of Python, so any changes would need to be considered in the light of how they would affect that. I do, however, consider this to be a regression that should be fixed. BTW, just for completeness,
works as I'd expect, and that's the idiom that is often used. So relying on a path search to find the correct Python can be considered an unusual case (but nevertheless one I'd expect to be fixed). I assume that the issue here is that the code is being run by the python.dll in the base environment, and whens searching for executables, Windows gives "exes that are in the same directory as the currently executing code" priority over PATH. See https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessw, specifically """
|
That subprocess lets CreateProcessW use a platform-dependent search that prioritizes the application directory has come up in previous issues. To avoid this, we'd have to implement our own search for the given or parsed executable name. Then pass the fully-qualified executable path as the lpApplicationName name of CreateProcessW. This is how CMD works, since it has its own search routine that incorporates the PATHEXT environment variable. Because the application directory is searched before the working directory (if the working directory is searched at all, depending on context), this issue also affects searching for executable paths that contain a path separator. In Unix a relative path that contains a path separator is always relative to the working directory, but Windows CreateProcessW uses a normal search for a relative name unless it explicitly references the working directory as "." (e.g. ".\Scripts\pip.exe" instead of "Scripts\pip.exe"). |
|
I presume there's also the option of setting up the environment (or however it's done now - I know the details changed as the feature was developed) so that the "base" python.exe pretends to be the venv one, exactly as the wrapper does. However, that may well have other difficult-to-fix implications, not least that calling the base Python using an explicit full path should act like the base Python, and *not* like the venv one. IMO, the key thing here is that either the various limitations/quirks of redirecting to the base Python should either be treated as bugs, or they should be documented (even if only in the form of explicitly saying not to rely on any specific behaviour - e.g. "running an unqualified python and expecting a PATH search to pick up the same executable as the parent shell would is not supported and may produce unexpected results"). Virtual environments are a key part of most Python development workflows, and virtualenv is in the process of switching to use the core venv module internally. When that happens, there will be a lot more visibility for unexpected behaviours like this one. |
CreateProcessW finds "python.exe" in __APPDIR__ before it even searches PATH. I expect that some scripts depend on this when python.exe isn't in PATH, or when a different version is in PATH. If subprocess implements its own search, it can continue to prioritize the *effective* application directory, from dirname(sys.executable). |
|
Yeah, this definitely relates to how Windows handles unqualified argv[0] in CreateProcess. Though I thought we checked that out in another issue and decided that "most" people are correctly using sys.executable here? (Or decided that it was documented well enough that they should, and using a "python" literal was relying on OS behaviour.) I'm not a fan of trying to override the OS handling in subprocess, though that would be the fix. Possibly we could handle "python[.exe]" literals by substituting sys.executable transparently? |
|
To provide concrete context, the problem I’m facing is with how Flit resolves Generally the setup is to install Flit into a globally available location (let’s name it env A), so it’s usable for every project and environment. For a project foo you’d have a virtual environment (env X) that’s created from a base interpreter (env B, which may or may not be the same as env A). So the comment workflow would look like this:
This results in the following subprocess call: subprocess.check_output( And ideally (pre-3.7.2 Windows, or current POSIX behaviour) this would give you the absolute path to pythonX. But right now on Windows the result is pythonB. So if this is to be determined as acceptable behaviour, we’d need to come up with a suggestion how this code can be rewritten. |
This would be fine if you still run the process to get its sys.executable. Your specific example would never have worked, FWIW, as it always would have picked up pythonA rather than the application one or the base one, unless you were relying on python3/python3.7 not being available on Windows (which is no longer true - they are included in the Store package now). |
Perhaps generalize this as splitext(basename(sys.executable))[0] in order to support names other than "python" and "pythonw". It would have to handle quoting enough to ignore an initial double quote when parsing the executable name out of an args string, or _winapi could wrap the shell's CommandLineToArgvW function to handle this. |
This is where we've hit the point of complexity that it would have to be added as a new API. And since it's now opt-in, we may as well document that shutil.which() is the recommended way to resolve a filename against PATH (and make that work properly... ISTR it has some inconsistencies with the OS). |
But then I need two separate workflows based on what is passed in. For py.exe I need to run it and get sys.executable. But for python.exe I cannot use sys.executable because that’s the base interepeter, not the venv path I want. And
It is an illustration. I am fully aware of Windows not having version-named python executables. Thanks for the reminder anyway. |
If you use shutil.which() to resolve "python" or "py" against PATH (which doesn't include the application directory), then you'll get the full path to the correct python.exe and will get the expected sys.executable. So it's only one flow that works for both once you add the shutil.which step. |
|
Is there any update/solution for this issue? This issue is the root cause of this SO post: https://stackoverflow.com/q/61290972/2506522 |
|
I posted a workaround right above your post for when you want to resolve executables against PATH instead of using Windows's normal rules. I'm not sure we reached any good approach for launching sys.executable in a venv and automatically bypassing the redirector. |
|
encountered what I believe is the same problem, so here are some more details I noticed, after checking what exactly that process name lookup does using ProcessMonitor:
|
|
Thanks for confirming that reality aligns with the documentation.
Correct, because a fully qualified path to cmd.exe is used, and cmd.exe's application folder is not Python's. Ultimately, the above behaviour could have occurred anyway, so all that has really happened is that it is now more obvious. This is what always happens when you include support for other platforms - you find out which of your previous assumptions were incorrect (same principle applies to all of life if you think about it). If someone wants to update subprocess to manually search PATH when just provided with a filename, I'm okay with that. It would be a new enhancement though. For earlier versions, updating the docs to clarify that subprocess does not do a PATH search but relies on operating system behaviour for relative paths is probably the most helpful way to raise awareness. |
|
Hello, I've just run into a related issue. I have a python script, which starts an another python script using subprocess.Popen(). The parent script gets pid of the child and monitors up its activity through a database by this pid. The child script updates its activity in the database using the pid it gotten from os.getpid() Both scripts live in a virtual environment. It worked fine in Python 3.5 and stopped working after migration to Python 3.8. My base Python location: D:\Python\Python38\pythonw.exe I have realized, that when I run the following from a command prompt: 2 processes with the different PIDs are created: PID: 85548 It is definitely a regression, and will potentially break a lot of applications expecting a child Python process to be a direct descendant of its parent. Also it is a waste of system resources |
|
Another victim of this change in |
|
The fact that now there is a redirector process seems to me like a regression issue. Now this breaks because the subprocess pid is the one of the redirector and not the actual one. This is similar to 'Ray' issue mentioned earlier. |
|
In some cases, the problem can be worked around by setting the __PYVENV_LAUNCHER__ environment variable and executing the base executable. For example: import os
import sys
import subprocess
def get_python_exe_env():
if sys.executable == getattr(sys, '_base_executable', sys.executable):
return sys.executable, None
environ = os.environ.copy()
environ['__PYVENV_LAUNCHER__'] = sys.executable
return sys._base_executable, environ--- As is, I don't know how to solve the problem in which an arbitrary application runs a script using the "python" launcher. Half of the problem could be solved. When the launcher creates the base Python process, it could set the parent via PROC_THREAD_ATTRIBUTE_PARENT_PROCESS [1], presuming it can open a handle for the parent application with PROCESS_CREATE_PROCESS access. However, this doesn't solve the problem from the application's perspective. It still gets the handle and process ID of the launcher. If the purpose of using a launcher was only to allow an in-place upgrade of the base "python3x.dll", then an alternative design would be to set the __PYVENV_LAUNCHER__ environment variable, load the Python DLL from the pyvenv.cfg "home" directory, and call Py_Main(argc, argv). This would eliminate the parent<->child problem and the application directory (APPDIR) problem. However, naively implementing something like that cannot work for the store app distribution. "python3x.dll" in the app installation directory under "%ProgramFiles%\WindowsApps" only grants execute access to users that have the app's WIN://SYSAPPID identifier in their access token, which gets added by CreateProcessW() when the "python[3[.x]].exe" appexec link is executed. I'd have to experiment to see what works. Maybe "python.exe" in the virtual environment could be created as an appexec link to "venvlauncher.exe" in the app, which loads the DLL, etc. --- |
|
The main motivator for the redirector was to support the Store package, which can only be *executed* from outside its container (we can't load the DLL directly). Given we want the two distributions to be interchangeable from a code POV, it made the most sense to switch both over. That said, there were other issues with the previous system that were worth fixing, it just happens that a new one was introduced (though it definitely already existed for some users - IPC is a complex pattern to get right even when you fully control the processes). I still haven't been able to come up with a viable workaround that doesn't break more cases than it helps. Passing the internal environment variable around might help, but I don't want to fully support it because then that would prevent us fixing this properly! PEP-582 (or otherwise getting away from virtual environments in favour of some other way of having project-specific package installation) may be our best hope. |
The idea I was pondering was to create "python.exe" in the virtual environment as an appexec link to "C:\Program Files\WindowsApps\venvlauncher.exe" and reverse engineer whatever is needed to make CreateProcessW() build the required access token. For example, if it simply checks for the existence of "venvlauncher.exe" in the user's "WindowsApps" directory, then set that as an appexec alias as well. However, it can't work for a simple reason. For an appexec link, CreateProcessW() reads and executes the target of the link. The original appexec link path is not retained, so there's no way for the "venvlauncher.exe" process to know about the virtual environment.
"__pypackages__" doesn't seem like a drop-in replacement for virtual environments. The current directory (in the REPL) and script directory have always had precedence over the standard library, except in isolated mode, so "__pypackages__" seems more about providing a directory for Python packages that's neatly separated from the script directory. I presume also that "__pypackages__" would be added to sys.path even in isolated mode, though the PEP makes no mention of it. |
The above was supposed to be "C:\Program Files\WindowsApps\Python...\venvlauncher.exe", where "Python..." is the elided name of the package directory. |
If it was a drop-in replacement, we'd have dropped it in already. All |
|
It seems to me This is from a PR to ray: |
|
Unfortunately, getting the PID from the process directly rather than through a backchannel is simply not compatible with the other scenarios that venv has to support. We need the redirector process, which means you can't launch it and get the actual PID directly (this is one of the issues I'd hoped to solve by moving to a PEP 582-style world). Simple waits against the redirector will work fine, but for anything else you need the child process to print out its PID for the host, or use some other mechanism to communicate (named pipes, shared memory, etc.). |
|
Is there any updates on this @zooba? |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: