Test explorer doesn't show test when addopts have --ruff or --mypy

[thevoiddancer]

Type: Bug

Behaviour

Test detection finds all the tests correctly. But test explorer has some issues.

When using

[tool.pytest.ini_options]
addopts = "-rfxX --verbose --cov-report html --cov-report term-missing --cov=src"

everything works correctly

But when I add --ruff to it:

[tool.pytest.ini_options]
addopts = "-rfxX --ruff --verbose --cov-report html --cov-report term-missing --cov=src"

then test explorer doesn't show previously discovered tests and only stops at the first file in the test folder, for instance init.py

However, the tests are correctly detected and they can be run completely. However, as no tests are shown in test explorer, they can't be run individually. Including that run option is missing from the test file.

Steps to reproduce:

1. Set up a repo containing pytest and pytest-cov along with pyproject.toml and addopts in [tool.pytest.ini_options].
2. Check that all tests are present in test explorer.
3. Add ruff and pytest-ruff.
4. Verify that all tests are still present.
5. Add --ruff option in addopts.
6. Verify that tests are no longer shown in test explorer, and only first file appears.

Alternatively, copy this minimum reproducible example: https://github.com/thevoiddancer/min_example

VSCOde version:
Version: 1.87.2 (user setup)
Commit: 863d2581ecda6849923a2118d93a088b0745d9d6
Date: 2024-03-08T15:20:17.278Z
Electron: 27.3.2
ElectronBuildId: 26836302
Chromium: 118.0.5993.159
Node.js: 18.17.1
V8: 11.8.172.18-electron.0
OS: Windows_NT x64 10.0.19045

Diagnostic data

Python output without --ruff option

2024-08-09 16:53:30.172 [info] Discover tests for workspace name: min_repr_ex - uri: /home/naztom/tvd/min_repr_ex
2024-08-09 16:53:30.172 [info] Running discovery for pytest using the new test adapter.
2024-08-09 16:53:30.173 [info] All environment variables set for pytest discovery for workspace /home/naztom/tvd/min_repr_ex: {"USER":"naztom","VSCODE_WSL_EXT_LOCATION":"/mnt/c/Users/ttomnaz/.vscode/extensions/ms-vscode-remote.remote-wsl-0.86.0","SHLVL":"1","WT_PROFILE_ID":"{07b52e3e-de2c-5db4-bd2d-ba144ed6c273}","HOME":"/home/naztom","WSL_DISTRO_NAME":"Ubuntu-20.04","LOGNAME":"naztom","NAME":"Code","WSL_INTEROP":"/run/WSL/839_interop","_":"/mnt/c/Users/ttomnaz/AppData/Local/Programs/Microsoft VS Code/bin/code","TERM":"xterm-256color","WT_SESSION":"a10858b4-fdda-412f-b0f5-e2d09ba6e15b","LANG":"C.UTF-8","SHELL":"/bin/bash","PWD":"/mnt/c/Users/ttomnaz/AppData/Local/Programs/Microsoft VS Code","HOSTTYPE":"x86_64","WSLENV":"ELECTRON_RUN_AS_NODE/w:WT_SESSION:WT_PROFILE_ID:\n","VSCODE_HANDLES_SIGPIPE":"true","PYENV_SHELL":"bash","CONDA_EXE":"/home/naztom/miniconda3/bin/conda","_CE_M":"","PYENV_VIRTUALENV_INIT":"1","CONDA_PREFIX":"/home/naztom/miniconda3","LS_COLORS":"rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.zst=01;31:.tzst=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.wim=01;31:.swm=01;31:.dwm=01;31:.esd=01;31:.jpg=01;35:.jpeg=01;35:.mjpg=01;35:.mjpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.ogv=01;35:.ogx=01;35:.aac=00;36:.au=00;36:.flac=00;36:.m4a=00;36:.mid=00;36:.midi=00;36:.mka=00;36:.mp3=00;36:.mpc=00;36:.ogg=00;36:.ra=00;36:.wav=00;36:.oga=00;36:.opus=00;36:.spx=00;36:*.xspf=00;36:","CONDA_PROMPT_MODIFIER":"(base) ","LESSCLOSE":"/usr/bin/lesspipe %s %s","_CE_CONDA":"","LESSOPEN":"| /usr/bin/lesspipe %s","CONDA_SHLVL":"1","CONDA_PYTHON_EXE":"/home/naztom/miniconda3/bin/python","CONDA_DEFAULT_ENV":"base","PYENV_ROOT":"/home/naztom/.pyenv","XDG_DATA_DIRS":"/usr/local/share:/usr/share:/var/lib/snapd/desktop","VSCODE_AMD_ENTRYPOINT":"vs/workbench/api/node/extensionHostProcess","VSCODE_HANDLES_UNCAUGHT_ERRORS":"true","VSCODE_NLS_CONFIG":"{"locale":"en","osLocale":"en","availableLanguages":{}}","VSCODE_CWD":"/mnt/c/Users/ttomnaz/AppData/Local/Programs/Microsoft VS Code","ELECTRON_RUN_AS_NODE":"1","VSCODE_IPC_HOOK_CLI":"/tmp/vscode-ipc-ea4ab0a9-8a50-4875-9b68-160c55126674.sock","PATH":"/home/naztom/.vscode-server/bin/863d2581ecda6849923a2118d93a088b0745d9d6/bin/remote-cli:/home/naztom/.local/bin:/home/naztom/miniconda3/bin:/home/naztom/miniconda3/condabin:/home/naztom/.pyenv/plugins/pyenv-virtualenv/shims:/home/naztom/.pyenv/shims:/home/naztom/.pyenv/bin:/home/naztom/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0:/mnt/c/windows/System32/OpenSSH:/mnt/c/Users/ttomnaz/AppData/Local/Programs/Python/Python312/Scripts:/mnt/c/Users/ttomnaz/AppData/Local/Programs/Python/Python312:/mnt/c/Users/ttomnaz/AppData/Local/Programs/Python/Launcher:/mnt/c/Users/ttomnaz/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/ttomnaz/AppData/Local/Programs/Microsoft VS Code/bin:/snap/bin","PYTHONPATH":"/home/naztom/.vscode-server/extensions/ms-python.python-2024.4.1/python_files","TEST_UUID":"b2ea121f-6386-470a-a059-9ab283bb4eda","TEST_PORT":"36991"}

2024-08-09 16:53:30.199 [info] > conda run -n min_repr_ex --no-capture-output python ~/.vscode-server/extensions/ms-python.python-2024.4.1/python_files/get_output_via_markers.py -m pytest -p vscode_pytest --collect-only tests
2024-08-09 16:53:30.199 [info] cwd: .
2024-08-09 16:53:31.940 [info] >>>PYTHON-EXEC-OUTPUT

2024-08-09 16:53:32.233 [info] ============================= test session starts ==============================
platform linux -- Python 3.11.9, pytest-8.3.2, pluggy-1.5.0 -- /home/naztom/miniconda3/envs/min_repr_ex/bin/python
cachedir: .pytest_cache
rootdir: /home/naztom/tvd/min_repr_ex
configfile: pyproject.toml
plugins: cov-5.0.0, ruff-0.4.1
collecting ... collected 1 item

========================== 1 test collected in 0.01s ===========================
<<<PYTHON-EXEC-OUTPUT

2024-08-09 16:53:32.233 [info] Test server connected to a client.
2024-08-09 16:53:32.235 [info] ResultResolver EOT received for discovery.
2024-08-09 16:53:32.353 [info] Disposing data receiver for /home/naztom/tvd/min_repr_ex and deleting UUID; pytest discovery.

Python output with --ruff option

2024-08-09 16:52:35.830 [info] Discover tests for workspace name: min_repr_ex - uri: /home/naztom/tvd/min_repr_ex
2024-08-09 16:52:35.830 [info] Running discovery for pytest using the new test adapter.
2024-08-09 16:52:35.831 [info] All environment variables set for pytest discovery for workspace /home/naztom/tvd/min_repr_ex: {"USER":"naztom","VSCODE_WSL_EXT_LOCATION":"/mnt/c/Users/ttomnaz/.vscode/extensions/ms-vscode-remote.remote-wsl-0.86.0","SHLVL":"1","WT_PROFILE_ID":"{07b52e3e-de2c-5db4-bd2d-ba144ed6c273}","HOME":"/home/naztom","WSL_DISTRO_NAME":"Ubuntu-20.04","LOGNAME":"naztom","NAME":"Code","WSL_INTEROP":"/run/WSL/839_interop","_":"/mnt/c/Users/ttomnaz/AppData/Local/Programs/Microsoft VS Code/bin/code","TERM":"xterm-256color","WT_SESSION":"a10858b4-fdda-412f-b0f5-e2d09ba6e15b","LANG":"C.UTF-8","SHELL":"/bin/bash","PWD":"/mnt/c/Users/ttomnaz/AppData/Local/Programs/Microsoft VS Code","HOSTTYPE":"x86_64","WSLENV":"ELECTRON_RUN_AS_NODE/w:WT_SESSION:WT_PROFILE_ID:\n","VSCODE_HANDLES_SIGPIPE":"true","PYENV_SHELL":"bash","CONDA_EXE":"/home/naztom/miniconda3/bin/conda","_CE_M":"","PYENV_VIRTUALENV_INIT":"1","CONDA_PREFIX":"/home/naztom/miniconda3","LS_COLORS":"rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.zst=01;31:.tzst=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.wim=01;31:.swm=01;31:.dwm=01;31:.esd=01;31:.jpg=01;35:.jpeg=01;35:.mjpg=01;35:.mjpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.ogv=01;35:.ogx=01;35:.aac=00;36:.au=00;36:.flac=00;36:.m4a=00;36:.mid=00;36:.midi=00;36:.mka=00;36:.mp3=00;36:.mpc=00;36:.ogg=00;36:.ra=00;36:.wav=00;36:.oga=00;36:.opus=00;36:.spx=00;36:*.xspf=00;36:","CONDA_PROMPT_MODIFIER":"(base) ","LESSCLOSE":"/usr/bin/lesspipe %s %s","_CE_CONDA":"","LESSOPEN":"| /usr/bin/lesspipe %s","CONDA_SHLVL":"1","CONDA_PYTHON_EXE":"/home/naztom/miniconda3/bin/python","CONDA_DEFAULT_ENV":"base","PYENV_ROOT":"/home/naztom/.pyenv","XDG_DATA_DIRS":"/usr/local/share:/usr/share:/var/lib/snapd/desktop","VSCODE_AMD_ENTRYPOINT":"vs/workbench/api/node/extensionHostProcess","VSCODE_HANDLES_UNCAUGHT_ERRORS":"true","VSCODE_NLS_CONFIG":"{"locale":"en","osLocale":"en","availableLanguages":{}}","VSCODE_CWD":"/mnt/c/Users/ttomnaz/AppData/Local/Programs/Microsoft VS Code","ELECTRON_RUN_AS_NODE":"1","VSCODE_IPC_HOOK_CLI":"/tmp/vscode-ipc-ea4ab0a9-8a50-4875-9b68-160c55126674.sock","PATH":"/home/naztom/.vscode-server/bin/863d2581ecda6849923a2118d93a088b0745d9d6/bin/remote-cli:/home/naztom/.local/bin:/home/naztom/miniconda3/bin:/home/naztom/miniconda3/condabin:/home/naztom/.pyenv/plugins/pyenv-virtualenv/shims:/home/naztom/.pyenv/shims:/home/naztom/.pyenv/bin:/home/naztom/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0:/mnt/c/windows/System32/OpenSSH:/mnt/c/Users/ttomnaz/AppData/Local/Programs/Python/Python312/Scripts:/mnt/c/Users/ttomnaz/AppData/Local/Programs/Python/Python312:/mnt/c/Users/ttomnaz/AppData/Local/Programs/Python/Launcher:/mnt/c/Users/ttomnaz/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/ttomnaz/AppData/Local/Programs/Microsoft VS Code/bin:/snap/bin","PYTHONPATH":"/home/naztom/.vscode-server/extensions/ms-python.python-2024.4.1/python_files","TEST_UUID":"b39fa85b-f774-40dd-94d5-9921d67fae3e","TEST_PORT":"36991"}

2024-08-09 16:52:35.853 [info] > conda run -n min_repr_ex --no-capture-output python ~/.vscode-server/extensions/ms-python.python-2024.4.1/python_files/get_output_via_markers.py -m pytest -p vscode_pytest --collect-only tests
2024-08-09 16:52:35.854 [info] cwd: .
2024-08-09 16:52:37.027 [info] >>>PYTHON-EXEC-OUTPUT

2024-08-09 16:52:37.291 [info] ============================= test session starts ==============================
platform linux -- Python 3.11.9, pytest-8.3.2, pluggy-1.5.0 -- /home/naztom/miniconda3/envs/min_repr_ex/bin/python

2024-08-09 16:52:37.296 [info] cachedir: .pytest_cache
rootdir: /home/naztom/tvd/min_repr_ex
2024-08-09 16:52:37.296 [info]
configfile: pyproject.toml
plugins: cov-5.0.0, ruff-0.4.1
collecting ...
2024-08-09 16:52:37.305 [info] collected 3 items

2024-08-09 16:52:37.305 [info]

2024-08-09 16:52:37.307 [info] Test server connected to a client.
2024-08-09 16:52:37.308 [info] ResultResolver EOT received for discovery.
2024-08-09 16:52:37.309 [info]
========================== 3 tests collected in 0.02s ==========================
<<<PYTHON-EXEC-OUTPUT

2024-08-09 16:52:37.433 [info] Disposing data receiver for /home/naztom/tvd/min_repr_ex and deleting UUID; pytest discovery.

Extension version: 2024.4.1
VS Code version: Code 1.87.2 (863d2581ecda6849923a2118d93a088b0745d9d6, 2024-03-08T15:20:17.278Z)
OS version: Windows_NT x64 10.0.19045
Modes:
Remote OS version: Linux x64 5.10.102.1-microsoft-standard-WSL2

• Python version (& distribution if applicable, e.g. Anaconda): 3.11.9
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Conda
• Value of the python.languageServer setting: Default

User Settings

languageServer: "Pylance"

testing
• pytestArgs: "<placeholder>"
• pytestEnabled: true

Installed Extensions

Extension Name	Extension Id	Version
JavaScript Debugger	ms-vscode.js-debug	1.87.0
Pylance	ms-python.vscode-pylance	2024.7.1
Python	ms-python.python	2024.4.1
Python Debugger	ms-python.debugpy	2024.8.0
Table Visualizer for JavaScript Profiles	ms-vscode.vscode-js-profile-table	1.0.8

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-10610U CPU @ 1.80GHz (8 x 2304)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	undefined
Memory (System)	31.78GB (16.51GB free)
Process Argv	--folder-uri=vscode-remote://wsl+Ubuntu-20.04/home/naztom/tvd/min_repr_ex --remote=wsl+Ubuntu-20.04 --crash-reporter-id c68671e5-3452-4e7c-8a77-d296a81dc0bd
Screen Reader	no
VM	0%

Item	Value
Remote	WSL: Ubuntu-20.04
OS	Linux x64 5.10.102.1-microsoft-standard-WSL2
CPUs	Intel(R) Core(TM) i7-10610U CPU @ 1.80GHz (8 x 2304)
Memory (System)	24.88GB (24.01GB free)
VM	0%

A/B Experiments

vsliv368cf:30146710
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscoreces:30445986
vscod805cf:30301675
binariesv615:30325510
vsaa593:30376534
py29gd2263:31024239
c4g48928:30535728
azure-dev_surveyone:30548225
962ge761:30959799
pythongtdpath:30769146
welcomedialog:30910333
pythonnoceb:30805159
asynctok:30898717
pythonregdiag2:30936856
pythonmypyd1:30879173
h48ei257:31000450
pythontbext0:30879054
accentitlementsc:30995553
dsvsc016:30899300
dsvsc017:30899301
dsvsc018:30899302
cppperfnew:31000557
dsvsc020:30976470
pythonait:31006305
dsvsc021:30996838
724cj586:31013169
a69g1124:31058053
dvdeprecation:31068756
dwnewjupyter:31046869
2f103344:31071589
impr_priority:31102340
nativerepl1:31104043
refactort:31108082
pythonrstrctxt:31112756
wkspc-onlycs-c:31111717

[dalonsoa]

I can confirm this behaviour with the mypy flag. Haven't check ruff.

Here's my configuration:

VS Code

Version: 1.93.1 (system setup)
Commit: 38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40
Date: 2024-09-11T17:20:05.685Z
Electron: 30.4.0
ElectronBuildId: 10073054
Chromium: 124.0.6367.243
Node.js: 20.15.1
V8: 12.4.254.20-electron.0
OS: Windows_NT x64 10.0.22631

Python Extension

Id: ms-python.python
Version: 2024.16.0

Python

version: 3.12.4

Mypy

version: 1.11.2

Pytest

version: 8.3.3

pytest-mypy plugin

version: 0.10.3

[eleanorjboyd]

sorry for the delay- will look into this! Thanks for the detailed example

[eleanorjboyd]

ok so what happens when the --ruff is include is another pytest item is added to the session when running called a RuffItem. A RuffItem is a child of a RuffFile and is different than the regular pytest types. This means two file nodes get created per file- one of them RuffFile and one a regular TestFile node and they cause a collision in the tree making neither show up.

In order to make the ruff run executable from the test explorer I am going to add another node under each file which is running the ruff test for the file. Does this make sense / work for your usecase?

Exploring executing this scenario and have hit a bit of a blocker with the type recognition and attempting to convert ruffFiles to regular pytest file objects so Ill keep working but thats the explanation

[eleanorjboyd]

opened this discussion too with a question I have on the pytest-ruff repo: businho/pytest-ruff#32

[eleanorjboyd]

started discussion for pytest-mypy here