Skipped distributed tests show up as passed (return 0)

[carmocca]

          Note that this can be very misleading because a skipped test also returns 0, so it can make it seem like a test passed when it didn't run

Originally posted by @carmocca in #130 (comment)

lightning-thunder/scripts/run_standalone_tests.sh

Lines 41 to 48 in 5c86534

	python -um pytest -sv "$test" --pythonwarnings ignore --junitxml="$test-results.xml" 2>&1 > "$test-output.txt"
	pytest_status=$?
	printf "$test status >>> $pytest_status\n"
	if [ $pytest_status -ne 0 ]; then
	status=$pytest_status
	cat "$test-output.txt"
	fi

cc @Borda

[carmocca]

@t-vi adds:

Seems like Python 3.12 fixes that in unittest ( python/cpython#106584 ), so there probably is a canonical "what to do" but I am not sure if there is an easy way to do this for pytest, too.

[carmocca]

https://stackoverflow.com/questions/31538941/pytest-overall-result-pass-when-all-tests-are-skipped has some suggestions via pytest's conftest.py