Skipped test with the `skip` decorator reports test results as `unknown`

[gchan]

PyTest tests skipped with the @pytest.mark.skip decorator [1] appear to report the test result as unknown to Test Engine.

Tests dynamically skipped with xfail or pytest.skip are unaffected and report the correct result.

Here's a copy of the debug output

❯ BUILDKITE_ANALYTICS_DEBUG_ENABLED=1 pytest tests/test_skipped.py
========================================================================================================= test session starts ==========================================================================================================
platform darwin -- Python 3.13.2, pytest-8.4.1, pluggy-1.6.0
rootdir: ..../test-engine-client-examples/pytest
plugins: buildkite-test-collector-1.1.1, xdist-3.8.0
collected 1 item
buildkite-test-collector - DEBUG - hook=pytest_runtest_logstart nodeid=tests/test_skipped.py::test_skipped_example
buildkite-test-collector - DEBUG - -> started_at=Instant(seconds=65273.672679541)(monotonic)

tests/test_skipped.py buildkite-test-collector - DEBUG - hook=pytest_runtest_makereport nodeid=tests/test_skipped.py::test_skipped_example when=setup
buildkite-test-collector - DEBUG - hook=pytest_runtest_logreport nodeid=tests/test_skipped.py::test_skipped_example when=setup
sbuildkite-test-collector - DEBUG - hook=pytest_runtest_makereport nodeid=tests/test_skipped.py::test_skipped_example when=teardown
buildkite-test-collector - DEBUG - -> finalize_test nodeid=tests/test_skipped.py::test_skipped_example
buildkite-test-collector - DEBUG - -> finalize_test nodeid=tests/test_skipped.py::test_skipped_example duration=0:00:00.000319
buildkite-test-collector - DEBUG - hook=pytest_runtest_logreport nodeid=tests/test_skipped.py::test_skipped_example when=teardown
buildkite-test-collector - DEBUG - hook=pytest_runtest_logfinish nodeid=tests/test_skipped.py::test_skipped_example
buildkite-test-collector - DEBUG - -> finalize_test nodeid=tests/test_skipped.py::test_skipped_example
buildkite-test-collector - DEBUG - -> finalize_test: not in flight: tests/test_skipped.py::test_skipped_example

Internal reference: https://linear.app/buildkite/issue/TE-4941#comment-4bba1eaa

[1] https://docs.pytest.org/en/stable/how-to/skipping.html#skipping-test-functions

[gchan]

For a dynamically skipped tests, the call phase is still entered whereas a decorated skip does not. Here's the debug log for a dynamically skipped test where the call phase is invoked:

tests/test_dynamic_skip_and_xfail.py buildkite-test-collector - DEBUG - hook=pytest_runtest_makereport nodeid=tests/test_dynamic_skip_and_xfail.py::test_dynamic_skip when=setup
buildkite-test-collector - DEBUG - hook=pytest_runtest_logreport nodeid=tests/test_dynamic_skip_and_xfail.py::test_dynamic_skip when=setup
buildkite-test-collector - DEBUG - hook=pytest_runtest_makereport nodeid=tests/test_dynamic_skip_and_xfail.py::test_dynamic_skip when=call
buildkite-test-collector - DEBUG - hook=pytest_runtest_logreport nodeid=tests/test_dynamic_skip_and_xfail.py::test_dynamic_skip when=call
buildkite-test-collector - DEBUG - -> test skipped
sbuildkite-test-collector - DEBUG - hook=pytest_runtest_makereport nodeid=tests/test_dynamic_skip_and_xfail.py::test_dynamic_skip when=teardown
buildkite-test-collector - DEBUG - -> finalize_test nodeid=tests/test_dynamic_skip_and_xfail.py::test_dynamic_skip
buildkite-test-collector - DEBUG - -> finalize_test nodeid=tests/test_dynamic_skip_and_xfail.py::test_dynamic_skip duration=0:00:00.000454
buildkite-test-collector - DEBUG - hook=pytest_runtest_logreport nodeid=tests/test_dynamic_skip_and_xfail.py::test_dynamic_skip when=teardown
buildkite-test-collector - DEBUG - hook=pytest_runtest_logfinish nodeid=tests/test_dynamic_skip_and_xfail.py::test_dynamic_skip
buildkite-test-collector - DEBUG - -> finalize_test nodeid=tests/test_dynamic_skip_and_xfail.py::test_dynamic_skip
buildkite-test-collector - DEBUG - -> finalize_test: not in flight: tests/test_dynamic_skip_and_xfail.py::test_dynamic_skip

[gchan]

I wonder if we need to alter this line to allow result collection during the setup (and maybe teardown) phase if the report is skipped?

test-collector-python/src/buildkite_test_collector/pytest_plugin/buildkite_plugin.py

Line 50 in 484f325

if report.when == 'call' or (report.when == 'setup' and report.failed):