Skip to content

Retried tests with pytest-rerunfailures report test results as unknown #78

@gchan

Description

@gchan

pytest-rerunfailures is a pytest plugin to retry/rerun test failures.

The test results for rerun tests is reported by the Python Test Collector as 'unknown' for both the failed execution and the passed execution.

Internal reference: https://linear.app/buildkite/issue/TE-4941/rippling-test-reliability-score-showing-na#comment-a1094f05

And here's the log output:

platform linux -- Python 3.13.2, pytest-8.4.2, pluggy-1.6.0
rootdir: /workdir/pytest-xdist
configfile: pytest.ini
plugins: repeat-0.9.4, xdist-3.8.0, buildkite-test-collector-1.1.1, rerunfailures-12.0
3 workers [3 items]     
scheduling tests via LoadScheduling
tests/test_dynamic_skip_and_xfail.py::test_dynamic_skip
tests/test_flaky.py::test_flaky
[gw1] [ 33%] SKIPPED tests/test_dynamic_skip_and_xfail.py::test_dynamic_skip
tests/test_dynamic_skip_and_xfail.py::test_xfail_example
[gw2] [ 66%] XFAIL tests/test_dynamic_skip_and_xfail.py::test_xfail_example
[gw0] [100%] RERUN tests/test_flaky.py::test_flaky
tests/test_flaky.py::test_flaky
[gw0] [100%] PASSED tests/test_flaky.py::test_flaky

======================================================= 1 passed, 1 skipped, 1 xfailed, 1 rerun in 0.38s =======================================================

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions