Pytest 6: Dynamically adding xfail marker in test no longer ignores failure

[rowan-stein]

User request

With pytest 5.x, we can dynamically add an xfail to a test request object using request.node.add_marker(mark) (see example below). In 5.x this treated the failing test like a test marked statically with an xfail. With 6.0.0rc0 it raises.

Example:

# file: test_foo.py
import pytest

def test_xfail_test(request):
    mark = pytest.mark.xfail(reason="xfail")
    request.node.add_marker(mark)
    assert 0

With 5.4.3:

$ pytest -rsx test_foo.py
... xfail output ...

With 6.0.0rc0:

$ pytest -rsx test_foo.py
... fails instead of xfail ...

Environment: Pytest 6.0.1rc0, macOS 10.14.5.

Research specification

• Regression introduced by commit c9737ae ("skipping: simplify xfail handling during call phase"). Early caching of xfail evaluation (in src/_pytest/skipping.py) prevents dynamically added xfail marks during the call phase from affecting the test outcome.
• Affected logic: pytest_runtest_setup, pytest_runtest_call (hookwrapper), pytest_runtest_makereport (hookwrapper) in src/_pytest/skipping.py; xfailed_key storage and evaluate_xfail_marks(item).
• Fix approach:
  • In pytest_runtest_setup: evaluate xfail; enforce xfail(run=False) if present; only store xfailed in item._store when not None (do not store None).
  • In pytest_runtest_call: remove pre-call reevaluation; only enforce NOTRUN if a precomputed xfailed exists and has run=False; do not evaluate when absent before the test runs.
  • In pytest_runtest_makereport: if xfailed is None or not present in the store, evaluate now before converting outcome, so xfail marks added during the call are respected.
• Acceptance criteria: a test dynamically adding xfail via request.node.add_marker within the test body yields XFAIL (not FAIL); xfail(run=False) declared prior to execution still prevents running the body; strict and raises behaviors remain unchanged; reporting unchanged except for correct XFAIL outcome.