Skip to content

Replace SciATH test framework with pytest #7

@timlichtenberg

Description

@timlichtenberg

Problem

SPIDER currently uses SciATH as its test framework. SciATH is a custom testing tool developed for PETSc-based applications, but it has several practical problems:

  • Not pip-installable: SciATH is distributed as a git submodule, not a standard Python package. This makes installation fragile and adds a dependency that is not managed by any package manager.
  • Not installed on most developer machines: Most PROTEUS developers do not have SciATH set up, so make test fails silently. The validation tests added in PR Add external mesh input and EOS out-of-range safety #5 are standalone Python scripts precisely because SciATH was unavailable.
  • Inconsistent with the ecosystem: Every other component in the PROTEUS ecosystem (PROTEUS, Zalmoxis, CALLIOPE, JANUS, MORS, ZEPHYRUS, Aragog) uses pytest. Having SPIDER use a different framework creates friction for contributors who work across multiple repos.
  • Limited CI integration: pytest has mature GitHub Actions integration, coverage reporting, and parallel execution support. SciATH requires custom CI configuration.
  • No marker system: pytest markers (@pytest.mark.unit, @pytest.mark.smoke, @pytest.mark.slow) allow selective test execution that matches the PROTEUS CI pipeline. SciATH does not have an equivalent.

Proposed approach

  1. Write pytest wrappers for existing SPIDER tests: Each existing SciATH test case becomes a pytest function that runs the SPIDER binary as a subprocess (via subprocess.run) and validates the output. This is the same pattern used in PROTEUS for SPIDER integration tests.

  2. Port the validation scripts from PR Add external mesh input and EOS out-of-range safety #5: The standalone scripts (generate_aw_mesh.py, generate_super_earth_mesh.py, validate_mesh_fields.py) become proper pytest test functions with assertions and markers.

  3. Add standard markers: @pytest.mark.smoke for quick binary-runs-without-crash tests, @pytest.mark.unit for output validation against expected values, @pytest.mark.slow for full-evolution tests.

  4. Remove SciATH submodule: Once all tests are ported, remove the SciATH git submodule and its configuration files.

  5. Add pyproject.toml (or extend existing one) with pytest configuration, matching the PROTEUS ecosystem conventions.

Benefits

  • Contributors can run pytest in the SPIDER directory and get immediate feedback.
  • CI can use the same pytest -m smoke pattern as the rest of the ecosystem.
  • Coverage reporting becomes possible via pytest-cov.
  • Test discovery and parameterisation work out of the box.

Notes

  • SPIDER is a C codebase, so the pytest tests would be "integration-style" — they invoke the compiled binary and check output files/exit codes. This is appropriate and matches how PROTEUS already tests SPIDER.
  • The existing tests/opts/*.opts option files and expected output files can be reused as-is.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    Status

    TBD

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions