-
Notifications
You must be signed in to change notification settings - Fork 0
Description
Problem
SPIDER currently uses SciATH as its test framework. SciATH is a custom testing tool developed for PETSc-based applications, but it has several practical problems:
- Not pip-installable: SciATH is distributed as a git submodule, not a standard Python package. This makes installation fragile and adds a dependency that is not managed by any package manager.
- Not installed on most developer machines: Most PROTEUS developers do not have SciATH set up, so
make testfails silently. The validation tests added in PR Add external mesh input and EOS out-of-range safety #5 are standalone Python scripts precisely because SciATH was unavailable. - Inconsistent with the ecosystem: Every other component in the PROTEUS ecosystem (PROTEUS, Zalmoxis, CALLIOPE, JANUS, MORS, ZEPHYRUS, Aragog) uses pytest. Having SPIDER use a different framework creates friction for contributors who work across multiple repos.
- Limited CI integration: pytest has mature GitHub Actions integration, coverage reporting, and parallel execution support. SciATH requires custom CI configuration.
- No marker system: pytest markers (
@pytest.mark.unit,@pytest.mark.smoke,@pytest.mark.slow) allow selective test execution that matches the PROTEUS CI pipeline. SciATH does not have an equivalent.
Proposed approach
-
Write pytest wrappers for existing SPIDER tests: Each existing SciATH test case becomes a pytest function that runs the SPIDER binary as a subprocess (via
subprocess.run) and validates the output. This is the same pattern used in PROTEUS for SPIDER integration tests. -
Port the validation scripts from PR Add external mesh input and EOS out-of-range safety #5: The standalone scripts (
generate_aw_mesh.py,generate_super_earth_mesh.py,validate_mesh_fields.py) become proper pytest test functions with assertions and markers. -
Add standard markers:
@pytest.mark.smokefor quick binary-runs-without-crash tests,@pytest.mark.unitfor output validation against expected values,@pytest.mark.slowfor full-evolution tests. -
Remove SciATH submodule: Once all tests are ported, remove the SciATH git submodule and its configuration files.
-
Add
pyproject.toml(or extend existing one) with pytest configuration, matching the PROTEUS ecosystem conventions.
Benefits
- Contributors can run
pytestin the SPIDER directory and get immediate feedback. - CI can use the same
pytest -m smokepattern as the rest of the ecosystem. - Coverage reporting becomes possible via
pytest-cov. - Test discovery and parameterisation work out of the box.
Notes
- SPIDER is a C codebase, so the pytest tests would be "integration-style" — they invoke the compiled binary and check output files/exit codes. This is appropriate and matches how PROTEUS already tests SPIDER.
- The existing
tests/opts/*.optsoption files and expected output files can be reused as-is.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status