Motivation
The benchmarks repository currently has no releases or tags. Tagged releases serve multiple purposes:
- Pin methodology versions: The
METHODOLOGY.md already has a changelog policy, but there is no corresponding versioned snapshot to reference.
- Stable references: Dependents and consumers of benchmark results need a way to reference a known-good state.
- Project maturity signal: Releases indicate an actively maintained project and provide clear milestones.
Goal
Define a semver-based release strategy starting from v0.1.0 on the current stable state.
Requirements
- Initial tag: Create
v0.1.0 on current main once this spec is approved.
- Version bump semantics: Define what constitutes each bump level:
- Major: Breaking methodology changes, incompatible fixture contract changes.
- Minor: New benchmark targets, new report dimensions, additive fixture changes.
- Patch: Bug fixes, script improvements, documentation corrections.
- CHANGELOG policy: Decide between
CHANGELOG.md (in-repo) vs GitHub Release notes (auto-generated or hand-written).
- CI automation: Evaluate goreleaser or a manual GitHub Actions workflow for tag-triggered releases.
- Alignment with METHODOLOGY.md: Ensure methodology changelog entries reference the corresponding release version.
Open questions
- Should releases include generated artifacts (report.md, summary.json) as release assets?
- Should methodology version and release version be kept in sync or tracked independently?