Add QA subprocess loop and script-side template assembly

[nick-inkeep]

Summary

Gives /qa the same subprocess iteration architecture that /implement uses: structured state file (qa.json), iteration prompt template, shell loop (qa.sh) that spawns fresh Claude Code subprocesses per scenario. Also moves template assembly out of the orchestrator and into the shell scripts themselves — both implement.sh and qa.sh now read their template, select the correct variant, compute a fresh diff, extract context from JSON, and fill all placeholders each iteration.

Architectural decisions

QA subprocess loop (new): Rather than running QA entirely in-process (single context window, no recovery), QA now follows the same pattern as /implement — an orchestrator generates qa.json (structured test plan with scenarios), the script assembles the prompt from a template, and qa.sh spawns fresh Claude Code subprocesses per scenario. Each subprocess picks up the next incomplete scenario, executes it, records results back to qa.json, and logs progress. The loop detects completion via a sentinel string and cross-verifies against qa.json.

Script-side template assembly (both scripts): Previously the orchestrator (Phase 2 agent) had to read the template, select a variant, fill placeholders, and save the prompt — a multi-step manual process that was error-prone and couldn't refresh context between iterations. Now the scripts handle this: they accept --spec-path, --typecheck-cmd, --lint-cmd, --test-cmd as CLI args, extract context fields from the JSON state file, compute a fresh git diff each iteration (with noise filtering and size-based fallback to stat tree), select the correct template variant (A: with SPEC.md, B: without), and use python3 for multi-line placeholder substitution. The orchestrator's job simplifies to: ensure the JSON state file has rich context fields, then run the script.

qa.json schema: Modeled after spec.json but adapted for testing — scenarios instead of userStories, category field for test classification, tools per scenario, availableTools at top level (detected once by orchestrator), result with richer status (pass / fail-fixed / blocked / skipped), successCriteria as explicit pass/fail checklist, and testContext for system startup instructions.

Changes

New files (QA infrastructure):

• qa/scripts/qa.sh — Iteration loop (377 lines), mirrors implement.sh with QA-specific adaptations (reads scenarios, QA COMPLETE sentinel, --spec-path for variant selection)
• qa/scripts/validate-qa.ts — Schema + semantic validator for qa.json (sequential priorities, unique IDs, valid categories/tools, field types)
• qa/templates/qa-prompt.template.md — Two-variant prompt template (A: with SPEC.md, B: without) with {{PLACEHOLDERS}} for script-side assembly
• qa/references/testing-guidance.md — Testing methodology extracted from SKILL.md (scenario categories, tool-specific execution patterns, formalization gate, depth calibration)

Modified files:

• implement/scripts/implement.sh — Added template assembly: compute_diff(), assemble_prompt(), noise filters, new CLI flags (--spec-path, --template, --typecheck-cmd, --lint-cmd, --test-cmd), {{DIFF}} injection
• implement/templates/implement-prompt.template.md — Added {{DIFF}} placeholder, updated source descriptions to reflect script-side assembly
• implement/SKILL.md — Simplified Phase 2 (no manual template crafting), updated Phase 3 invocation examples with new CLI flags
• qa/SKILL.md — Restructured from in-process workflow to orchestrator + subprocess loop (5 phases: gather context, generate qa.json, template assembly, execute loop, report)
• ship/SKILL.md — Added qa.json and qa-progress.txt to state files table, updated Phase 3 to describe subprocess loop, updated exit gate

Test plan

• implement.sh --help shows new flags (--spec-path, --template, --typecheck-cmd, --lint-cmd, --test-cmd)
• implement.sh auto-detects template from ../templates/implement-prompt.template.md when --template is omitted
• implement.sh assembles prompt correctly: variant A selected when --spec-path provided, variant B when omitted
• implement.sh computes diff with noise filtering (excludes lock files, tmp/, dist/, build/)
• implement.sh falls back to stat tree when diff exceeds 30K chars
• qa.sh --help shows expected flags
• bun validate-qa.ts passes against a well-formed qa.json and rejects malformed ones
• qa.sh spawns subprocess, reads qa.json, attempts scenario execution
• End-to-end: create qa.json with 1 CLI scenario, run qa.sh --max-iterations 1 — verify subprocess reads state and attempts the scenario

Generated with Claude Code