🎯 Repository Quality Improvement Report - Workflow Compilation Consistency

[github-actions[bot]]

🎯 Repository Quality Improvement Report - Workflow Compilation Consistency

Analysis Date: 2025-11-26
Focus Area: Workflow Compilation Consistency
Strategy Type: Custom (60% algorithm - repository-specific focus area)
Custom Area: Yes - This focus area addresses gh-aw's unique challenge of maintaining consistency between source workflow files (.md) and their compiled GitHub Actions artifacts (.lock.yml)

Executive Summary

This analysis reveals significant inconsistencies in the workflow compilation state, particularly affecting the 42 shared workflow components in .github/workflows/shared/. While top-level workflows maintain 100% compilation coverage, 40 out of 42 shared workflows (95.2%) are missing their compiled .lock.yml artifacts. Additionally, 6 top-level workflows have stale compilations where the source .md file was modified after the last compilation.

The root cause is the absence of automated enforcement mechanisms. Despite documentation stating "ALWAYS run gh aw compile after every change," there are no pre-commit hooks, no CI validation of compilation state, and limited visibility into staleness. This creates a maintenance burden and potential runtime issues when workflows reference uncompiled or stale shared components.

Full Analysis Report

Focus Area: Workflow Compilation Consistency

Current State Assessment

The gh-aw repository uses a unique compilation model where human-readable workflow definitions (.md files with YAML frontmatter) are compiled into GitHub Actions YAML (.lock.yml files). This compilation is critical for workflow execution but currently relies entirely on manual developer discipline.

Metrics Collected:

Metric	Value	Status
Total source workflows (.md)	132	ℹ️ Info
Top-level source workflows	90	ℹ️ Info
Shared workflow components	42	ℹ️ Info
Compiled workflows (.lock.yml)	92	⚠️ Warning
Top-level compilation rate	100.0%	✅ Good
Shared compilation rate	4.8% (2/42)	❌ Critical
Stale compilations detected	6	⚠️ Warning
Pre-commit hooks installed	0	❌ Missing
CI compilation validation	Partial	⚠️ Warning
Documentation completeness	Minimal	⚠️ Warning

Findings

Strengths

• Perfect top-level coverage: All 90 top-level workflow .md files have corresponding .lock.yml compilations
• CI rebuild automation: The ci.yml workflow includes make recompile to rebuild lock files on merge
• Purge capability: The --purge flag exists to remove orphaned .lock.yml files
• Clear documentation principle: Instructions state the compilation requirement explicitly

Areas for Improvement

1. Shared Workflow Compilation Gap (CRITICAL)

• 95.2% of shared workflows uncompiled: 40 out of 42 shared workflows in .github/workflows/shared/ lack .lock.yml artifacts
• Impact: Workflows importing these shared components may have undefined behavior or compilation errors
• Affected workflows include: actions-ai-inference.md, app-config.md, changeset-format.md, charts-with-trending.md, copilot-pr-data-fetch.md, and 35 others
• Severity: High - affects workflow reliability and import functionality

2. Stale Compilation Detection (HIGH)

• 6 workflows with outdated compilations: Source files modified after their .lock.yml was generated
• Affected workflows:
  • ai-triage-campaign.lock.yml
  • commit-changes-analyzer.lock.yml
  • copilot-session-insights.lock.yml
  • glossary-maintainer.lock.yml
  • pdf-summary.lock.yml
  • release.lock.yml
• Impact: Runtime behavior may not match source intent; debugging confusion
• Severity: Medium - creates maintenance confusion and potential bugs

3. No Pre-Commit Enforcement (HIGH)

• No automation: Developers must remember to run make recompile manually
• Git hooks unused: .git/hooks/ contains only sample files, no active pre-commit hooks
• Human error prone: Easy to forget compilation step, especially for new contributors
• Severity: High - root cause of inconsistencies

4. Limited Compilation State Visibility (MEDIUM)

• No status command: No easy way to check which workflows need recompilation
• No CI warnings: CI doesn't fail or warn on stale compilations (only rebuilds silently)
• Manual verification required: Developers must use filesystem commands to check staleness
• Severity: Medium - reduces developer awareness

5. Documentation Gaps (MEDIUM)

• Sparse mentions: Only 1 mention of make recompile in CONTRIBUTING.md
• No troubleshooting guide: No guidance on handling compilation errors or staleness
• Setup instructions missing: DEVGUIDE.md doesn't mention pre-commit hook setup
• Severity: Medium - increases onboarding friction

Detailed Analysis

Shared Workflow Compilation Mystery

The shared workflows are designed as reusable components (imported via imports: field), but they are not currently being compiled. Analysis suggests:

1. Intentional design decision?: Shared workflows may be import-only (no direct execution)
2. Compiler behavior: The gh aw compile command may skip shared directories by default
3. Missing documentation: No explanation of whether shared workflows should be compiled

Recommendation: Clarify the intended behavior and either:

• Document that shared workflows are import-only and don't need .lock.yml
• Update the compiler to compile shared workflows when they have workflow triggers
• Create a validation check to ensure shared workflows without triggers don't have .lock.yml

Staleness Detection Challenges

Current staleness happens because:

1. Developer modifies .md file
2. Developer forgets to run make recompile
3. Commit includes .md changes but no .lock.yml update
4. CI runs make recompile and commits changes separately (creating extra commits)

This creates "repair commits" that pollute git history with compilation-only changes.

CI Workflow Analysis

The current ci.yml includes:

- name: Rebuild lock files
  run: make recompile
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

This rebuilds on merge but doesn't:

• Fail the build if compilations are stale
• Comment on PRs about missing compilations
• Provide diff showing what changed in .lock.yml

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for processing.

Improvement Tasks

The following code regions and tasks should be processed by the Copilot agent. Each section is marked for easy identification by the planner agent.

---

Task 1: Create Pre-Commit Hook for Automatic Compilation

Priority: High
Estimated Effort: Medium
Focus Area: Workflow Compilation Consistency

Description:
Create a Git pre-commit hook that automatically runs make recompile (or at minimum, validates compilation state) before allowing commits. This prevents developers from forgetting to compile workflows and reduces "repair commits" in the git history.

Acceptance Criteria:

• Hook script created in .git/hooks/pre-commit (and provided as installable template)
• Hook detects modified .md workflow files in the commit
• Hook runs compilation check (staleness detection or full recompile)
• Hook provides clear error messages if compilation needed
• Hook installation documented in DEVGUIDE.md setup section
• Hook respects --no-verify flag for emergency commits

Code Region: .git/hooks/ and scripts/install-hooks.sh

Create a pre-commit Git hook that enforces workflow compilation consistency.

The hook should:
1. Detect if any `.md` files in `.github/workflows/` are staged for commit
2. Check if corresponding `.lock.yml` files are up-to-date
3. If stale or missing, either:
   - Run `make recompile` automatically, OR
   - Block the commit with a helpful error message
4. Provide clear feedback about what needs to be compiled

Also create an installation script `scripts/install-hooks.sh` that developers can run during setup, and document it in `DEVGUIDE.md`.

Consider using a lightweight check (timestamp comparison) rather than full recompilation to keep the hook fast (<2 seconds).

---

Task 2: Add Compilation Status Command

Priority: High
Estimated Effort: Medium
Focus Area: Workflow Compilation Consistency

Description:
Implement a new CLI command gh aw status (or gh aw compile --status) that shows the compilation state of all workflows: which are compiled, which are stale, which are missing .lock.yml, and which shared workflows are uncompiled.

Acceptance Criteria:

• New command shows compilation status for all workflows
• Output includes: workflow name, compilation status, last compiled time
• Clearly identifies stale compilations (source newer than .lock.yml)
• Identifies missing .lock.yml files
• Supports --json output for scripting
• Exit code 0 if all current, non-zero if any stale/missing
• Command documented in CLI help and user documentation

Code Region: pkg/cli/compile_command.go or new pkg/cli/status_command.go

Create a compilation status command that helps developers understand which workflows need recompilation.

The command should provide a table or list showing:
- Workflow name
- Compilation status: ✅ Current, ⚠️ Stale, ❌ Missing
- Last modified (source)
- Last compiled (.lock.yml timestamp)
- Time delta between source and compilation

Support both human-readable and JSON output formats.

Example output:

Workflow Compilation Status:

workflow-name.md ✅ Current (compiled 2h ago)
ai-triage-campaign.md ⚠️ Stale (source modified 1d ago, compiled 3d ago)
shared/reporting.md ❌ Missing (no .lock.yml found)

Summary: 88 current, 6 stale, 40 missing

This visibility will help developers catch compilation issues before committing.

---

Task 3: Enhance CI to Fail on Stale Compilations

Priority: High
Estimated Effort: Small
Focus Area: Workflow Compilation Consistency

Description:
Update .github/workflows/ci.yml to explicitly check for stale or missing compilations BEFORE running make recompile. If staleness detected, fail the build with a clear error message. This prevents merging PRs with uncompiled workflows.

Acceptance Criteria:

• New CI step added before the current make recompile step
• Step runs compilation status check (use Task 2 if available, or custom script)
• CI fails with exit code 1 if stale/missing compilations found
• Error message includes list of affected workflows
• Error message explains how to fix locally (make recompile)
• Existing make recompile step preserved for fallback/repair

Code Region: .github/workflows/ci.yml

Enhance the CI workflow to catch compilation inconsistencies before merge.

Add a new step in `.github/workflows/ci.yml` before the current "Rebuild lock files" step:

```yaml
- name: Check compilation status
  run: |
    # Check for stale or missing .lock.yml files
    # Use `gh aw status` command if available (Task 2), or implement inline check
    
    # Detect workflows where .md is newer than .lock.yml
    # Exit 1 if any found
    
    # Provide clear error message with affected workflow list

This prevents PRs with uncompiled workflows from being merged. Developers must run make recompile locally before pushing.

Keep the existing make recompile step as a safety net but consider making it less silent (add warnings or diff output).

---

#### Task 4: Clarify Shared Workflow Compilation Policy

**Priority**: Medium  
**Estimated Effort**: Small  
**Focus Area**: Workflow Compilation Consistency

**Description:**
Investigate whether shared workflows in `.github/workflows/shared/` are intended to have `.lock.yml` compilations. Document the policy clearly in `CONTRIBUTING.md` or `DEVGUIDE.md`. If they should be compiled, update the compiler to handle them. If not, add validation to ensure they don't have triggers.

**Acceptance Criteria:**
- [ ] Decision documented: Should shared workflows be compiled or not?
- [ ] If YES: Compiler updated to compile shared workflows with triggers
- [ ] If NO: Validation added to warn if shared workflow has `on:` trigger
- [ ] Documentation updated in `CONTRIBUTING.md` with clear policy
- [ ] Existing 2 shared `.lock.yml` files addressed (remove if not needed)
- [ ] `make recompile` behavior clarified for shared directory

**Code Region:** `pkg/workflow/compiler.go`, `CONTRIBUTING.md`, `.github/workflows/shared/README.md` (create if needed)

```markdown
Investigate and document the intended compilation behavior for shared workflows.

Research questions:
1. Do shared workflows ever run directly, or are they only imported?
2. Should workflows in `.github/workflows/shared/` have `.lock.yml` files?
3. Why do 2 shared workflows have `.lock.yml` but 40 don't?
4. Does the compiler skip the shared directory intentionally?

Based on findings, either:
- **Option A**: Update compiler to compile shared workflows if they have triggers
- **Option B**: Document that shared workflows are import-only and validate they have no triggers

Create documentation in `.github/workflows/shared/README.md` explaining:
- Purpose of the shared directory
- Whether these workflows should be compiled
- How to create new shared workflows
- Naming conventions and best practices

---

Task 5: Add Compilation Freshness Documentation

Priority: Medium
Estimated Effort: Small
Focus Area: Workflow Compilation Consistency

Description:
Expand DEVGUIDE.md and CONTRIBUTING.md to include comprehensive guidance on workflow compilation: when to compile, how to check staleness, troubleshooting compilation errors, and best practices for maintaining compilation consistency.

Acceptance Criteria:

• New section in DEVGUIDE.md titled "Workflow Compilation"
• Documentation includes: when to run make recompile, how to check staleness, common errors
• Troubleshooting guide for compilation failures
• Best practices: compile before committing, use pre-commit hooks, check status
• Cross-references to pre-commit hook installation (Task 1)
• Examples of compilation workflow for common scenarios

Code Region: DEVGUIDE.md, CONTRIBUTING.md

Enhance developer documentation with comprehensive workflow compilation guidance.

Add a new section to `DEVGUIDE.md`:

```markdown
## Workflow Compilation

Agentic workflows are defined in `.md` files and compiled to `.lock.yml` GitHub Actions files.

### When to Compile

Run `make recompile` whenever you:
- Create a new workflow `.md` file
- Modify an existing workflow `.md` file
- Update imports or shared workflow components
- Change workflow frontmatter (triggers, permissions, tools)

### Checking Compilation Status

```bash
# Check if compilations are current
gh aw status  # (if Task 2 implemented)

# Or manually check staleness
make recompile --dry-run  # (if flag exists)

Troubleshooting

"Compilation failed" error:

• Check YAML syntax in frontmatter
• Validate imports exist and are accessible
• Review error message for specific issue

Stale .lock.yml detected:

• Run make recompile to update
• Commit both .md and .lock.yml together

Best Practices

1. Install pre-commit hooks: scripts/install-hooks.sh (see Task 1)
2. Always compile before committing
3. Review .lock.yml diffs in PRs
4. Use --purge flag to remove orphaned files

Also update `CONTRIBUTING.md` with a brief reminder about compilation requirements.

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2025-11-25	Workflow Observability & Debugging	Custom	Y	Identified error message quality gaps, console formatting issues, strong debugging ecosystem
2025-11-26	Workflow Compilation Consistency	Custom	Y	Found 95.2% shared workflow compilation gap, 6 stale compilations, no enforcement automation

Statistics:

• Total runs: 2
• Custom focus rate: 100%
• Unique areas explored: 2
• Average tasks per run: 5

---

🎯 Recommendations

Immediate Actions (This Week)

1. Implement pre-commit hook (Task 1) - Prevents future inconsistencies at commit time
2. Add compilation status command (Task 2) - Provides visibility into current state
3. Enhance CI validation (Task 3) - Catches issues before merge

Short-term Actions (This Month)

1. Clarify shared workflow policy (Task 4) - Resolves the 95.2% compilation gap question
2. Expand documentation (Task 5) - Reduces onboarding friction and maintenance confusion
3. Run comprehensive recompile - Address all 6 stale compilations and verify shared workflow behavior

Long-term Actions (This Quarter)

1. Add compilation diff in CI - Show developers what changed in .lock.yml during recompilation
2. Create compilation metrics - Track compilation freshness over time in observability dashboards
3. Consider compilation caching - Speed up make recompile for large repositories

---

📈 Success Metrics

Track these metrics to measure improvement in Workflow Compilation Consistency:

• Stale compilation rate: 6.7% (6/90) → Target: 0%
• Shared compilation rate: 4.8% (2/42) → Target: 100% or 0% (based on Task 4 decision)
• Pre-commit hook adoption: 0% → Target: 100%
• CI failures due to stale compilations: Unknown → Track and minimize
• Average time to detect staleness: Manual → Target: Immediate (via pre-commit hook)
• Developer compilation errors: Unknown → Track and reduce via documentation

---

Next Steps

1. Prioritize Task 1, 2, and 3 - These provide the most immediate impact on compilation consistency
2. Assign tasks to Copilot agent via planner agent for implementation
3. Review shared workflow policy (Task 4) - This decision affects the interpretation of the 95.2% gap
4. Run full make recompile after Task 4 decision to establish baseline
5. Monitor metrics after changes deployed to measure effectiveness

---

Generated by Repository Quality Improvement Agent
Next analysis: 2025-11-27 - Focus area will be selected based on diversity algorithm (aim for continued custom areas to maintain 60%+ rate)

AI generated by Repository Quality Improvement Agent