[Schema Consistency] 🔍 Schema Consistency Check - 2025-11-24 - Execution Context Analysis

[github-actions[bot]]

🔍 Schema Consistency Check - 2025-11-24

This analysis discovered 6 critical/moderate inconsistencies where runtime execution depends on environment variables, filesystem layouts, and configurations not documented in the JSON schema.

Summary

• Inconsistencies Found: 6 (2 critical, 4 moderate)
• Categories Analyzed: Schema, Parser, Compiler, Documentation, Workflows
• Strategy Used: Strategy-019: Execution Context & Runtime Environment Assumptions
• New Strategy: YES - First analysis to examine runtime execution prerequisites vs schema

Critical Issues

Critical Issues

1. GitHub Actions Environment Variables Not in Schema

Severity: CRITICAL
Location: Throughout pkg/workflow/*.go, schema has no documentation

Issue: Workflows implicitly depend on 7+ GitHub Actions environment variables that are never mentioned in the schema.

Variables Used in Code:

• GITHUB_WORKSPACE (pkg/workflow/compiler_yaml.go:542, copilot_engine.go:93)
• GITHUB_TOKEN (pkg/workflow/codex_engine.go:68-70, multiple engines)
• GITHUB_REPOSITORY (pkg/workflow/codex_engine.go:69)
• GITHUB_RUN_ATTEMPT (pkg/workflow/compiler_yaml.go:879)
• COPILOT_GITHUB_TOKEN (pkg/workflow/copilot_engine.go:91)
• GH_AW_GITHUB_TOKEN (throughout codebase, fallback chains)
• COPILOT_CLI_TOKEN (legacy, pkg/workflow/copilot_engine.go:91)

Schema Status: No mention of available environment variables or GitHub Actions context

Documentation Status: Partially documented in docs/src/content/docs/reference/engines.md and compilation-process.md, but NOT in schema

Impact:

• IDE autocomplete cannot suggest ${{ env.* }} or ${{ github.* }} expressions
• Users don't know which environment variables are available
• External tooling cannot validate environment variable usage

Evidence:

// pkg/workflow/codex_engine.go:68-70
envVars["GITHUB_TOKEN"] = true
envVars["GITHUB_REPOSITORY"] = true

// pkg/workflow/compiler_yaml.go:542
yaml.WriteString(fmt.Sprintf("This workflow is running in directory $GITHUB_WORKSPACE..."))

// pkg/workflow/copilot_engine.go:91
copilotGitHubToken := "${{ secrets.COPILOT_GITHUB_TOKEN || secrets.COPILOT_CLI_TOKEN }}"

Recommendation: Add $comment field to schema documenting available GitHub Actions context variables

---

2. Filesystem Path Assumptions Not Documented

Severity: CRITICAL
Location: pkg/workflow/runtime_setup.go, workflows, documentation

Issue: Runtime assumes specific filesystem layout without schema documentation

Hardcoded Assumptions:

• go.mod at specific path (pkg/workflow/runtime_setup.go:569-588)
• /tmp/gh-aw/cache-memory/ for cache storage (workflows, docs)
• Workflow files relative to $GITHUB_WORKSPACE
• Log output paths in /tmp/gh-aw/

Schema Status: No filesystem requirements or conventions documented

Impact:

• Workflows may fail in Docker containers or custom runners
• Users don't know where to place configuration files
• No guidance on required disk space or path structure

Evidence:

// pkg/workflow/runtime_setup.go:569-588
goModPath := "go.mod"
if req.GoModFile != "" {
    goModPath = req.GoModFile
}
step = append(step, fmt.Sprintf("  go-version-file: %s", goModPath))

# Real workflows use hardcoded paths
.github/workflows/audit-workflows.md:
  run: ./gh-aw logs --start-date -1d -o /tmp/gh-aw/aw-mcp/logs

Recommendation: Document filesystem requirements in schema description or $comment field

Moderate Issues

Moderate Issues

3. Token Precedence Logic Not in Schema

Severity: MODERATE
Location: Multiple engine files, schema github-token field

Issue: Complex token fallback chains exist in code but not fully documented in schema

Token Chains:

• Standard workflows: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }}
• Copilot workflows: ${{ secrets.COPILOT_GITHUB_TOKEN || secrets.COPILOT_CLI_TOKEN || secrets.GH_AW_COPILOT_TOKEN || secrets.GH_AW_GITHUB_TOKEN }}
• Context-dependent: Different chains for copilot assignees vs regular assignees

Schema Status: github-token field exists with description mentioning precedence, but doesn't show fallback chain examples

Documentation Status: ✅ Well documented in engines.md and frontmatter-full.md

Impact: Users must read prose documentation to understand token selection behavior

Recommendation: Add schema examples array showing common fallback patterns

---

4. Engine-Specific Secret Requirements Not in Schema

Severity: MODERATE
Location: pkg/workflow/agentic_engine.go, engine schema definitions

Issue: Each engine requires different secrets, validated at runtime but not documented in schema

Evidence:

// pkg/workflow/agentic_engine.go:105-129
func GenerateSecretValidationStep(engineName, secretName string) []string {
    fmt.Sprintf("echo \"The %s engine requires the %s secret to be configured.\"", 
        engineName, secretName)
}

Schema Status: Engine schema doesn't document required vs optional secrets per engine

Documentation Status: ✅ Well documented per-engine in engines.md

Impact: Users don't know which secrets to configure until workflow execution fails

Recommendation: Add engine-specific required-secrets metadata to schema or $comment

---

5. Runtime Auto-Detection Not Schema-Documented

Severity: MODERATE
Location: pkg/workflow/runtime_setup.go:166-993

Issue: Sophisticated automatic runtime setup happens invisibly without schema explanation

Auto-Detection Behavior:

• Scans custom steps for commands (npm, pip, go, ruby, etc.)
• Detects from MCP tool configurations
• Automatically adds Python when UV detected (dependency chain)
• Serena language service maps languages to runtimes
• Deduplicates user-provided setup actions intelligently

Evidence:

// pkg/workflow/runtime_setup.go:192-199
// Add Python as dependency when uv is detected (uv requires Python)
if _, hasUV := requirements["uv"]; hasUV {
    if _, hasPython := requirements["python"]; !hasPython {
        pythonRuntime := findRuntimeByID("python")
        if pythonRuntime != nil {
            updateRequiredRuntime(pythonRuntime, "", requirements)
        }
    }
}

Supported Runtimes (11 total): bun, deno, dotnet, elixir, go, haskell, java, node, python, ruby, uv

Schema Status: runtimes field exists but doesn't explain auto-detection behavior

Documentation Status: ⚠️ Not prominently documented

Impact: "Magic" behavior that users don't understand; surprises when runtimes appear

Recommendation: Expand runtimes field description with auto-detection logic and examples

---

6. Tool Permission Requirements Not in Schema

Severity: MODERATE
Location: pkg/workflow/compiler.go, tool configurations

Issue: Tools have implicit permission requirements validated at runtime

Evidence:

// pkg/workflow/compiler.go:1337
message := "ERROR: Missing required permission for agentic-workflows tool:\n"
message += "The agentic-workflows tool requires actions: read permission to access GitHub Actions data.\n\n"

Schema Status: Tools schema doesn't document permission requirements per tool

Impact: Permission denied errors only appear at runtime after workflow starts

Recommendation: Add permission requirements to tool schema definitions or use $comment to document

Positive Findings

Positive Findings

✅ Environment variables well-documented in reference docs (engines.md, compilation-process.md)
✅ Token configuration comprehensively explained in engines.md and frontmatter-full.md
✅ Secret validation happens at workflow start (fail-fast pattern)
✅ Runtime detection is sophisticated and helpful (11 runtimes with command scanning, MCP detection, language services)
✅ Good error messages for missing secrets/permissions
✅ Documentation quality high where it exists (just not in schema)

Detailed Recommendations

Recommendations by Priority

HIGH PRIORITY

1. Add GitHub Actions Context Documentation

What: Schema $comment or description documenting available environment variables
Where: Top-level schema or env field
Format:

{
  "env": {
    "$comment": "Available GitHub Actions context: GITHUB_WORKSPACE, GITHUB_TOKEN, GITHUB_REPOSITORY, GITHUB_RUN_ATTEMPT, GITHUB_EVENT_PATH. See https://docs.github.com/en/actions/learn-github-actions/variables",
    "type": "object",
    "description": "Environment variables..."
  }
}

2. Document Filesystem Requirements

What: Schema description or $comment with filesystem layout assumptions
Where: Top-level schema or in relevant fields (cache-memory, etc.)
Content: Document /tmp/gh-aw/ paths, go.mod conventions, GITHUB_WORKSPACE relativity

---

MEDIUM PRIORITY

3. Engine Secret Requirements

What: Add required/optional secrets metadata per engine
Where: Engine schema definitions
Format: Add required-secrets and optional-secrets arrays or use $comment

4. Tool Permission Requirements

What: Document permission needs per tool
Where: Tools schema or individual tool definitions
Format: Add required-permissions field or $comment

5. Runtime Auto-Detection Explanation

What: Expand runtimes field description
Where: Schema runtimes property
Content: Explain command scanning, dependency chains (UV→Python), Serena language detection

---

LOW PRIORITY

6. Token Fallback Examples

What: Add schema examples showing token precedence chains
Where: github-token field examples array

7. Runner Resource Guidance

What: Document minimum runner specs for AI workloads
Where: runs-on field description or $comment
Content: CPU, memory, disk space recommendations

Strategy Analysis & Methodology

Strategy Performance

Strategy ID: strategy-019
Name: Execution Context & Runtime Environment Assumptions
Findings: 6 (2 critical, 4 moderate)
Effectiveness: VERY HIGH
Novelty: ⭐ NEW - First strategy to analyze runtime execution assumptions vs schema promises

Why This Strategy Works

1. Bottom-up from implementation: Started with actual code behavior rather than schema definitions
2. Execution-focused: Examined what happens at runtime vs what schema documents
3. Cross-layer analysis: Validated assumptions across compiler, engines, docs, workflows
4. Environment-aware: Considered GitHub Actions execution context specifically

Methodology

✅ Grep for environment variable usage (GITHUB_*, RUNNER_*, etc.)
✅ Analyzed pkg/workflow/runtime_setup.go for auto-detection patterns
✅ Traced token precedence chains through engine implementations
✅ Cross-referenced schema with documentation for execution context
✅ Validated against real workflow examples in .github/workflows/
✅ Identified filesystem and secret configuration assumptions

Complementary Strategies

This strategy pairs well with:

• Strategy-005: Conditional Logic & Feature Flag Analysis (finds runtime validation gaps)
• Strategy-009: Cross-File Feature Consistency (end-to-end tracing)
• Strategy-012: Dynamic Validation (runtime behavior validation)

Implementation Quality Notes

Implementation Quality

Overall Assessment: ⭐⭐⭐⭐⭐ Excellent

The implementation quality is very high. The issues found are documentation gaps rather than implementation bugs:

• ✅ Environment variable handling is correct and consistent
• ✅ Token precedence logic is well-implemented with proper fallbacks
• ✅ Runtime auto-detection is sophisticated and helpful
• ✅ Secret validation happens early (fail-fast)
• ✅ Error messages are clear and actionable

Main Gap: Schema metadata doesn't reflect the sophistication of the implementation

Recommendation: Focus on enhancing schema documentation rather than changing code

Next Steps

• Add GitHub Actions context documentation to schema
• Document filesystem layout requirements and conventions
• Add engine-specific secret requirements to schema
• Expand runtime auto-detection documentation
• Document tool permission requirements
• Add token fallback chain examples

Strategy Update

Cache Updated: Strategy-019 added to /tmp/gh-aw/cache-memory/strategies.json
Success Count: 1
Last Used: 2025-11-24
Recommendation: Use every 5-6 analyses to audit runtime requirements and execution prerequisites

---

Analysis generated using Strategy-019: Execution Context & Runtime Environment Assumptions
Focus: Runtime execution assumptions vs schema documentation
Novel approach: First strategy to analyze execution environment prerequisites

AI generated by Schema Consistency Checker