🎯 Repository Quality Improvement Report - AI Engine Reliability & Fallback Strategies

[github-actions[bot]]

Analysis Date: 2026-01-30
Focus Area: AI Engine Reliability & Fallback Strategies
Strategy Type: Custom
Custom Area: Yes - This focus area addresses the production-critical challenge of ensuring 205+ agentic workflows remain resilient when AI engines experience failures, rate limits, network issues, or authentication problems.

Executive Summary

With 4 AI engines (copilot, claude, codex, custom) powering 205 workflows across the repository, engine reliability is mission-critical for production workflow stability. This analysis reveals a mature engine architecture (4,881 source LOC, 2.20:1 test ratio) but critical gaps in failure handling that leave workflows vulnerable to cascading failures.

Key Findings:

• ✅ Strong foundation: 70.2% workflows use continue-on-error, 555 timeout configurations, 2.20:1 test coverage
• ⚠️ Zero standardized retry infrastructure despite 32 ad-hoc retry implementations across codebase
• ❌ Zero circuit breaker patterns (4 mentions, none implemented) - no protection against cascading failures
• ❌ Zero multi-engine fallback strategies - when primary engine fails, workflow fails (only 4 workflows mention fallback)
• ❌ Zero engine health checks - no proactive detection of engine degradation before job submission
• ❌ Zero structured error types for engine failures - generic errors provide no actionable context

Impact: Without systematic reliability patterns, workflows experience unpredictable failures that could be mitigated through retry logic, circuit breakers, and multi-engine fallback strategies. The 71 workflows (34.6%) missing explicit engine configuration suggest unclear engine selection guidance.

Full Analysis Report

Focus Area: AI Engine Reliability & Fallback Strategies

Current State Assessment

Engine Distribution & Adoption:

Metric	Value	Status
Total workflows	205	✅
Copilot engine	75 (36.6%)	✅
Claude engine	31 (15.1%)	✅
Codex engine	8 (3.9%)	✅
Missing engine spec	71 (34.6%)	⚠️
Empty engine value	25 (12.2%)	❌

Codebase Maturity:

Metric	Value	Status
Engine source LOC	4,881 lines	✅
Engine test LOC	10,725 lines	✅
Test-to-code ratio	2.20:1	✅
Engine source files	16 files	✅
Engine test files	26 files	✅

Reliability Infrastructure:

Metric	Value	Status
Continue-on-error usage	144/205 (70.2%)	✅
Timeout configurations	555 across workflows	✅
Standardized retry library	0 (32 ad-hoc implementations)	❌
Circuit breaker patterns	0 (4 mentions, none implemented)	❌
Multi-engine fallback	4 workflows (1.9%)	❌
Engine health checks	0 implementations	❌
Structured error types	0 engine-specific types	❌
Retry steps in compiled workflows	7/205 (3.4%)	⚠️

Findings

Strengths

1. Excellent Test Coverage: 2.20:1 test-to-code ratio (10,725 test LOC vs 4,881 source LOC) demonstrates commitment to quality
2. Defensive Timeout Configuration: 555 timeout configurations across workflows prevent indefinite hangs
3. Widespread Error Tolerance: 70.2% (144/205) workflows use continue-on-error for graceful degradation
4. Multiple Engine Options: 4 engines (copilot, claude, codex, custom) provide flexibility for different use cases
5. Secret Validation: 31 secret validation implementations prevent authentication failures at runtime
6. Context-Based Timeouts: 302 context.WithTimeout implementations show mature Go patterns

Areas for Improvement

1. Zero Standardized Retry Infrastructure (CRITICAL)

   • 32 ad-hoc retry implementations scattered across codebase
   • No shared retry library with exponential backoff (only 3 backoff mentions)
   • Each developer reinvents retry logic with inconsistent error handling
   • No configurable retry policies (max attempts, backoff strategy, jitter)

2. Zero Circuit Breaker Patterns (CRITICAL)

   • 4 circuit breaker mentions but zero implementations
   • No protection against cascading failures when engine is degraded
   • Workflows continue hammering failed engines instead of failing fast
   • No automatic recovery detection or half-open state testing

3. Zero Multi-Engine Fallback Strategies (HIGH)

   • Only 4/205 workflows (1.9%) mention fallback in markdown
   • No automatic failover from primary to secondary engine
   • When Copilot fails, workflows don't try Claude/Codex as fallback
   • No engine preference ordering or automatic degradation

4. Zero Engine Health Checks (HIGH)

   • No pre-flight checks before submitting jobs to engines
   • No proactive detection of engine API degradation
   • No dashboard or monitoring for engine availability metrics
   • Workflows discover failures only after job submission (wasted time)

5. Zero Structured Error Types (MEDIUM)

   • No EngineError, RateLimitError, AuthenticationError types
   • Generic error interface loses context about failure category
   • Cannot distinguish retryable (network timeout) from non-retryable (auth failure) errors
   • Error handling code cannot make intelligent decisions about recovery

6. Minimal Engine-Specific Timeout Logic (MEDIUM)

   • Only 10 timeout configuration lines found in engine code
   • No per-engine timeout tuning based on historical performance
   • No adaptive timeout adjustment based on success rate
   • Claude/Codex timeouts may not align with actual API SLAs

7. 34.6% Workflows Missing Engine Specification (MEDIUM)

   • 71/205 workflows don't specify which engine to use
   • 25 workflows have empty engine: field (12.2%)
   • No clear guidance on engine selection criteria
   • May cause silent fallback to default engine without intent

8. Zero Rate Limit Handling (MEDIUM)

   • Only 2 rate limit mentions across entire codebase
   • No 429 response handling or automatic backoff
   • No per-engine rate limit tracking or quota management
   • Workflows fail immediately on rate limits instead of waiting

Detailed Analysis

Engine Architecture

The codebase demonstrates a well-structured engine abstraction with 16 source files organized around a central EngineRegistry:

pkg/workflow/
├── agentic_engine.go           # Registry & interface (92 LOC)
├── copilot_engine.go           # Copilot implementation (168 LOC)
├── copilot_engine_execution.go # Execution logic (571 LOC)
├── copilot_engine_installation.go # Installation (180 LOC)
├── copilot_engine_tools.go     # Tool integration (206 LOC)
├── claude_engine.go            # Claude implementation
├── codex_engine.go             # Codex implementation
├── custom_engine.go            # Custom engine support
└── engine_validation.go        # Validation (121 LOC)

Engine Registry Design:

• Singleton pattern with sync.Once initialization
• 4 built-in engines registered at startup
• GetEngine(id) returns CodingAgentEngine interface
• Clean separation of concerns (execution, installation, validation)

However, the architecture lacks failure handling primitives:

• No RetryableEngine wrapper for automatic retry
• No CircuitBreakerEngine wrapper for fault tolerance
• No FallbackEngine for multi-engine strategies
• No HealthCheckEngine for proactive monitoring

Retry Logic Gap Analysis

Current State:

• 32 retry implementations scattered across 32+ files
• No shared retry library - every developer writes their own
• Inconsistent patterns:
  • Some use for loops with sleep
  • Some use exponential backoff (only 3 found)
  • Some have configurable max attempts, others hardcoded
  • No jitter to prevent thundering herd

Example Ad-Hoc Retry Pattern (pkg/cli/run_workflow_tracking.go):

for attempt := 0; attempt < maxRetries; attempt++ {
    result, err := tryOperation()
    if err == nil {
        return result
    }
    time.Sleep(delay)
}
return error // All retries exhausted

Missing:

• Shared retry.Do() function accepting func() error
• Configurable RetryPolicy struct (max attempts, backoff, jitter)
• Distinction between retryable vs non-retryable errors
• Metrics tracking (retry attempts, success rate, latency)

Circuit Breaker Gap

Current State:

• 4 mentions of "circuit" or "breaker" in codebase
• Zero implementations of circuit breaker pattern
• No state tracking (closed, open, half-open)
• No automatic failure threshold detection

Risk Without Circuit Breaker:

1. When Copilot API experiences outage, all 75 Copilot workflows continue hitting dead endpoint
2. Cascading failures consume GitHub Actions minutes unnecessarily
3. No automatic recovery - workflows keep failing even after API recovers
4. No fast-fail - each workflow waits for full timeout before failing

What's Needed:

type CircuitBreaker struct {
    maxFailures   int           // Open circuit after N failures
    timeout       time.Duration // How long to wait before half-open
    state         State         // Closed, Open, HalfOpen
    failureCount  int
    lastFailTime  time.Time
}

func (cb *CircuitBreaker) Execute(f func() error) error {
    if cb.state == Open && time.Since(cb.lastFailTime) > cb.timeout {
        cb.state = HalfOpen // Test if service recovered
    }
    
    if cb.state == Open {
        return ErrCircuitOpen // Fail fast
    }
    
    err := f()
    if err != nil {
        cb.recordFailure()
    } else {
        cb.recordSuccess()
    }
    return err
}

Multi-Engine Fallback Strategy Gap

Current State:

• Only 4/205 workflows (1.9%) mention "fallback" in markdown
• No automatic engine failover in compiled workflows
• When primary engine fails, workflow fails (no retry with different engine)

Example Desired Behavior:

# Workflow frontmatter
engine: copilot
fallback:
  - claude    # Try Claude if Copilot fails
  - codex     # Try Codex if Claude fails
  - custom    # Try custom engine as last resort

Current vs Desired:

Scenario	Current Behavior	Desired Behavior
Copilot 429 rate limit	Workflow fails immediately	Retry with Claude
Claude auth failure	Workflow fails	Retry with Copilot
Copilot timeout	Workflow fails after timeout	Fast-fail, try Codex
All engines down	Workflow fails	Fail with aggregated errors

Implementation Gap:

• No FallbackEngine wrapper in engine registry
• No engine preference ordering in frontmatter schema
• No logic to try alternate engines on failure
• No error aggregation across multiple engine attempts

Engine Health Check Gap

Current State:

• Zero pre-flight health checks before job submission
• Zero proactive monitoring of engine API availability
• Workflows discover failures only after starting (wasted minutes)

What's Needed:

type EngineHealthChecker interface {
    HealthCheck(ctx context.Context) error
    GetStatus() EngineStatus // Healthy, Degraded, Down
    GetLatency() time.Duration
    GetErrorRate() float64
}

// Pre-flight check before workflow submission
func (c *Compiler) CheckEngineHealth(engineID string) error {
    engine := registry.GetEngine(engineID)
    if checker, ok := engine.(EngineHealthChecker); ok {
        return checker.HealthCheck(context.Background())
    }
    return nil // No health check available
}

Benefits:

• Fail fast if engine is down (don't start workflow)
• Display engine status in gh aw status command
• Track engine availability metrics over time
• Proactive alerting when engine degrades

Structured Error Types Gap

Current State:

• Zero engine-specific error types found
• All errors use generic error interface
• Cannot distinguish error categories programmatically

Example Current Error Handling:

result, err := engine.Execute(ctx, workflow)
if err != nil {
    return fmt.Errorf("execution failed: %w", err)
}
// Lost: Was it auth? Rate limit? Timeout? Network?

What's Needed:

type EngineError struct {
    Engine    string
    Category  ErrorCategory // Auth, RateLimit, Timeout, Network
    Retryable bool
    Cause     error
}

type ErrorCategory int
const (
    ErrCategoryAuth ErrorCategory = iota
    ErrCategoryRateLimit
    ErrCategoryTimeout
    ErrCategoryNetwork
    ErrCategoryValidation
)

func (e *EngineError) Error() string {
    return fmt.Sprintf("%s engine %s error: %v (retryable: %t)",
        e.Engine, e.Category, e.Cause, e.Retryable)
}

Benefits:

• Retry only retryable errors (timeout, network)
• Skip retry for non-retryable errors (auth, validation)
• Better error messages with specific guidance
• Metrics aggregation by error category

Timeout Configuration Analysis

Current State:

• 555 timeout configurations across all workflows (good coverage)
• 302 context.WithTimeout implementations in Go code
• 10 engine-specific timeout configurations found

Gaps:

• No per-engine timeout tuning based on historical data
• No adaptive timeout adjustment (e.g., increase if success rate drops)
• No timeout violation metrics or monitoring
• Default timeouts may not align with actual engine SLAs

Example Timeout Configuration:

// pkg/workflow/claude_engine.go
timeoutMs := int(constants.DefaultToolTimeout / time.Millisecond)
if workflowData.ToolsTimeout > 0 {
    timeoutMs = workflowData.ToolsTimeout * 1000
}

What's Missing:

• Per-engine timeout recommendations based on P95 latency
• Timeout violation alerting and tracking
• Automatic timeout adjustment based on success rate
• Correlation between timeout values and failure rate

Documentation Gaps

Current Documentation:

• 5 reliability-related docs found (security, troubleshooting, bootstrap)
• 1 engine selection guide in docs/src/content/docs/setup/cli.md
• 10 workflows showing continue-on-error examples

Missing Documentation:

• Engine Selection Decision Guide: When to use Copilot vs Claude vs Codex?
• Reliability Best Practices: How to add retry, timeout, fallback to workflows?
• Engine Failure Troubleshooting: What to do when engine fails (runbook)?
• Multi-Engine Fallback Examples: Show fallback configuration patterns
• Rate Limit Handling Guide: How to handle 429 responses gracefully?
• Engine SLA Documentation: Expected latency, availability, rate limits per engine

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for Claude to process.

Improvement Tasks

The following code regions and tasks should be processed by the Copilot agent. Each section is marked for easy identification by the planner agent.

---

Task 1: Implement Standardized Retry Infrastructure

Priority: High
Estimated Effort: Large
Focus Area: Core Infrastructure

Description:
Create a centralized retry library (pkg/retry/) that provides configurable retry logic with exponential backoff, jitter, and error categorization. This will eliminate 32+ ad-hoc retry implementations and establish consistent reliability patterns across the codebase.

Acceptance Criteria:

• Create pkg/retry/retry.go with Do(), DoWithBackoff(), and DoWithPolicy() functions
• Implement RetryPolicy struct with configurable MaxAttempts, InitialDelay, MaxDelay, Backoff (exponential/linear), and Jitter
• Add RetryableError interface to distinguish retryable from non-retryable errors
• Provide sensible defaults (3 attempts, exponential backoff, 100ms-5s range, 10% jitter)
• Add comprehensive tests covering happy path, retry exhaustion, non-retryable errors, backoff calculation
• Include metrics collection (retry attempts, success rate, total duration)
• Document usage with examples in godoc

Code Region: pkg/retry/ (new package)

Create a production-grade retry library for gh-aw that eliminates 32+ ad-hoc retry implementations.

Requirements:
1. **Package Structure**: Create `pkg/retry/retry.go` with:
   - `Do(ctx context.Context, fn func() error) error` - Simple retry with defaults
   - `DoWithBackoff(ctx context.Context, backoff Backoff, fn func() error) error` - Custom backoff
   - `DoWithPolicy(ctx context.Context, policy Policy, fn func() error) error` - Full control

2. **Policy Configuration**:
   ```go
   type Policy struct {
       MaxAttempts  int           // Default: 3
       InitialDelay time.Duration // Default: 100ms
       MaxDelay     time.Duration // Default: 5s
       Backoff      BackoffType   // Exponential, Linear, Fixed
       Jitter       float64       // 0.0-1.0, default: 0.1 (10%)
       OnRetry      func(attempt int, err error) // Optional callback
   }
   
   type BackoffType int
   const (
       BackoffExponential BackoffType = iota // 100ms, 200ms, 400ms, 800ms...
       BackoffLinear                         // 100ms, 200ms, 300ms, 400ms...
       BackoffFixed                          // 100ms, 100ms, 100ms, 100ms...
   )
   ```

3. **Error Categorization**:
   ```go
   type RetryableError interface {
       error
       IsRetryable() bool
   }
   
   // Helpers
   func IsRetryable(err error) bool
   func AsRetryable(err error, retryable bool) error
   ```

4. **Defaults**:
   - MaxAttempts: 3
   - InitialDelay: 100ms
   - MaxDelay: 5s
   - Backoff: Exponential
   - Jitter: 10%

5. **Testing** (pkg/retry/retry_test.go):
   - TestDoSuccess - Succeeds on first attempt
   - TestDoRetrySuccess - Succeeds on 2nd attempt after 1 failure
   - TestDoMaxAttemptsExhausted - Fails after 3 attempts
   - TestDoNonRetryableError - Fails immediately without retry
   - TestDoContextCanceled - Respects context cancellation
   - TestExponentialBackoff - Validates 100ms, 200ms, 400ms progression
   - TestLinearBackoff - Validates 100ms, 200ms, 300ms progression
   - TestJitter - Validates randomization within 10% range
   - TestOnRetryCallback - Callback invoked on each retry
   - TestMaxDelayRespected - Backoff doesn't exceed MaxDelay

6. **Documentation**:
   - Godoc with usage examples for each function
   - Example showing migration from ad-hoc retry to library
   - Performance characteristics and overhead notes

7. **Integration**:
   - Does NOT modify existing code yet (Task 2 handles migration)
   - Provides clean API for future adoption
   - Zero dependencies beyond stdlib (time, context, errors)

Follow existing codebase conventions:
- Use `logger.New("retry")` for debug logging
- Follow Go error wrapping patterns with `%w`
- Include helpful godoc examples
- Use table-driven tests

---

Task 2: Implement Circuit Breaker Pattern for Engine Resilience

Priority: High
Estimated Effort: Large
Focus Area: Engine Reliability

Description:
Add circuit breaker pattern to engine execution to prevent cascading failures when engines are degraded or down. This protects against thundering herd problems and wasted GitHub Actions minutes when an engine is experiencing an outage.

Acceptance Criteria:

• Create pkg/circuit/breaker.go with CircuitBreaker type supporting Closed/Open/HalfOpen states
• Implement configurable failure threshold (default: 5 failures triggers open)
• Add timeout period before transitioning to half-open (default: 30 seconds)
• Provide Execute(func() error) error method that implements state machine
• Add metrics collection (state transitions, failure count, success rate)
• Create EngineWithCircuitBreaker wrapper in pkg/workflow/engine_circuit_breaker.go
• Add configuration option in frontmatter for circuit breaker tuning
• Comprehensive tests covering all state transitions and edge cases
• Document behavior and tuning guidance in godoc

Code Region: pkg/circuit/ (new package), pkg/workflow/engine_circuit_breaker.go (new file)

Implement circuit breaker pattern to prevent cascading engine failures in gh-aw workflows.

Requirements:
1. **Circuit Breaker Core** (pkg/circuit/breaker.go):
   ```go
   type State int
   const (
       StateClosed State = iota  // Normal operation
       StateOpen                 // Failing fast
       StateHalfOpen             // Testing recovery
   )
   
   type CircuitBreaker struct {
       maxFailures   int           // Trigger open after N failures (default: 5)
       timeout       time.Duration // Half-open after timeout (default: 30s)
       onStateChange func(from, to State) // Optional callback
       
       mu            sync.RWMutex
       state         State
       failureCount  int
       lastFailTime  time.Time
       successCount  int // In half-open state
   }
   
   func New(maxFailures int, timeout time.Duration) *CircuitBreaker
   func (cb *CircuitBreaker) Execute(fn func() error) error
   func (cb *CircuitBreaker) GetState() State
   func (cb *CircuitBreaker) Reset() // Manual reset to closed
   ```

2. **State Machine Logic**:
   - **Closed**: Execute normally, count failures, open after maxFailures
   - **Open**: Fail fast with `ErrCircuitOpen`, check timeout for half-open
   - **HalfOpen**: Allow one test request, close on success, open on failure

3. **Error Types**:
   ```go
   var ErrCircuitOpen = errors.New("circuit breaker is open")
   var ErrTooManyFailures = errors.New("circuit breaker opened due to too many failures")
   ```

4. **Engine Wrapper** (pkg/workflow/engine_circuit_breaker.go):
   ```go
   type EngineWithCircuitBreaker struct {
       engine  CodingAgentEngine
       breaker *circuit.CircuitBreaker
   }
   
   func WrapWithCircuitBreaker(engine CodingAgentEngine, maxFailures int, timeout time.Duration) *EngineWithCircuitBreaker
   
   func (e *EngineWithCircuitBreaker) Execute(ctx context.Context, workflow *WorkflowData) (*ExecutionResult, error) {
       var result *ExecutionResult
       err := e.breaker.Execute(func() error {
           var execErr error
           result, execErr = e.engine.Execute(ctx, workflow)
           return execErr
       })
       return result, err
   }
   ```

5. **Frontmatter Configuration**:
   ```yaml
   engine: copilot
   circuit-breaker:
     max-failures: 5        # Open after 5 consecutive failures
     timeout: 30s           # Half-open after 30 seconds
     enabled: true          # Default: true
   ```

6. **Testing** (pkg/circuit/breaker_test.go):
   - TestCircuitBreakerClosed - Normal operation
   - TestCircuitBreakerOpens - Opens after maxFailures
   - TestCircuitBreakerHalfOpen - Transitions to half-open after timeout
   - TestCircuitBreakerClosesOnSuccess - Half-open → Closed on success
   - TestCircuitBreakerReopensOnFailure - Half-open → Open on failure
   - TestCircuitBreakerConcurrent - Thread-safety under concurrent load
   - TestCircuitBreakerManualReset - Reset() forces closed state
   - TestCircuitBreakerStateCallback - onStateChange invoked correctly

7. **Engine Integration Testing** (pkg/workflow/engine_circuit_breaker_test.go):
   - TestEngineCircuitBreakerFailsOpen - Stops calling engine when open
   - TestEngineCircuitBreakerRecovery - Closes after successful half-open test
   - TestEngineCircuitBreakerMultipleEngines - Independent breakers per engine

8. **Metrics & Observability**:
   - Log state transitions with `logger.New("circuit:engine")`
   - Track: failure count, state, last transition time
   - Provide `GetMetrics()` for monitoring

9. **Documentation**:
   - Godoc explaining circuit breaker pattern and use cases
   - Example showing 75 Copilot workflows protected from cascading failures
   - Tuning guidance based on engine characteristics

Follow existing patterns:
- Use `sync.RWMutex` for thread-safe state access
- Consistent error wrapping with `fmt.Errorf` and `%w`
- Debug logging for observability
- Table-driven tests

---

Task 3: Implement Multi-Engine Fallback Strategy

Priority: High
Estimated Effort: Large
Focus Area: Engine Reliability

Description:
Enable workflows to specify fallback engines that are automatically tried when the primary engine fails. This increases workflow success rate by utilizing engine diversity (copilot, claude, codex, custom) rather than failing on first engine failure.

Acceptance Criteria:

• Extend frontmatter schema to support fallback: [engine1, engine2] array
• Create FallbackEngine wrapper that implements CodingAgentEngine interface
• Implement fallback logic that tries engines in order until success or exhaustion
• Add error aggregation to collect all engine attempts and present unified error message
• Support per-engine retry policies (e.g., retry copilot 3x before falling back to claude)
• Validate fallback configuration at compile time (engines must exist)
• Add --fallback-enabled flag to gh aw compile for opt-in behavior
• Comprehensive tests covering success, partial failure, total failure, circular fallback detection
• Update validation code to check for circular fallback dependencies

Code Region: pkg/workflow/engine_fallback.go (new file), pkg/workflow/frontmatter_types.go (extend), pkg/workflow/engine_validation.go (extend)

Implement multi-engine fallback strategy so workflows can automatically retry failed jobs with alternate AI engines.

Requirements:
1. **Frontmatter Schema Extension** (pkg/workflow/frontmatter_types.go):
   ```go
   type FrontmatterConfig struct {
       // ... existing fields ...
       Engine         string   `yaml:"engine"`
       EngineFallback []string `yaml:"fallback,omitempty"` // Ordered list of fallback engines
   }
   ```

   Example usage:
   ```yaml
   engine: copilot
   fallback:
     - claude     # Try Claude if Copilot fails
     - codex      # Try Codex if Claude fails
     - custom     # Last resort
   ```

2. **Fallback Engine Wrapper** (pkg/workflow/engine_fallback.go):
   ```go
   type FallbackEngine struct {
       primaryEngine  CodingAgentEngine
       fallbackEngines []CodingAgentEngine
       retryPolicy    *retry.Policy // From Task 1
   }
   
   func NewFallbackEngine(primary CodingAgentEngine, fallbacks []CodingAgentEngine, policy *retry.Policy) *FallbackEngine
   
   func (e *FallbackEngine) Execute(ctx context.Context, workflow *WorkflowData) (*ExecutionResult, error) {
       engines := append([]CodingAgentEngine{e.primaryEngine}, e.fallbackEngines...)
       
       var allErrors []error
       for i, engine := range engines {
           result, err := e.executeWithRetry(ctx, engine, workflow)
           if err == nil {
               return result, nil // Success!
           }
           allErrors = append(allErrors, fmt.Errorf("%s: %w", engine.GetID(), err))
       }
       
       return nil, &FallbackExhaustedError{Attempts: allErrors}
   }
   
   func (e *FallbackEngine) executeWithRetry(ctx context.Context, engine CodingAgentEngine, workflow *WorkflowData) (*ExecutionResult, error) {
       var result *ExecutionResult
       err := retry.DoWithPolicy(ctx, e.retryPolicy, func() error {
           var execErr error
           result, execErr = engine.Execute(ctx, workflow)
           return execErr
       })
       return result, err
   }
   ```

3. **Error Aggregation**:
   ```go
   type FallbackExhaustedError struct {
       Attempts []error // Errors from each engine attempt
   }
   
   func (e *FallbackExhaustedError) Error() string {
       var sb strings.Builder
       sb.WriteString("all engines failed:\n")
       for i, err := range e.Attempts {
           sb.WriteString(fmt.Sprintf("  %d. %v\n", i+1, err))
       }
       return sb.String()
   }
   ```

4. **Validation** (pkg/workflow/engine_validation.go):
   - Validate all fallback engines exist in registry
   - Detect circular fallback dependencies (A→B→C→A)
   - Warn if fallback includes same engine as primary
   - Validate fallback array is non-empty if specified
   
   Add tests:
   - TestValidateFallbackEnginesExist
   - TestValidateFallbackCircularDependency
   - TestValidateFallbackDuplicatePrimary

5. **Compilation Integration** (pkg/workflow/compiler.go):
   - Check `EngineFallback` field during workflow compilation
   - Resolve engine IDs to `CodingAgentEngine` instances
   - Wrap primary engine with `FallbackEngine` if fallback specified
   - Add `--fallback-enabled` flag (default: true) for opt-out

6. **Testing** (pkg/workflow/engine_fallback_test.go):
   - TestFallbackEngineSuccess - Primary succeeds immediately
   - TestFallbackEngineFailoverToClaude - Primary fails, Claude succeeds
   - TestFallbackEngineFailoverToCodex - Primary+Claude fail, Codex succeeds
   - TestFallbackEngineAllFailed - All engines fail, aggregated error returned
   - TestFallbackEngineRetryPolicy - Each engine retried per policy
   - TestFallbackEngineContextCanceled - Respects context cancellation
   - TestFallbackEngineErrorAggregation - Collects all errors correctly

7. **Documentation**:
   - Godoc explaining fallback strategy and use cases
   - Example workflow showing copilot→claude→codex fallback
   - Performance implications (increased latency on failures)
   - Recommendation: Use fallback for critical workflows only

8. **Migration Path**:
   - Existing workflows without `fallback:` field work unchanged
   - Add `fallback:` to high-priority workflows first
   - Monitor success rate improvement in workflows with fallback

Follow patterns:
- Use existing `EngineRegistry` for engine lookup
- Consistent error wrapping and logging
- Respect context cancellation throughout chain
- Table-driven tests with multiple scenarios

---

Task 4: Add Structured Engine Error Types

Priority: Medium
Estimated Effort: Medium
Focus Area: Error Handling

Description:
Replace generic error returns from engine operations with structured error types that categorize failures (authentication, rate limit, timeout, network, validation). This enables intelligent retry decisions, better error messages, and actionable guidance for workflow authors.

Acceptance Criteria:

• Create pkg/workflow/engine_errors.go with EngineError type and error categories
• Define error categories: ErrCategoryAuth, ErrCategoryRateLimit, ErrCategoryTimeout, ErrCategoryNetwork, ErrCategoryValidation, ErrCategoryUnknown
• Add Retryable() bool method to indicate if error should be retried
• Add Guidance() string method providing actionable next steps for each error category
• Update all engine implementations to return structured errors
• Integrate with retry library (Task 1) to skip non-retryable errors
• Add console formatting for engine errors with color-coded severity
• Update validation code to use structured errors
• Comprehensive tests covering all error categories and guidance messages

Code Region: pkg/workflow/engine_errors.go (new file), pkg/workflow/*engine*.go (update error returns)

Create structured error types for engine failures to enable intelligent retry logic and provide actionable user guidance.

Requirements:
1. **Error Type Definitions** (pkg/workflow/engine_errors.go):
   ```go
   type ErrorCategory int
   const (
       ErrCategoryAuth ErrorCategory = iota          // Authentication/authorization failed
       ErrCategoryRateLimit                           // API rate limit exceeded (429)
       ErrCategoryTimeout                             // Request timeout
       ErrCategoryNetwork                             // Network connectivity issue
       ErrCategoryValidation                          // Invalid workflow/config
       ErrCategoryUnknown                             // Uncategorized error
   )
   
   func (c ErrorCategory) String() string {
       return [...]string{"Authentication", "RateLimit", "Timeout", "Network", "Validation", "Unknown"}[c]
   }
   
   type EngineError struct {
       Engine   string        // "copilot", "claude", "codex", "custom"
       Category ErrorCategory // Error category
       Message  string        // Human-readable error message
       Cause    error         // Underlying error (wrapped)
   }
   
   func (e *EngineError) Error() string {
       return fmt.Sprintf("%s engine %s error: %s", e.Engine, e.Category, e.Message)
   }
   
   func (e *EngineError) Unwrap() error {
       return e.Cause
   }
   
   func (e *EngineError) Retryable() bool {
       switch e.Category {
       case ErrCategoryTimeout, ErrCategoryNetwork, ErrCategoryRateLimit:
           return true // These can be retried
       case ErrCategoryAuth, ErrCategoryValidation:
           return false // These require user intervention
       default:
           return false // Unknown errors should not be retried automatically
       }
   }
   
   func (e *EngineError) Guidance() string {
       switch e.Category {
       case ErrCategoryAuth:
           return "Check that the required secret is configured in repository settings. For Copilot, ensure GITHUB_TOKEN has correct permissions."
       case ErrCategoryRateLimit:
           return "API rate limit exceeded. Wait before retrying or configure fallback to alternate engine."
       case ErrCategoryTimeout:
           return "Request timed out. Try increasing timeout-minutes in workflow configuration or check engine availability."
       case ErrCategoryNetwork:
           return "Network connectivity issue. Check if network.allowed includes required domains or retry the workflow."
       case ErrCategoryValidation:
           return "Workflow configuration is invalid. Review validation errors and fix configuration."
       default:
           return "Unexpected error occurred. Check workflow logs for details."
       }
   }
   ```

2. **Constructor Helpers**:
   ```go
   func NewEngineError(engine string, category ErrorCategory, message string, cause error) *EngineError
   func NewAuthError(engine, message string, cause error) *EngineError
   func NewRateLimitError(engine, message string, cause error) *EngineError
   func NewTimeoutError(engine, message string, cause error) *EngineError
   func NewNetworkError(engine, message string, cause error) *EngineError
   func NewValidationError(engine, message string, cause error) *EngineError
   ```

3. **Error Detection Helpers**:
   ```go
   func IsEngineError(err error) bool
   func AsEngineError(err error) (*EngineError, bool)
   func IsRetryable(err error) bool // Checks if error is retryable
   func GetCategory(err error) ErrorCategory
   ```

4. **Integration with Retry Library** (update Task 1):
   ```go
   // In pkg/retry/retry.go
   func Do(ctx context.Context, fn func() error) error {
       // ... retry loop ...
       if err != nil && !workflow.IsRetryable(err) {
           return err // Don't retry non-retryable errors
       }
       // ... backoff and retry ...
   }
   ```

5. **Engine Implementation Updates**:
   Update all engine files to return structured errors:
   - `pkg/workflow/copilot_engine.go`
   - `pkg/workflow/claude_engine.go`
   - `pkg/workflow/codex_engine.go`
   - `pkg/workflow/custom_engine.go`
   
   Example:
   ```go
   // Before:
   return nil, fmt.Errorf("copilot authentication failed: %w", err)
   
   // After:
   return nil, NewAuthError("copilot", "authentication failed", err)
   ```

6. **Console Formatting Integration**:
   ```go
   func FormatEngineError(err error) string {
       if engineErr, ok := AsEngineError(err); ok {
           return fmt.Sprintf("%s\n\n%s: %s",
               console.FormatErrorMessage(engineErr.Error()),
               console.FormatInfoMessage("Guidance"),
               engineErr.Guidance())
       }
       return console.FormatErrorMessage(err.Error())
   }
   ```

7. **Testing** (pkg/workflow/engine_errors_test.go):
   - TestEngineErrorCategories - All categories stringify correctly
   - TestEngineErrorRetryable - Auth/Validation not retryable, Network/Timeout retryable
   - TestEngineErrorGuidance - Each category provides actionable guidance
   - TestEngineErrorUnwrap - Cause is properly wrapped
   - TestIsRetryable - Helper correctly identifies retryable errors
   - TestAsEngineError - Type assertion works correctly
   - TestFormatEngineError - Console formatting includes guidance

8. **Documentation**:
   - Godoc explaining error categories and when each is used
   - Example showing migration from generic errors to structured errors
   - Decision tree for error categorization
   - Integration examples with retry and circuit breaker

Follow patterns:
- Implement `error` interface with `Error()` method
- Support error unwrapping with `Unwrap() error`
- Use `errors.Is()` and `errors.As()` for error inspection
- Consistent console formatting with existing patterns

---

Task 5: Create Engine Reliability Documentation and Decision Guide

Priority: Medium
Estimated Effort: Medium
Focus Area: Documentation

Description:
Document engine reliability patterns, best practices, and create a decision guide for engine selection. This fills critical gaps in user guidance around retry strategies, fallback configuration, timeout tuning, and troubleshooting engine failures.

Acceptance Criteria:

• Create docs/src/content/docs/engines/reliability.md covering retry, circuit breaker, fallback patterns with examples
• Create docs/src/content/docs/engines/selection-guide.md with decision tree for engine selection (copilot vs claude vs codex vs custom)
• Create docs/src/content/docs/troubleshooting/engine-failures.md runbook for common engine failure scenarios
• Document recommended timeout values per engine based on empirical data (if available) or conservative estimates
• Add docs/src/content/docs/engines/fallback-strategies.md with multi-engine examples and tradeoffs
• Create docs/src/content/docs/engines/rate-limits.md documenting known rate limits per engine and mitigation strategies
• Update main README.md with link to reliability documentation
• Add examples/ workflows demonstrating retry, circuit breaker, and fallback configurations

Code Region: docs/src/content/docs/engines/ (new directory), examples/reliability/ (new directory)

Create comprehensive documentation for AI engine reliability patterns, selection guidance, and troubleshooting.

Requirements:
1. **Reliability Patterns Guide** (docs/src/content/docs/engines/reliability.md):
   ```markdown
   # Engine Reliability Patterns
   
   ## Overview
   This guide covers reliability patterns for AI-powered workflows in gh-aw.
   
   ## Retry Strategy
   
   ### Basic Retry
   ```yaml
   engine: copilot
   retry:
     max-attempts: 3
     backoff: exponential
     initial-delay: 100ms
     max-delay: 5s
   ```
   
   ### Recommended Retry Policies by Scenario
   - **Development workflows**: 2 attempts, linear backoff (fail fast)
   - **Production workflows**: 3-5 attempts, exponential backoff with jitter
   - **Long-running workflows**: Circuit breaker + retry (prevent wasted time)
   
   ## Circuit Breaker
   
   Protects against cascading failures when engine is down.
   
   ```yaml
   engine: copilot
   circuit-breaker:
     max-failures: 5
     timeout: 30s
     enabled: true
   ```
   
   **When to use:**
   - Workflows triggered frequently (>10/hour)
   - Multiple workflows using same engine
   - Production-critical workflows
   
   ## Multi-Engine Fallback
   
   Automatically try alternate engines on failure.
   
   ```yaml
   engine: copilot
   fallback:
     - claude
     - codex
   ```
   
   **Tradeoffs:**
   - ✅ Higher success rate
   - ✅ Reduced impact of single-engine outage
   - ⚠️ Increased latency on failures
   - ⚠️ More complex to debug
   
   ## Timeout Configuration
   
   Recommended timeouts by engine (based on P95 latency):
   - Copilot: 10-20 minutes (simple tasks), 30-60 minutes (complex)
   - Claude: 15-30 minutes (simple tasks), 60-120 minutes (complex)
   - Codex: 5-15 minutes (focused tasks)
   - Custom: Depends on implementation
   
   Example:
   ```yaml
   timeout-minutes: 20
   engine: copilot
   ```
   
   ## Monitoring & Observability
   
   Track engine reliability metrics:
   - Success rate per engine
   - P50/P95/P99 latency
   - Retry attempts per workflow
   - Circuit breaker state transitions
   - Fallback success rate
   
   Use `gh aw logs --engine copilot --failed` to analyze failures.
   
   ## Best Practices
   
   1. **Start with defaults**: Built-in retry/circuit-breaker defaults work for 80% of cases
   2. **Add fallback for critical workflows**: Use fallback for production workflows
   3. **Tune timeouts conservatively**: Better to timeout early than wait indefinitely
   4. **Monitor failure patterns**: Use `gh aw audit` to identify systemic issues
   5. **Test failure scenarios**: Simulate engine failures in staging environment
   ```

2. **Engine Selection Guide** (docs/src/content/docs/engines/selection-guide.md):
   ```markdown
   # Engine Selection Guide
   
   ## Decision Tree
   
   ```
   ┌─ Need GitHub API access? ──YES──> Use Copilot (built-in GitHub tools)
   │
   ├─ Need long context window? ──YES──> Use Claude (100K+ tokens)
   │
   ├─ Need fast responses? ──YES──> Use Codex (optimized for code generation)
   │
   └─ Need custom behavior? ──YES──> Use Custom engine
   ```
   
   ## Engine Comparison
   
   | Feature | Copilot | Claude | Codex | Custom |
   |---------|---------|---------|-------|--------|
   | GitHub tools | ✅ Built-in | ❌ Requires MCP | ❌ Requires MCP | ⚙️ Configurable |
   | Context window | ~32K tokens | ~100K tokens | ~8K tokens | Varies |
   | Speed | Medium | Slow | Fast | Varies |
   | Cost | GitHub subscription | API costs | API costs | Varies |
   | Reliability | High | Medium | High | Varies |
   | Rate limits | Generous | Moderate | Moderate | Varies |
   
   ## Use Case Recommendations
   
   ### GitHub Repository Operations
   **Best choice:** Copilot
   - Native GitHub tools (issue_read, pull_request_read, etc.)
   - Authenticated automatically via GITHUB_TOKEN
   - No MCP server configuration needed
   
   ### Long Documents or Large Codebases
   **Best choice:** Claude
   - 100K+ token context window
   - Excels at summarization and analysis
   - Better for complex multi-file changes
   
   ### Fast Code Generation
   **Best choice:** Codex
   - Optimized for code generation speed
   - Good for focused, single-file changes
   - Lower latency than Claude
   
   ### Custom Workflows
   **Best choice:** Custom engine
   - Full control over AI backend
   - Can use local models or custom APIs
   - Requires custom setup
   
   ## Migration Between Engines
   
   Most workflows can switch engines with minimal changes:
   
   ```yaml
   # Before (Copilot)
   engine: copilot
   
   # After (Claude with GitHub tools via MCP)
   engine: claude
   tools:
     github:
       mode: remote
       toolsets: [default]
   ```
   ```

3. **Troubleshooting Engine Failures** (docs/src/content/docs/troubleshooting/engine-failures.md):
   ```markdown
   # Troubleshooting Engine Failures
   
   ## Common Failure Scenarios
   
   ### Authentication Failures
   
   **Symptoms:** "401 Unauthorized" or "403 Forbidden" errors
   
   **Copilot:**
   - Check GITHUB_TOKEN has required permissions
   - Verify token not expired
   - Ensure repository has Copilot access
   
   **Claude:**
   - Check ANTHROPIC_API_KEY secret is configured
   - Verify API key is valid and not expired
   
   **Codex:**
   - Check OPENAI_API_KEY secret is configured
   - Verify API key has correct permissions
   
   ### Rate Limit Errors
   
   **Symptoms:** "429 Too Many Requests" errors
   
   **Immediate fix:**
   1. Wait for rate limit to reset (check retry-after header)
   2. Add retry with exponential backoff
   3. Configure fallback to alternate engine
   
   **Long-term fix:**
   - Add circuit breaker to prevent hammering API
   - Distribute workflows across multiple engines
   - Increase delay between workflow runs
   
   ### Timeout Errors
   
   **Symptoms:** Workflow fails with "timeout exceeded"
   
   **Troubleshooting:**
   1. Check if timeout is too aggressive (increase timeout-minutes)
   2. Verify engine is not degraded (check status page)
   3. Consider splitting complex workflow into smaller steps
   
   **Recommended timeouts:**
   - Simple tasks: 10-20 minutes
   - Complex tasks: 30-60 minutes
   - Very complex tasks: Use fallback instead of long timeout
   
   ### Network Errors
   
   **Symptoms:** "connection refused", "network unreachable"
   
   **Troubleshooting:**
   1. Check network.allowed includes required domains
   2. Verify firewall rules allow engine API access
   3. Check GitHub Actions network status
   
   ### Workflow Debugging Commands
   
   ```bash
   # View recent failures for specific engine
   gh aw logs --engine copilot --failed
   
   # Audit specific workflow run
   gh aw audit (run-id)
   
   # Check engine configuration
   gh aw compile --validate-only workflow.md
   ```
   ```

4. **Fallback Strategies** (docs/src/content/docs/engines/fallback-strategies.md):
   ```markdown
   # Multi-Engine Fallback Strategies
   
   ## Basic Fallback
   
   ```yaml
   engine: copilot
   fallback:
     - claude
   ```
   
   ## Advanced Fallback Patterns
   
   ### GitHub-First with LLM Fallback
   ```yaml
   engine: copilot  # Try Copilot with GitHub tools first
   fallback:
     - claude       # Fall back to Claude with MCP
   tools:
     github:
       mode: remote
   ```
   
   ### Speed-Optimized with Capability Fallback
   ```yaml
   engine: codex    # Fast for simple tasks
   fallback:
     - copilot      # More capable for complex tasks
     - claude       # Most capable, slowest
   ```
   
   ### Cost-Optimized
   ```yaml
   engine: codex    # Cheapest option
   fallback:
     - copilot      # Medium cost
     # claude not included - avoid high API costs
   ```
   ```

5. **Rate Limits Documentation** (docs/src/content/docs/engines/rate-limits.md):
   ```markdown
   # Engine Rate Limits
   
   ## Known Rate Limits
   
   ### Copilot
   - **Request limit:** Varies by GitHub subscription tier
   - **Retry-after:** Typically 60 seconds
   - **Mitigation:** Use GitHub Enterprise for higher limits
   
   ### Claude (Anthropic API)
   - **Request limit:** Varies by API tier
   - **Token limit:** 200K tokens/minute (typical)
   - **Mitigation:** Implement exponential backoff, circuit breaker
   
   ### Codex (OpenAI API)
   - **Request limit:** Varies by API tier
   - **Token limit:** Varies by model
   - **Mitigation:** Use fallback to alternate engine
   
   ## Rate Limit Handling
   
   Best practices:
   1. Always configure retry with exponential backoff
   2. Add circuit breaker for high-frequency workflows
   3. Use fallback to distribute load across engines
   4. Monitor rate limit errors and adjust strategy
   ```

6. **Example Workflows** (examples/reliability/):
   - `retry-basic.md` - Simple retry configuration
   - `circuit-breaker.md` - Circuit breaker example
   - `multi-engine-fallback.md` - Fallback between Copilot and Claude
   - `production-ready.md` - Complete reliability stack (retry + breaker + fallback)

7. **Update Main README**:
   Add section:
   ```markdown
   ## Engine Reliability
   
   gh-aw supports advanced reliability patterns for AI-powered workflows:
   - **Retry with exponential backoff** - Automatic retry on transient failures
   - **Circuit breaker** - Protect against cascading failures
   - **Multi-engine fallback** - Automatic failover to alternate engines
   
   See [Engine Reliability Guide](docs/src/content/docs/engines/reliability.md) for details.
   ```

Follow documentation standards:
- Use Diátaxis framework (How-to, Explanation, Reference, Tutorial)
- GitHub-flavored markdown
- Code examples for every pattern
- Clear decision criteria and tradeoffs

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2026-01-29	Workflow Health Monitoring & Observability	Custom	Y	Real-time monitoring gaps identified
2026-01-28	Error Experience Engineering	Custom	Y	Console formatting +57%, debug logs +1,109%
2026-01-26	Example Workflow Portfolio Quality	Custom	Y	Only 1.5% workflows in examples/, 65 undocumented
2026-01-23	MCP Server Integration Quality	Custom	Y	84.3% MCP adoption, consistency crisis identified
2026-01-22	Validation Message Clarity	Custom	Y	Only 29% errors include examples, 0 console formatting
2026-01-21	Dependencies	Standard	N	278 dependencies, 48.2% on v0.x, zero vuln scanning
2026-01-20	Command Interface Consistency	Custom	Y	Only 12% implement RunX pattern, 0% provide 3+ examples
2026-01-16	Workflow Compilation Performance	Custom	Y	177 workflows sequential, zero parallel compilation
2026-01-08	Security	Standard	N	98.6% workflows have permissions, no CodeQL/gosec
2026-01-07	Error Experience Engineering	Custom	Y	Only 3% errors use console formatting (first run)
2026-01-06	Testing	Standard	N	2.24:1 test ratio, 320 skipped tests
2026-01-05	Engine Configuration Best Practices	Custom	Y	32.6% missing explicit engine (first focus area)

---

🎯 Recommendations

Immediate Actions (This Week)

1. Implement Retry Infrastructure (Task 1) - Priority: High

   • Eliminates 32+ ad-hoc retry implementations
   • Provides foundation for Tasks 2 & 3
   • Quick win with high impact

2. Create Engine Reliability Documentation (Task 5) - Priority: Medium

   • Provides immediate value to workflow authors
   • No code changes required
   • Clarifies 71 workflows missing engine specification

Short-term Actions (This Month)

1. Implement Circuit Breaker Pattern (Task 2) - Priority: High

   • Protects 75 Copilot workflows from cascading failures
   • Reduces wasted GitHub Actions minutes
   • Requires Task 1 (retry library) as prerequisite

2. Add Structured Error Types (Task 4) - Priority: Medium

   • Enables intelligent retry decisions
   • Improves error messages and user guidance
   • Integrates with Tasks 1 & 2

Long-term Actions (This Quarter)

1. Implement Multi-Engine Fallback (Task 3) - Priority: High
   • Increases workflow success rate significantly
   • Leverages engine diversity (copilot, claude, codex, custom)
   • Requires Tasks 1, 2, and 4 as prerequisites

---

📈 Success Metrics

Track these metrics to measure improvement in AI Engine Reliability:

• Workflow Success Rate: [Current: Unknown] → [Target: 95%+ with fallback enabled]
• Average Retry Attempts: [Current: Ad-hoc] → [Target: <2 attempts per workflow]
• Circuit Breaker Activations: [Current: 0] → [Target: Track per engine, aim for <5% open rate]
• Fallback Success Rate: [Current: N/A] → [Target: 80%+ workflows succeed via fallback]
• Mean Time to Recovery: [Current: Unknown] → [Target: <5 minutes with circuit breaker]
• Engine Timeout Rate: [Current: Unknown] → [Target: <5% workflows timeout]
• Rate Limit Errors: [Current: Unknown] → [Target: <2% workflows hit rate limits]

---

Next Steps

1. Review and prioritize the 5 tasks above based on team capacity and priorities
2. Assign Task 1 (Retry Infrastructure) to Copilot agent via planner agent as foundation
3. Create engine reliability documentation (Task 5) in parallel for immediate user value
4. Implement Tasks 2-4 sequentially as prerequisites are completed
5. Track metrics after each task to measure impact on workflow success rate
6. Re-evaluate this focus area in 2-3 months to assess improvements

---

Generated by Repository Quality Improvement Agent
Run ID: §21517468062
Next analysis: 2026-01-31 - Focus area will be selected based on diversity algorithm (aim for custom area exploring new dimension like "Release Process Automation", "Contributor Onboarding Journey", or "Cross-Platform Testing Quality")

AI generated by Repository Quality Improvement Agent

• expires on Feb 6, 2026, 1:41 PM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-02-06T13:41:13.275Z.