User Experience Analysis - February 17, 2026

[github-actions[bot]]

Executive Summary

Today's analysis focused on:

• 2 documentation files (Quick Start, Engines reference)
• 2 CLI commands (compile, run)
• 1 workflow message configuration (Auto-Triage Issues)
• 1 validation file (Agent validation)

Overall Quality: ⚠️ Professional with opportunities for improvement

Key Finding: Documentation and CLI help text contain comprehensive information but could benefit from clearer visual hierarchy and more concise initial descriptions to improve scanability for enterprise users.

---

Quality Highlights ✅

Example 1: Comprehensive CLI Examples

• File: cmd/gh-aw/main.go (compile command, lines 234-243)
• What works well: The compile command provides 9 concrete examples covering common use cases, advanced flags, and different input formats. This exceeds enterprise standards and helps users quickly find the pattern they need.
• Quote/Reference:

  Examples:
    gh aw compile                    # Compile all Markdown files
    gh aw compile ci-doctor    # Compile a specific workflow
    gh aw compile ci-doctor daily-plan  # Compile multiple workflows
    gh aw compile workflow.md        # Compile by file path
    gh aw compile --dir custom/workflows  # Compile from custom directory
    gh aw compile --watch ci-doctor     # Watch and auto-compile
    gh aw compile --trial --logical-repo owner/repo  # Compile for trial mode
    gh aw compile --dependabot        # Generate Dependabot manifests
    gh aw compile --dependabot --force  # Force overwrite existing dependabot.yml
  

Example 2: Clear Formatting Guidelines for AI Agents

• File: .github/workflows/auto-triage-issues.md (lines 48-100)
• What works well: The workflow provides explicit, detailed formatting guidelines for the AI agent with clear rationale, concrete examples, and specific structure patterns. This ensures consistent, professional output.
• Quote/Reference: "Use h3 (###) or lower for all headers in triage reports to maintain proper document hierarchy" with complete example structure pattern

---

Improvement Opportunities 💡

High Priority

Opportunity 1: Improve Quick Start Prerequisites Clarity - Single File Improvement

• File: docs/src/content/docs/setup/quick-start.mdx
• Current State (Lines 26-34): Prerequisites list includes "AI Account" as first bullet but lacks clarity on whether ALL three services are needed or just ONE
• Issue: Ambiguous "OR" relationship between AI services could cause user confusion about requirements
• User Impact: New users may waste time setting up multiple AI services when only one is needed, or may be unsure if they meet prerequisites
• Suggested Change: Make the OR relationship explicit and reorder to prioritize the most common choice

Before (lines 28-30):

- **AI Account** - [GitHub Copilot](https://github.com/features/copilot), [Anthropic Claude]((www.anthropic.com/redacted) or [OpenAI Codex]((openai.com/redacted)

After:

- **AI Account** - Choose ONE of the following:
  - [GitHub Copilot](https://github.com/features/copilot) (recommended)
  - [Anthropic Claude]((www.anthropic.com/redacted)
  - [OpenAI Codex]((openai.com/redacted)

Why This Matters

• User Impact: Eliminates ambiguity during initial setup, reducing time-to-first-success
• Quality Factor: Clarity and precision - makes requirements explicit
• Frequency: Every new user encounters this section

Success Criteria

• Changes made to docs/src/content/docs/setup/quick-start.mdx only
• Prerequisites section clearly indicates "choose ONE" requirement
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: docs/src/content/docs/setup/quick-start.mdx
• No changes to other files required
• Can be completed independently

---

Opportunity 2: Simplify Engines Documentation Opening - Single File Improvement

• File: docs/src/content/docs/reference/engines.md
• Current State (Lines 1-18): Opening section contains technical jargon in the first sentence that may confuse users ("AI Engines" vs "coding agents" terminology)
• Issue: The dual terminology (AI Engines / coding agents) appears in the first sentence without clear definition, potentially confusing users about what these terms mean
• User Impact: Enterprise users scanning documentation may be unclear whether "AI Engine" and "coding agent" refer to the same concept or different components
• Suggested Change: Lead with clear, direct language and defer technical terminology until after establishing context

Before (lines 1-14):

---
title: AI Engines (aka Coding Agents)
description: Complete guide to AI engines (coding agents) usable with GitHub Agentic Workflows...
---

GitHub Agentic Workflows use [AI Engines](/gh-aw/reference/glossary/#engine) (normally a coding agent) to interpret and execute natural language instructions. Each coding agent has unique capabilities and configuration options.

## Available Coding Agents

- [**Copilot CLI**](#using-copilot-cli)
- [**Claude by Anthropic (Claude Code)**](#using-claude-by-anthropic-claude-code)
- [**OpenAI Codex**](#using-openai-codex)

After:

---
title: AI Engines
description: Complete guide to AI engines (Copilot, Claude, Codex) for GitHub Agentic Workflows, including configuration and setup.
---

GitHub Agentic Workflows execute natural language instructions using AI-powered coding agents. You configure which agent to use via the `engine` field in your workflow frontmatter.

## Supported Engines

GitHub Agentic Workflows supports these AI coding agents:

- [**Copilot CLI**](#using-copilot-cli) (default, recommended)
- [**Claude by Anthropic**](#using-claude-by-anthropic-claude-code)
- [**OpenAI Codex**](#using-openai-codex)

Each engine has specific authentication requirements and optional configuration detailed below.

Why This Matters

• User Impact: Users can immediately understand what engines are and which options are available
• Quality Factor: Efficiency and productivity - reduces cognitive load, clear information hierarchy
• Frequency: High-traffic reference page for workflow configuration

Success Criteria

• Changes made to docs/src/content/docs/reference/engines.md only
• Opening paragraph uses consistent terminology
• Recommendation for Copilot CLI is explicit
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: docs/src/content/docs/reference/engines.md
• No changes to other files required
• Can be completed independently

---

Medium Priority

Opportunity 3: Enhance Validation Error Message Specificity - Single File Improvement

• File: pkg/workflow/agent_validation.go
• Current State (Lines 85-86): Error message for missing agent file is clear but could provide more actionable guidance
• Issue: Error message states "Ensure the file exists in the repository and is properly imported" but doesn't explain what "properly imported" means or where the file should be located
• User Impact: Users may not understand the correct directory structure or import syntax when encountering this error
• Suggested Change: Provide specific location guidance and example in the error message

Before (line 86):

fmt.Sprintf("agent file '%s' does not exist. Ensure the file exists in the repository and is properly imported.", agentPath), nil)

After:

fmt.Sprintf("agent file '%s' does not exist. Agent files must be located in the .github/agents/ directory (e.g., .github/agents/%s). Ensure the file exists and is referenced correctly in your workflow's frontmatter imports.", agentPath, filepath.Base(agentPath)), nil)

Why This Matters

• User Impact: Reduces troubleshooting time by providing explicit location requirements
• Quality Factor: Trust and reliability - actionable error messages with clear solutions
• Frequency: Occurs whenever users create custom agent configurations

Success Criteria

• Changes made to pkg/workflow/agent_validation.go only
• Error message includes expected directory location
• Error message includes concrete example
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: pkg/workflow/agent_validation.go
• No changes to other files required
• Can be completed independently

---

Files Reviewed

Documentation

• docs/src/content/docs/setup/quick-start.mdx - Rating: ⚠️ (comprehensive but prerequisites need clarity)
• docs/src/content/docs/reference/engines.md - Rating: ⚠️ (complete but opening needs simplification)

CLI Commands

• compile command (main.go lines 216-318) - Rating: ✅ (excellent examples, clear structure)
• run command (main.go lines 345-413) - Rating: ✅ (comprehensive help text with good examples)

Workflow Messages

• .github/workflows/auto-triage-issues.md - Rating: ✅ (excellent formatting guidelines and structure)

Validation Code

• pkg/workflow/agent_validation.go - Rating: ⚠️ (clear validation but error messages could be more specific)

---

Metrics

• Files Analyzed: 6
• Quality Distribution:
  • ✅ Professional: 3 (50%)
  • ⚠️ Needs Minor Work: 3 (50%)
  • ❌ Needs Significant Work: 0 (0%)

---

🎯 Actionable Tasks

Here are 3 targeted improvement tasks, each affecting a single file:

Task 1: Clarify Prerequisites OR Relationship - Improve Quick Start

File to Modify: docs/src/content/docs/setup/quick-start.mdx

Current Experience (Lines 26-34)

The prerequisites section lists three AI services with "or" in the text, but the formatting doesn't make it immediately clear that users need to choose ONE option, not all three.

Quality Issue

Design Principle: Clarity and Precision

The current presentation could lead new users to believe they need accounts for all three services, or leave them uncertain about which option to choose. Enterprise users value explicit requirements to minimize setup time.

Proposed Improvement

Make the "choose ONE" requirement explicit through formatting and add a recommendation for the most common choice.

Before:

- **AI Account** - [GitHub Copilot](https://github.com/features/copilot), [Anthropic Claude]((www.anthropic.com/redacted) or [OpenAI Codex]((openai.com/redacted)

After:

- **AI Account** - Choose ONE of the following:
  - [GitHub Copilot](https://github.com/features/copilot) (recommended)
  - [Anthropic Claude]((www.anthropic.com/redacted)
  - [OpenAI Codex]((openai.com/redacted)

Why This Matters

• User Impact: Eliminates ambiguity during initial setup, reduces time-to-first-success
• Quality Factor: Clarity and precision in requirements
• Frequency: Every new user encounters this during onboarding

Success Criteria

• Changes made to docs/src/content/docs/setup/quick-start.mdx only
• Prerequisites section clearly indicates "choose ONE" requirement
• Copilot is marked as recommended option
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: docs/src/content/docs/setup/quick-start.mdx
• No changes to other files required
• Can be completed independently

---

Task 2: Simplify Engines Documentation Opening - Improve Reference Page

File to Modify: docs/src/content/docs/reference/engines.md

Current Experience (Lines 1-18)

The opening section uses dual terminology ("AI Engines" and "coding agents") without clear definition, and references a glossary link before establishing core concepts. This creates unnecessary cognitive load for users trying to understand the basics.

Quality Issue

Design Principle: Efficiency and Productivity

Enterprise users scanning reference documentation need to quickly understand: (1) what engines are, (2) which options exist, and (3) which one to use. The current opening requires parsing technical terminology before delivering these basics.

Proposed Improvement

Lead with clear, direct language. Establish what engines do, list available options with a recommendation, then provide detailed configuration.

Before:

---
title: AI Engines (aka Coding Agents)
description: Complete guide to AI engines (coding agents) usable with GitHub Agentic Workflows...
---

GitHub Agentic Workflows use [AI Engines](/gh-aw/reference/glossary/#engine) (normally a coding agent) to interpret and execute natural language instructions. Each coding agent has unique capabilities and configuration options.

## Available Coding Agents

- [**Copilot CLI**](#using-copilot-cli)
- [**Claude by Anthropic (Claude Code)**](#using-claude-by-anthropic-claude-code)
- [**OpenAI Codex**](#using-openai-codex)

After:

---
title: AI Engines
description: Complete guide to AI engines (Copilot, Claude, Codex) for GitHub Agentic Workflows, including configuration and setup.
---

GitHub Agentic Workflows execute natural language instructions using AI-powered coding agents. You configure which agent to use via the `engine` field in your workflow frontmatter.

## Supported Engines

GitHub Agentic Workflows supports these AI coding agents:

- [**Copilot CLI**](#using-copilot-cli) (default, recommended)
- [**Claude by Anthropic**](#using-claude-by-anthropic-claude-code)
- [**OpenAI Codex**](#using-openai-codex)

Each engine has specific authentication requirements and optional configuration detailed below.

Why This Matters

• User Impact: Users can immediately understand what engines are and make an informed choice
• Quality Factor: Efficiency - reduces cognitive load with clear information hierarchy
• Frequency: High-traffic reference page consulted during workflow configuration

Success Criteria

• Changes made to docs/src/content/docs/reference/engines.md only
• Opening paragraph uses consistent, simple terminology
• Copilot CLI is explicitly marked as default and recommended
• Technical jargon deferred until after core concepts established
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: docs/src/content/docs/reference/engines.md
• No changes to other files required
• Can be completed independently

---

Task 3: Enhance Agent Validation Error Message - Improve Error Guidance

File to Modify: pkg/workflow/agent_validation.go

Current Experience (Lines 84-91)

When a custom agent file is missing, the error message states "Ensure the file exists in the repository and is properly imported" but doesn't specify where the file should be located or what "properly imported" means.

Quality Issue

Design Principle: Trust and Reliability

Enterprise users encountering errors need specific, actionable guidance. The current message requires users to search documentation or code to understand the expected file location and structure.

Proposed Improvement

Include the expected directory location (.github/agents/) and a concrete example in the error message.

Before (line 86):

fmt.Sprintf("agent file '%s' does not exist. Ensure the file exists in the repository and is properly imported.", agentPath), nil)

After:

fmt.Sprintf("agent file '%s' does not exist. Agent files must be located in the .github/agents/ directory (e.g., .github/agents/%s). Ensure the file exists and is referenced correctly in your workflow's frontmatter imports.", agentPath, filepath.Base(agentPath)), nil)

Why This Matters

• User Impact: Reduces troubleshooting time by providing explicit location requirements and example
• Quality Factor: Clear error messages with actionable solutions
• Frequency: Occurs whenever users create custom agent configurations

Success Criteria

• Changes made to pkg/workflow/agent_validation.go only
• Error message includes expected directory location (.github/agents/)
• Error message includes concrete example path
• Error message explains what "properly referenced" means
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: pkg/workflow/agent_validation.go
• No changes to other files required
• Can be completed independently

---

Analysis Date: 2026-02-17
Workflow Run: §22096798518

AI generated by Delight

• expires on Feb 24, 2026, 11:37 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-02-24T11:37:22.498Z.

Closed by Workflow