User Experience Analysis Report - 2026-02-12

[github-actions[bot]]

Executive Summary

Today's analysis focused on:

• 2 documentation files (Quick Start, MCP Getting Started)
• 2 CLI commands (compile, logs)
• 2 workflow message configurations (delight, smoke-copilot)
• 2 validation files (engine_validation.go, permissions_validation.go)

Overall Quality: ✅ Professional - The codebase demonstrates strong enterprise software design with clear documentation, comprehensive help text, and well-structured error messages.

Key Finding: Documentation provides excellent technical depth but could enhance user confidence with more explicit success criteria and expected outcomes at key decision points.

---

Quality Highlights ✅

Example 1: Comprehensive CLI Help Text

• File: pkg/cli/logs_command.go
• What works well:
  • Extensive documentation with practical orchestrator usage examples (lines 48-78)
  • Multiple concrete examples covering common scenarios (lines 82-100)
  • Clear artifact descriptions help users understand output structure
  • Professional, technical tone appropriate for enterprise users
• Quote/Reference: "Orchestrator Usage: In an orchestrator workflow, use this command in a pre-step to download logs, then access the data in subsequent steps without needing GitHub CLI access" - This directly addresses a common enterprise workflow pattern with actionable guidance.

Example 2: Structured Error Messages with Context

• File: pkg/workflow/engine_validation.go
• What works well:
  • Clear problem statement with valid options listed (line 82-85)
  • "Did you mean" suggestions for typos (lines 88-93)
  • Concrete example showing correct usage format
  • Documentation link for deeper information
• Quote/Reference: Lines 82-85 demonstrate professional error handling: "invalid engine: %s. Valid engines are: %s.\n\nExample:\nengine: copilot\n\nSee: %s"

---

Improvement Opportunities 💡

High Priority

Opportunity 1: Enhance Quick Start Success Criteria - Single File Improvement

• File: docs/src/content/docs/setup/quick-start.mdx
• Current State: The guide explains what to do (install, add workflow, wait) but doesn't clearly set expectations for what success looks like at each step
  • Line 36: Installation step shows command but not how to verify success
  • Line 70-73: "Wait for workflow to complete" section lacks specific success indicators
  • Line 86-89: Customization section doesn't explain what success looks like after recompilation
• Issue: Users may be unsure if they've completed steps correctly, leading to hesitation and reduced confidence in the process
• User Impact: First-time users of enterprise software expect clear feedback loops. Ambiguity about success creates friction and increases support burden, particularly for users unfamiliar with GitHub Actions workflows.
• Suggested Change: Add explicit success criteria after each major step:

Before (lines 36-47):

### Step 1 - Install the extension

Install the GitHub Agentic Workflows extension:

```text wrap
gh extension install github/gh-aw

If you are encountering authentication issues, use this script instead:

curl -sL https://raw.githubusercontent.com/github/gh-aw/main/install-gh-aw.sh | bash

**After:**
```markdown
### Step 1 - Install the extension

Install the GitHub Agentic Workflows extension:

```text wrap
gh extension install github/gh-aw

Success indicator: You should see ✓ Installed extension github/gh-aw confirming installation. Verify with:

gh aw --version

If you are encountering authentication issues, use this script instead:

curl -sL https://raw.githubusercontent.com/github/gh-aw/main/install-gh-aw.sh | bash

**Why This Matters**
- **User Impact**: Reduces uncertainty and builds confidence at critical decision points during onboarding
- **Quality Factor**: Enhances trust and reliability - users know they're on the right path
- **Frequency**: Every new user encounters this guide; improving it has maximum leverage
- **Design Principle**: Trust and Reliability - "Clear expectations and predictable behavior"

**Success Criteria**
- [x] Changes made to `docs/src/content/docs/setup/quick-start.mdx` only
- [x] Added success indicators after Step 1, Step 2 completion, and Step 4 compilation
- [x] Quality rating improves from ⚠️ Needs Minor Work to ✅ Professional

**Scope Constraint**
- **Single file only**: `docs/src/content/docs/setup/quick-start.mdx`
- No changes to other files required
- Can be completed independently

---

#### Opportunity 2: Improve MCP Error Message Actionability - Single File Improvement
- **File**: `docs/src/content/docs/guides/getting-started-mcp.md`
- **Current State**: The troubleshooting section (lines 316-326) lists common errors but provides generic advice rather than specific resolution steps
  - Line 318: "Tool not found: Run `gh aw mcp inspect my-workflow`" - doesn't explain what to do with inspection results
  - Line 320: "Authentication errors: Verify the secret exists" - doesn't specify which secret or where to check
  - Line 322: "Connection failures: Check URL syntax" - too vague for users troubleshooting production issues
- **Issue**: Enterprise users expect actionable guidance that maps specific symptoms to concrete resolution steps
- **User Impact**: Troubleshooting production MCP integration issues requires precise guidance. Generic advice increases mean time to resolution and may require escalation to support.
- **Suggested Change**: Transform generic troubleshooting entries into step-by-step diagnostic procedures:

**Before (lines 316-326):**
```markdown
## Troubleshooting

**Tool not found:** Run `gh aw mcp inspect my-workflow` to verify available tools. Ensure the correct toolset is enabled or that tool names in `allowed:` match exactly.

**Authentication errors:** Verify the secret exists in repository settings and has required scopes. For remote mode, set `GH_AW_GITHUB_TOKEN` with a PAT.

**Connection failures:** Check URL syntax for HTTP servers, network configuration for containers, and verify Docker images exist.

**Validation errors:** Check YAML syntax, ensure `toolsets:` uses array format (`[default]` not `default`), and verify `allowed:` is an array.

After:

## Troubleshooting

### Tool Not Found Error

**Symptom**: Workflow fails with "tool 'create_issue' not found" or similar

**Resolution steps**:
1. Inspect available tools: `gh aw mcp inspect my-workflow`
2. Check the output under "Available tools" section
3. If using `toolsets:`, verify toolset name matches exactly (case-sensitive)
4. If using `allowed:`, verify tool name matches inspection output exactly
5. Example fix: Change `toolsets: default` to `toolsets: [default]` (array format required)

### Authentication Errors

**Symptom**: "401 Unauthorized" or "Bad credentials" in workflow logs

**Resolution steps**:
1. Navigate to repository Settings → Secrets and variables → Actions
2. Verify `GITHUB_TOKEN` or `GH_AW_GITHUB_TOKEN` exists
3. For remote mode: Create Personal Access Token with scopes: `repo`, `read:org`
4. Add as secret: `GH_AW_GITHUB_TOKEN`
5. Reference in workflow: `github-token: "$\{\{ secrets.GH_AW_GITHUB_TOKEN }}"`

### Connection Failures

**Symptom**: "Failed to connect to MCP server" or timeout errors

**Resolution for HTTP servers**:
1. Verify URL uses `https://` (not `(redacted)
2. Check server is accessible: `curl -I (url)`
3. Ensure network allows egress to domain (add to `network.allowed` if needed)

**Resolution for Docker containers**:
1. Verify image exists: `docker pull (container-name)`
2. Check container runs locally: `docker run (container-name)`
3. Review workflow logs for container startup errors

### Validation Errors

**Symptom**: "Compilation failed" with YAML syntax errors

**Common fixes**:
- `toolsets: default` → `toolsets: [default]` (array required)
- `allowed: "tool_name"` → `allowed: ["tool_name"]` (array required)
- Check indentation (use 2 spaces, not tabs)
- Validate YAML: `gh aw compile my-workflow --validate --strict`

Why This Matters

• User Impact: Reduces mean time to resolution for production issues; enables self-service troubleshooting
• Quality Factor: Enhances efficiency and productivity - direct paths to solutions without trial-and-error
• Frequency: MCP configuration errors are common during initial setup and after changes
• Design Principle: Trust and Reliability - "Clear error messages with actionable solutions"

Success Criteria

• Changes made to docs/src/content/docs/guides/getting-started-mcp.md only
• Each error type has symptom description and numbered resolution steps
• Includes specific commands and configuration examples
• Quality rating improves from ⚠️ Needs Minor Work to ✅ Professional

Scope Constraint

• Single file only: docs/src/content/docs/guides/getting-started-mcp.md
• No changes to other files required
• Can be completed independently

---

Files Reviewed

Documentation

• docs/src/content/docs/setup/quick-start.mdx - Rating: ⚠️ Needs Minor Work (missing success criteria)
• docs/src/content/docs/guides/getting-started-mcp.md - Rating: ⚠️ Needs Minor Work (generic troubleshooting)

CLI Commands

• pkg/cli/compile_command.go - Rating: ✅ Professional (well-structured modular design)
• pkg/cli/logs_command.go - Rating: ✅ Professional (comprehensive documentation)

Workflow Messages

• .github/workflows/delight.md - Rating: ✅ Professional (clear, appropriate emoji use)
• .github/workflows/smoke-copilot.md - Rating: ✅ Professional (creative but consistent tone)

Validation Code

• pkg/workflow/engine_validation.go - Rating: ✅ Professional (excellent error messages)
• pkg/workflow/permissions_validation.go - Rating: ✅ Professional (well-documented, type-safe)

---

Metrics

• Files Analyzed: 8
• Quality Distribution:
  • ✅ Professional: 6 (75%)
  • ⚠️ Needs Minor Work: 2 (25%)
  • ❌ Needs Significant Work: 0 (0%)

---

Design Principles Applied

This analysis evaluated files against five core enterprise software design principles:

1. Clarity and Precision - Documentation is technically accurate and comprehensive
2. Professional Communication - Tone is consistently appropriate for enterprise users
3. Efficiency and Productivity - CLI help text and validation errors enable quick resolution
4. Trust and Reliability - ⚠️ Opportunity: Success criteria and troubleshooting steps need enhancement
5. Documentation Quality - Strong technical depth; could improve practical guidance at decision points

---

Next Steps: The two improvement opportunities above represent focused, single-file changes that will significantly enhance user confidence and troubleshooting efficiency for the most frequently-accessed documentation.

AI generated by Delight

• expires on Feb 19, 2026, 9:27 PM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-02-19T21:27:40.612Z.

Closed by Workflow