UX Analysis Report — 2026-02-20

[github-actions[bot]]

Today's analysis focused on:

• 2 documentation files
• 1 CLI command help text
• 2 workflow message configurations
• 1 validation file

Overall Quality: Predominantly professional with two targeted improvement opportunities identified.

Key Finding: The gh aw logs --help text embeds orchestrator workflow YAML step examples and runtime instructions ("DO NOT call 'gh aw logs' in your environment") that belong in documentation, not CLI help — creating a noisy, hard-to-scan output for users who just want to understand command flags.

---

Quality Highlights ✅

Highlight 1: MCP Getting Started Guide

• File: docs/src/content/docs/guides/getting-started-mcp.md
• What works well: Clear progressive structure (Quick Start → Configuration Patterns → Practical Examples → Troubleshooting). Tables summarize toolsets compactly. Three complete, copy-paste-ready workflow examples cover 80% of use cases. Troubleshooting section uses concrete error messages, not vague categories.
• Reference: "Get your first MCP integration running in under 5 minutes" — sets a concrete, testable expectation.

Highlight 2: Engine Validation Error Messages

• File: pkg/workflow/engine_validation.go
• What works well: Error messages combine the invalid value, the full list of valid options, a "did you mean" suggestion, an inline example, and a documentation URL. This is the ideal error message structure for enterprise users.
• Reference: "invalid engine: %s. Valid engines are: %s.\n\nDid you mean: %s?\n\nExample:\nengine: copilot\n\nSee: %s"

---

Improvement Opportunities 💡

High Priority

Opportunity 1: gh aw logs --help embeds orchestrator workflow instructions — Single File Improvement

• File: pkg/cli/logs_command.go
• Current State (lines 47–77): The Long description contains a multi-paragraph "Orchestrator Usage" section with raw GitHub Actions YAML step examples, markdown-formatted bold headings, and the instruction **DO NOT call 'gh aw logs' or any GitHub CLI commands** — they will not work in your environment.
• Issue: This content is designed as guidance for AI agents running inside orchestrator workflows, not for humans running --help. It makes the help output ~80 lines longer, mixes two different audiences (CLI users vs. workflow AI agents), and the "DO NOT use this tool" note is confusing inside the tool's own help text.
• User Impact: Enterprise users running gh aw logs --help to understand flags must scroll past 30 lines of orchestrator-internal guidance before reaching the Examples section. Reduces trust that the help text is their primary resource.
• Suggested Change: Remove the "Orchestrator Usage" and "Live Tracking with Project Boards" sections from the Long description. Move this guidance to the documentation site or inline comments in example workflows. The CLI help should cover what the command does and how to use its flags.
• Design Principle: Efficiency and Productivity — minimize cognitive load, provide direct paths to outcomes.

Medium Priority

Opportunity 2: comment-triggered.md Quick Start is incomplete — Single File Improvement

• File: docs/src/content/docs/examples/comment-triggered.md
• Current State (lines 38–46): The Quick Start section shows a single gh aw add-wizard command and then ends with "Then trigger it by commenting /review on a pull request!" — with no explanation of what the wizard does, what files it creates, or what a complete command-triggered workflow looks like.
• Issue: Users landing on this page leave without understanding the mental model of slash commands. The example command triggers (on: slash_command) shown in lines 22–35 are shown as YAML snippets without a complete workflow context, making it unclear how to use them.
• User Impact: First-time users trying to implement ChatOps must navigate to the ChatOps pattern page to see a complete example. The hub page provides no standalone value.
• Suggested Change: Add a complete, minimal working example of a slash command workflow (frontmatter + body) directly on this page, similar to how getting-started-mcp.md shows a complete Quick Start workflow inline.
• Design Principle: Documentation Quality — practical examples that reflect real use cases.

---

Files Reviewed

Documentation

• docs/src/content/docs/guides/getting-started-mcp.md — Rating: ✅ Professional
• docs/src/content/docs/examples/comment-triggered.md — Rating: ⚠️ Needs Minor Work

CLI Commands

• pkg/cli/logs_command.go (help text) — Rating: ⚠️ Needs Minor Work

Workflow Messages

• .github/workflows/archie.md — Rating: ✅ Professional (consistent persona voice)
• .github/workflows/brave.md — Rating: ✅ Professional (consistent persona voice)

Validation Code

• pkg/workflow/engine_validation.go — Rating: ✅ Professional

---

Metrics

• Files Analyzed: 6
• Quality Distribution:
  • ✅ Professional: 4
  • ⚠️ Needs Minor Work: 2
  • ❌ Needs Significant Work: 0

---

🎯 Actionable Tasks

Here are 2 targeted improvement tasks, each affecting a single file:

---

Task 1: Trim orchestrator guidance from logs help text

File to Modify: pkg/cli/logs_command.go

Current Experience

Lines 47–77 of the Long description contain an "Orchestrator Usage" section with GitHub Actions YAML step examples, markdown bold formatting, and the instruction "DO NOT call 'gh aw logs' or any GitHub CLI commands". This content is written for AI agents running inside orchestrator workflows, not for CLI users reading --help.

Quality Issue

Design Principle: Efficiency and Productivity — help text should answer "what does this command do and how do I use it?" not "how should an AI workflow agent avoid calling me?"

The orchestrator guidance adds ~30 lines of content that most users never need, buries the flags and examples further down the output, and contains a "DO NOT use this tool" note that undermines confidence in the command.

Proposed Improvement

Remove the "Orchestrator Usage" and "Live Tracking with Project Boards" subsections (lines 47–77) from the Long string. The remaining content — artifact list, WorkflowIDExplanation, and Examples — fully describes the command for its primary audience.

Before (lines 47–77 in the Long string):

Orchestrator Usage:
	In an orchestrator workflow, use this command in a pre-step to download logs,
  then access the data in subsequent steps without needing GitHub CLI access:
  ...
  **DO NOT call 'gh aw logs' or any GitHub CLI commands** - they will not work in your environment.
  ...
	Live Tracking with Project Boards:
		Use the summary.json data to update your project board...

After: Those lines are removed. The Long description ends after the artifact list and before WorkflowIDExplanation.

Why This Matters

• User Impact: gh aw logs --help becomes scannable in a single screen; users find flags and examples faster.
• Quality Factor: Clarity and Precision — removes audience mismatch from user-facing help text.
• Frequency: Every user who runs gh aw logs --help encounters this today.

Success Criteria

• Changes made to pkg/cli/logs_command.go only
• gh aw logs --help output is ≤60 lines (currently ~100+ lines)
• No orchestrator-internal guidance appears in CLI help text
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: pkg/cli/logs_command.go
• No changes to other files required
• No documentation site changes required for this task

---

Task 2: Add a complete minimal example to the comment-triggered overview page

File to Modify: docs/src/content/docs/examples/comment-triggered.md

Current Experience

The page (45 lines total) shows two isolated YAML snippets for the trigger configuration but never shows a complete workflow. The Quick Start section (lines 38–45) points users to gh aw add-wizard without explaining what it produces, then ends with "Then trigger it by commenting /review on a pull request!" — leaving users without a concrete mental model.

Quality Issue

Design Principle: Documentation Quality — practical examples that reflect real use cases.

Users land on this page expecting to understand how comment-triggered workflows work. They see YAML fragments without context and an install command without explanation. They must navigate to the ChatOps pattern page to see a complete example.

Proposed Improvement

Add a complete, self-contained minimal example after the "Example Command Triggers" section, showing frontmatter + markdown body together — similar to the pattern used in getting-started-mcp.md.

Before (lines 38–45):

## Quick Start

Add a ChatOps workflow to your repository:

```bash
gh aw add-wizard githubnext/agentics/repo-ask
```

Then trigger it by commenting `/review` on a pull request!

After:

## Quick Start

Create `.github/workflows/review.md`:

```aw wrap
---
on:
  slash_command:
    name: review
    events: [pull_request_comment]
permissions:
  contents: read
  pull-requests: read
safe-outputs:
  add-comment:
---

# PR Review Assistant

When someone types /review in a pull request comment, analyze the code changes
and add a comment with a summary of potential issues and suggestions.
```

Compile and push the workflow:

```bash
gh aw compile review
git add .github/workflows/review.md .github/workflows/review.lock.yml
git commit -m "Add /review slash command"
git push
```

The workflow is now active. Comment `/review` on any open pull request to trigger it.

Or explore pre-built ChatOps workflows:

```bash
gh aw add-wizard githubnext/agentics/repo-ask
```

Why This Matters

• User Impact: New users understand the complete structure in 30 seconds without leaving the page.
• Quality Factor: Documentation Quality — complete information with practical examples.
• Frequency: This is an entry point for all ChatOps workflows; high discovery traffic expected.

Success Criteria

• Changes made to docs/src/content/docs/examples/comment-triggered.md only
• A complete workflow example (frontmatter + body) is visible on the page
• Quick Start includes compile and push steps
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: docs/src/content/docs/examples/comment-triggered.md
• No changes to other files required
• Can be completed independently

---

References: §22222379006

AI generated by Delight

• expires on Feb 27, 2026, 11:36 AM UTC

[github-actions[bot]]

💥 WHOOSH! 🦸

BZZZT! The smoke test agent has arrived! KA-POW!

"By the power of the agentic workflow, I, Claude Smoke Test Agent, hereby declare this system OPERATIONAL!"

🌟 ZAP! All systems nominal! The MCP servers respond! The build succeeds! The playwright navigates! KAPOW!

[sfx: dramatic hero music intensifies]

✨ Smoke test run 22222473735 was here — and left no fires behind! 🔥❌

💥 [THE END] — Illustrated by Smoke Claude for issue #17120