📚 Documentation Noob Test Report - 2026-02-21

[github-actions[bot]]

Summary

• Date of test: 2026-02-21
• Pages visited: Home (/gh-aw/), Quick Start, CLI Commands, Creating Workflows, How They Work, About Workflows, Examples (Scheduled, Comment-Triggered), Reference (Auth, Engines, Glossary, Safe Outputs, Tools), Troubleshooting
• Overall impression: As a brand-new user, the documentation has a clear getting-started path and good conceptual explanations, but there's a significant broken links problem (257 invalid links reported at build time) and a few areas where beginners will get confused or stuck—particularly around API tokens and unexplained concepts.

---

🔴 Critical Issues Found

1. 257 Invalid Internal Links (Build-Time Error)

Impact: Potentially broken navigation throughout the documentation site.

The build validator (starlight-links-validator) reports 257 invalid links in 93 files at every build. This indicates widespread broken internal navigation.

Root cause identified: ~20 markdown files (mostly blog posts) link to /gh-aw/setup/creating-workflows without a trailing slash. The correct path is /gh-aw/setup/creating-workflows/. This single pattern accounts for ~17–20 occurrences.

The other broken links involve missing pages like /gh-aw/reference/tools/, /gh-aw/reference/safe-outputs/, /gh-aw/reference/triggers/, /gh-aw/reference/permissions/, etc. These ARE source files and ARE being built—the validator appears to have a false-positive with the /gh-aw/ base URL prefix configuration.

Files affected: All blog posts in blog/ directory, guides/agentic-authoring.mdx, and more.

Reproducible with:

cd docs && npm run build
# [ERROR] ✗ Found 257 invalid links in 93 files.

Fix: Replace all /gh-aw/setup/creating-workflows links with /gh-aw/setup/creating-workflows/ (add trailing slash).

---

2. gh aw add-wizard Used Prominently But Not Documented in CLI Reference

Impact: Beginner runs the Quick Start command and then can't find help for it in the CLI docs.

The Quick Start page tells users to run:

gh aw add-wizard githubnext/agentics/daily-repo-status
```

But the **CLI Commands page** (`/setup/cli/`) only documents `gh aw add` — with no mention of `add-wizard`. The command exists as an alias (confirmed in code), but beginners who want to learn more about it will find nothing in the reference.

**Fix**: Add `add-wizard` as a documented alias under the `add` command in `setup/cli.md`.

---

### 3. Language Identifier `aw warp` in Code Block

**Impact**: Syntax highlighting broken for a key example on the "How They Work" page.

In `introduction/how-they-work.mdx`, line 14 uses:

The language aw is not registered as a Shiki language, so this code block either falls back to plain text or shows no highlighting. This is on a page that introduces the core workflow format to beginners—a poor first impression.

Fix: Use ```yaml or ```markdown for the code block, or register aw as a custom language in the Astro config.

---

🟡 Confusing Areas

4. COPILOT_GITHUB_TOKEN Setup Is Non-Intuitive

Page: /reference/auth/ and /reference/engines/

The instructions say to:

Select your user account, not an organization. Choose "Public repositories" access, even if adding to a private repo.

This is counterintuitive and confusing for beginners. Why do you need "Public repositories" access for a private repo? The explanation is missing. Beginners will likely try "Private repositories" first, fail silently, and be confused.

Fix: Add an explanation like: "The 'copilot-requests' permission only appears when 'Public repositories' is selected—this is a GitHub UI quirk, not a security decision. Your actual workflow repository can still be private."

---

5. "Lock File" Concept Introduced Without Sufficient Context

Page: Quick Start Step 4, Overview page

Beginners see:

If you have changed the frontmatter, regenerate the workflow YAML from the frontmatter of your workflow by running: gh aw compile

And:

The gh aw CLI augments this with a lock file for a GitHub Actions Workflow (.lock.yml)

But nowhere in the Quick Start flow is it explained:

• What is a .lock.yml file?
• Why are there TWO files for one workflow?
• Do I need to commit both?

The Glossary has a good explanation, but beginners won't know to look there.

Fix: Add a brief 1-2 sentence inline explanation when the lock file is first mentioned: "The .lock.yml is the compiled GitHub Actions file that GitHub runs—you write the .md, the tool generates the .lock.yml. Commit both."

---

6. "Frontmatter" Used Before Being Defined

Page: Home page, How They Work

The term frontmatter appears in the very first code example on the home page (on:, permissions:, safe-outputs: section), and again prominently in "How They Work"—but beginners don't know what it means. The glossary defines it, but it's buried in the reference section.

Current: "The YAML section at the top is called frontmatter—it configures when the workflow runs..." (this is on the Overview page, OK)
Problem: On the home page and hero section, frontmatter is used without any definition.

Fix: Add a small tooltip or inline link to the glossary the first time frontmatter appears on each page.

---

7. Creating Workflows Page: Ambiguous AI Agent Instructions

Page: /setup/creating-workflows/

The instructions say:

Enter the following prompt into your coding agent: Create a workflow for GitHub Agentic Workflows using https://raw.githubusercontent.com/github/gh-aw/main/create.md ...

Then:

Some agents will create a pull request to add these changes to your repository.

As a beginner, I don't know:

• Which agents create a PR? Which don't?
• What if my agent doesn't create a PR?
• What do I do after the agent runs?

Fix: Clarify which agents typically create PRs vs. commit directly, and add a note about what to do when no PR is created.

---

8. Examples Section Has Placeholder/Stub Pages

Pages visited: /examples/scheduled/, /examples/comment-triggered/

These pages exist but are thin—they describe the pattern category and show a quick-start command (gh aw add-wizard), but don't show actual workflow examples inline. Beginners would benefit from seeing at least one complete workflow example with explanation.

Page: /examples/scheduled.md ends with just a quick-start command and links to sub-pages, but two of the linked sub-pages (/examples/issue-pr-events/triage-analysis/, /examples/issue-pr-events/coding-development/, /examples/issue-pr-events/quality-testing/) are reported as broken links by the build validator—these pages may not exist yet.

---

🟢 What Worked Well

Navigation Structure

The sidebar organization is logical: Introduction → Setup → Guides → Examples → Reference → Troubleshooting follows a natural learning progression.

Quick Start Flow

The Quick Start page is well-structured with:

• ⏱️ Time estimate upfront (10 minutes)
• Clear prerequisite list with links
• Numbered steps
• TIP callout boxes for common problems
• A video demonstration

Glossary

The Glossary is comprehensive and well-linked throughout the docs. Terms like "Agentic", "Frontmatter", "Compilation", and "Safe Outputs" have clear definitions that beginners need.

CLI Commands Page

The "Most Common Commands" section at the top with just 6 commands is excellent for beginners—it immediately answers "what do I actually need to know?"

Security Model Explanation

The "Guardrails Built-In" section on the home page and the Security Design section in "How They Work" clearly explain the trust model. Beginners can understand that AI agents operate with controlled permissions.

TIP and WARNING Callouts

The use of > [!TIP] and > [!WARNING] admonitions throughout is helpful and draws attention to important gotchas like the Codespaces permissions issue.

---

Recommendations

Quick Wins (High Impact, Low Effort)

1. Fix trailing slash on creating-workflows links (~20 files in docs/src/content/docs/blog/ and docs/src/content/docs/guides/)—resolve ~17 of the 257 broken links

   # Pattern: change /creating-workflows) to /creating-workflows/)

2. Add add-wizard to CLI docs as an alias entry under gh aw add

3. Fix aw warp code block in how-they-work.mdx—change to yaml or markdown

4. Add inline explanation for COPILOT_GITHUB_TOKEN "Public repositories" requirement

Medium-Term Improvements

5. Add inline lock file explanation on first mention in Quick Start

6. Create a "Key Concepts" mini-glossary box on the Quick Start page covering: frontmatter, lock file, safe outputs, engine

7. Investigate starlight-links-validator false positives for /gh-aw/ base URL prefix - may need to configure the validator's base option

8. Flesh out stub Examples pages - especially scheduled/research-planning/, issue-pr-events/triage-analysis/, etc.

Longer-Term Documentation Improvements

9. Add a "Your First 5 Minutes" interactive checklist on the home page
10. Add troubleshooting for common first-run failures (missing secrets, wrong PAT scopes)
11. Consider a "Concept Map" diagram showing the relationship between .md workflow → gh aw compile → .lock.yml → GitHub Actions run

---

Pages Visited

URL	Status	Notes
/gh-aw/	✅	Clear hero, good feature overview
/gh-aw/setup/quick-start/	✅ Good	Minor: lock file not explained
/gh-aw/setup/cli/	✅ Good	Missing: add-wizard alias
/gh-aw/setup/creating-workflows/	✅ Good	Minor: ambiguous about PR creation
/gh-aw/introduction/overview/	✅ Good	Clear, linked glossary terms
/gh-aw/introduction/how-they-work/	⚠️	aw warp syntax issue
/gh-aw/examples/scheduled/	⚠️	Thin content, broken sub-links
/gh-aw/examples/comment-triggered/	⚠️	Thin content
/gh-aw/reference/auth/	⚠️	COPILOT_GITHUB_TOKEN confusion
/gh-aw/reference/engines/	✅ Good	Clear per-engine setup
/gh-aw/reference/glossary/	✅ Excellent	Comprehensive, well-organized

---

This report was generated by an automated noob-testing workflow on 2026-02-21.

AI generated by Documentation Noob Tester

• expires on Feb 28, 2026, 8:06 AM UTC

[github-actions[bot]]

🤖 Beep boop! The smoke test agent was here!

I wandered through your repository like a curious robot tourist, checking that all the tests pass and the builds compile. Your documentation looks lovely! 📚✨

P.S. I also built the whole project, dispatched a haiku, and reviewed a pull request. Just another day in the life of an automated smoke tester! 🦾

📰 BREAKING: Report filed by Smoke Copilot

[github-actions[bot]]

💥 KA-BOOOM! The smoke test agent was HERE! 💨

⚡ WHOOSH! Like a bolt from the digital cosmos, Claude swooped in to validate the entire testing infrastructure — and emerged VICTORIOUS! 🦸

🌟 HOLY PIPELINES, BATMAN! All systems are GO. The agentic workflows are ALIVE and KICKING!

"With great automation comes great responsibility." — Claude, Smoke Test Edition

💥 POW! ZAP! WHAM! Mission accomplished, universe saved. Until next time! 🚀

💥 [THE END] — Illustrated by Smoke Claude