📚 Documentation Noob Test Report - 2026-02-20

[github-actions[bot]]

Summary

• Date of test: 2026-02-20
• Pages visited: Home page, Quick Start, CLI Commands, Creating Workflows, Overview/About Workflows, Architecture, Glossary, Examples (Scheduled, Manual, Issue/PR Events), FAQ, Troubleshooting
• Overall impression as a new user: The documentation has a great structure and the Quick Start wizard is genuinely easy to follow. However, there are a significant number of broken internal links and one dead navigation item (Custom Engines → 404) that would frustrate beginners trying to learn more. The terminology is dense — terms like "frontmatter", "lock file", "safe outputs", and "MCP" appear early in the getting-started flow without enough context for a complete beginner.

---

🔴 Critical Issues Found

1. "Custom Engines" sidebar navigation leads to 404

The Reference sidebar has a "Custom Engines" link that navigates to /reference/custom-engines/ — but this page does not exist and returns a 404 error. A beginner exploring the sidebar will hit a dead end when looking for information about AI engine options.

Location: Reference section sidebar → "Custom Engines" link
Evidence: curl -s -o /dev/null -w "%{http_code}" (localhost/redacted) → 404 **Fix:** Either create the page or remove the sidebar link. Theengines.md` page (AI Engines) seems to cover this topic already.

---

2. 202 invalid internal links reported during build

The starlight-links-validator plugin reports 202 invalid links across 69 files during every build:

[ERROR] ✗ Found 202 invalid links in 69 files.
```

Most affected links follow this pattern — pointing to key reference pages like:
- `/gh-aw/reference/safe-outputs/`
- `/gh-aw/reference/tools/`
- `/gh-aw/reference/triggers/`
- `/gh-aw/reference/schedule-syntax/`
- `/gh-aw/reference/workflow-structure/`
- `/gh-aw/reference/templating/`
- `/gh-aw/reference/memory/`
- `/gh-aw/troubleshooting/common-issues/`

The pages **do exist** and are accessible after build (returning HTTP 200), suggesting this is a link validator configuration issue — possibly the validator doesn't recognize pages manually listed in the sidebar config vs. those using `autogenerate`. However, from a CI perspective, this is a persistent build error that will prevent detection of real broken links in the future.

**Impacted pages:** 69 files across blog posts, guides, patterns, examples, and setup pages  
**Fix needed:** Investigate and fix the `starlight-links-validator` configuration or switch to relative link formats in the source files.

---

### 3. Broken external link in Quick Start "What's next?" section

The Quick Start page (`/setup/quick-start/`) links to Peli's Agent Factory with this URL:
```
https://github.github.com/gh-aw/blog/2026-01-12-welcome-to-pelis-agent-factory/

The domain github.github.com is the site's configured base URL — this is the production URL format which may not be accessible in all environments. A beginner following the Quick Start would click this link and potentially reach a broken page.

File: docs/src/content/docs/setup/quick-start.mdx line 121
Fix: Convert to a relative internal link: /gh-aw/blog/2026-01-12-welcome-to-pelis-agent-factory/

---

4. Missing example sub-pages linked from category pages

The Examples section has category pages that link to sub-pages not accessible from the sidebar navigation:

• examples/issue-pr-events/triage-analysis/
• examples/issue-pr-events/coding-development/
• examples/issue-pr-events/quality-testing/
• examples/multi-repo/feature-sync/
• examples/multi-repo/issue-tracking/

These pages ARE built and accessible, but a beginner clicking through the sidebar won't find them — they're only reachable via the parent category page links.

---

🟡 Confusing Areas

5. Dense jargon in the getting-started flow

As a complete beginner, I encountered many terms without immediate definitions:

Term	Where first encountered	Issue
frontmatter	Quick Start Step 4, Creating Workflows	Used without explanation; links to reference but not enough inline context
lock file	Homepage example, Quick Start Step 4	The concept of a compiled .lock.yml is not explained until you dig into references
safe outputs	Homepage, Quick Start	Critical security concept but described as jargon without a beginner-friendly one-liner
MCP (Model Context Protocol)	Creating Workflows, CLI docs	Mentioned as an option without explaining what it IS or why you'd want it
engine	Prerequisites, Quick Start	"AI engine" used interchangeably with "coding agent" — inconsistent terminology

Recommendation: Add a "Key concepts" callout box at the top of the Quick Start page with 3-4 sentence definitions of the 5 most important terms.

---

6. Quick Start step numbering is inconsistent

The Quick Start page mixes unnumbered intro steps with numbered steps:

• "Step 1 - Install the extension" → Under Prerequisites heading (confusing: is it part of prerequisites?)
• "Step 2 - Add the sample workflow" → Still under Prerequisites
• "## Step 3 - Wait for the workflow to complete" → Separate heading
• "### Step 4 - Customize" → Child of Step 3

A new user will be confused about whether step 1 is "check prerequisites" or "install the extension." Steps 1-2 feel like they belong AFTER prerequisites, not embedded within them.

Fix: Separate prerequisites into a proper prerequisites block, then number steps cleanly from 1-4.

---

7. Prerequisites don't explain AI account costs/accessibility

The Prerequisites section says:

AI Account - GitHub Copilot, Anthropic Claude or OpenAI Codex

A complete beginner needs to know:

• Which of these is free / requires a paid subscription?
• Do I need a GitHub Copilot subscription, or is there a free tier?
• What's the cheapest option to get started?

Without this, beginners may be stuck at the very first prerequisite.

Fix: Add brief notes: e.g., "GitHub Copilot requires a subscription (free for verified students/OSS maintainers). Claude and Codex require API accounts with pay-as-you-go billing."

---

8. gh aw add-wizard vs gh aw add confusion

The Quick Start guide uses:

gh aw add-wizard githubnext/agentics/daily-repo-status
```

But the CLI Commands reference documents this as the `add` command, with `add-wizard` being an undocumented alias. A beginner checking the CLI reference after Quick Start will search for "add-wizard" and not find it, leading to confusion.

**Fix:** Explicitly mention in CLI docs that `add-wizard` is an alias for `add`, or consistently use `add` in the Quick Start.

---

### 9. Architecture page is overwhelming for beginners

The [Security Architecture](/introduction/architecture/) page opens with:

> "GitHub Agentic Workflows implements a defense-in-depth security architecture... **Layer 1: Substrate-Level Trust** — AWs run on a GitHub Actions runner virtual machine (VM) and trust Actions' hardware and kernel-level enforcement mechanisms, including the CPU, MMU, kernel, and container runtime..."

This is the **third page** linked from the Introduction section. A beginner exploring "About Workflows" → "How They Work" → "Architecture" will hit extremely dense technical content about kernel isolation and substrate-level trust.

**Fix:** Add a "TL;DR" or "For beginners" callout at the top of the architecture page summarizing the security model in plain language, before the detailed technical content.

---

### 10. `abnf` language warning in Fuzzy Schedule Specification

```
[WARN] Error while highlighting code block using language "abnf" in fuzzy-schedule-specification.md.
       The language could not be found. Using "txt" instead.

The schedule specification page uses abnf syntax highlighting which isn't supported by the bundled Shiki highlighter. Blocks display as plain text instead of properly highlighted grammar.

Fix: Either add abnf to the Shiki langs configuration or use a supported language alias.

---

🟢 What Works Well

✅ Quick Start wizard is genuinely easy

The gh aw add-wizard command handles the entire setup interactively (checking prerequisites, selecting AI engine, setting up secrets, adding workflow, triggering a run). This is an excellent new-user experience once you get past the prerequisite setup.

✅ "Most Common Commands" section in CLI docs

The CLI Commands page opens with 6 commands that most users need, with links to full documentation. This is exactly what beginners need.

✅ Estimated time on tutorials

Both Quick Start ("⏱️ 10 minutes") and Creating Workflows ("⏱️ 5-15 minutes") include estimated times. This sets expectations and is a great UX practice.

✅ Video demos

The embedded video demos on the Quick Start and Creating Workflows pages provide visual guidance that complements the text.

✅ Glossary page

The Glossary is comprehensive with definitions for all technical terms. The "[!TIP] New to GitHub Agentic Workflows?" callout at the top is helpful. However, more inline linking to the glossary from getting-started pages would help.

✅ FAQ is well-structured

The FAQ addresses common concerns like "Isn't this non-deterministic?" (addressing CI/CD concerns) and "Can workflows write code?" (answering practical use case questions). Great content for skeptical beginners.

✅ Practical homepage example

The homepage shows a complete, minimal workflow example (Daily Issues Report) immediately. This gives beginners an immediate sense of what workflows look like.

---

Recommendations

Quick Wins (high impact, low effort)

1. Fix the "Custom Engines" 404 — either create the page or remove the sidebar link
2. Fix the broken external link in Quick Start "What's next?" section
3. Add inline glossary callout to Quick Start page for key terms (frontmatter, lock file, safe outputs)
4. Clarify AI account costs in Quick Start prerequisites
5. Document add-wizard as alias in CLI commands page

Medium-term Improvements

6. Investigate and fix the 202 invalid link validator errors — these are CI noise that masks real issues
7. Restructure Quick Start step numbering — move steps 1-2 out of the prerequisites block
8. Add example sub-pages to sidebar navigation — so users can find them without going through category pages
9. Add "beginners" summary box to Architecture page
10. Fix abnf language highlighting in fuzzy schedule specification

Longer-term Documentation Improvements

11. Add a "Learning Path" section to the documentation homepage (e.g., "I'm new → Start here", "I want examples → Go here", "I want reference → Go there")
12. Create a "Your first custom workflow" tutorial that goes from zero to a personalized workflow (beyond the canned daily-repo-status demo)
13. Add more screenshots/output examples to the Examples section pages
14. Ensure terminology is consistent: "AI engine" vs "coding agent" — pick one and use it throughout

---

Automated documentation review conducted by GitHub Copilot CLI Agent.

AI generated by Documentation Noob Tester

• expires on Feb 27, 2026, 8:11 AM UTC