📚 Documentation Noob Test Report - November 26, 2025

[github-actions[bot]]

Documentation Noob Test Report - November 26, 2025

Executive Summary

As a complete beginner with no prior knowledge of GitHub Agentic Workflows, I navigated through the documentation site to evaluate the getting started experience. This report documents critical issues, confusing areas, and recommendations for improving the new user experience.

Pages Visited:

• Home page: (redacted)
• Quick Start: (redacted)
• CLI Commands: (redacted)
• Creating Workflows: (redacted)
• Multiple example pages (Triage & Analysis, ChatOps, DailyOps)

Overall First Impression:
The documentation is well-structured with clear navigation, but several areas assume prior knowledge that beginners may not have. Key terms like "frontmatter," "compiled workflows," and "safe outputs" are used without immediate explanation.

---

🔴 Critical Issues Found

1. Missing Glossary/Terminology Page

Severity: High - Blocks understanding
Location: No concepts/glossary page exists
Issue: The Quick Start and other pages use technical terms without definitions:

• "frontmatter" - mentioned but not explained
• "safe outputs" - critical concept, minimal explanation
• ".lock.yml" - purpose explained but format not clear
• "Personal Access Token (PAT)" - brief note but no detailed guidance
• "coding agent" vs "AI engine" - terms used interchangeably

Impact: New users don't understand what these terms mean, leading to confusion when following steps.

Recommendation: Create a dedicated glossary page at /concepts/glossary/ with clear definitions and links from first mentions.

---

2. Unclear Token Creation Process (Step 3)

Severity: High - Blocks workflow execution
Location: Quick Start, Step 3
Issue: Creating a Personal Access Token with "Copilot Requests" permission is confusing:

• Step says "If you are not finding this option, review steps 2 and 3" - but reviewing those steps doesn't help
• No clear explanation of WHY this permission is needed
• No troubleshooting for users who can't find the permission
• "There is an ongoing work to support authentication" note creates confusion - should I wait or proceed?

Direct Quote from Page:

"Under 'Permissions,' click 'Add permissions' and select 'Copilot Requests'. If you are not finding this option, review steps 2 and 3."

What a Noob Thinks: "I can't find 'Copilot Requests' in my permissions list. I reviewed steps 2 and 3 but they don't mention this. Am I doing something wrong? Should I wait for the 'ongoing work' mentioned?"

Recommendation:

• Add screenshots showing exactly where to find "Copilot Requests"
• Explain eligibility requirements (Copilot subscription, account type)
• Add troubleshooting section: "Can't find Copilot Requests permission?"
• Clarify the "ongoing work" note with timeline or remove it

---

3. Compilation Concept Not Intuitive

Severity: Medium - Causes confusion but not blocking
Location: Quick Start, Step 2 & "Why Compile?" section
Issue: The concept of compiling markdown to YAML is mentioned but not clearly explained upfront:

• Users see both .md and .lock.yml files but don't understand the relationship immediately
• "Why Compile?" box appears AFTER the compile happens, should be BEFORE

What a Noob Thinks: "Why do I need two files? Can't GitHub Actions just read markdown? What if I edit the .lock.yml by mistake?"

Recommendation:

• Move "Why Compile?" explanation BEFORE Step 2
• Add a visual diagram showing: "You write → .md file → Compile → .lock.yml → GitHub Actions runs"
• Clearly state: "NEVER edit .lock.yml directly - always edit .md and recompile"

---

4. Insufficient Error Handling Guidance

Severity: Medium
Location: All setup pages
Issue: No clear troubleshooting for common first-time errors:

• What if gh extension install fails?
• What if gh aw add fails?
• What if compilation fails?
• What if the workflow runs but produces no output?

Recommendation: Add a "Common First-Time Issues" section at the bottom of Quick Start with actual error messages and solutions.

---

🟡 Confusing Areas (Slow Down Learning)

1. Home Page - Unclear Value Proposition

Location: (redacted)
Issue: The home page says "Write automation in markdown instead of complex YAML" but doesn't immediately show:

• What kind of automation?
• Who is this for?
• What can it actually do?

Recommendation: Add a prominent "What is this?" section with 2-3 concrete examples:

• "Automatically triage issues using AI"
• "Generate daily status reports from repository activity"
• "Respond to commands in pull request comments"

---

2. Workflow Structure Not Explained

Location: Quick Start, "Understanding Your First Workflow"
Issue: The workflow example is shown with three parts (frontmatter, heading, content) but the structure isn't explicitly called out:

---
config here
---
# Title
Instructions here

What a Noob Thinks: "What are the three dashes? Is the # Title part important? Can I change it?"

Recommendation:

• Add a labeled diagram showing: "YAML Config Section" | "Workflow Title" | "AI Instructions"
• Explicitly state which parts are required vs optional

---

3. "Safe Outputs" Term Unclear

Location: Quick Start workflow example
Issue: safe-outputs: create-discussion: appears without explanation

What a Noob Thinks: "What makes it 'safe'? What's unsafe about normal outputs?"

Recommendation: Add inline note: "safe-outputs: GitHub API operations that are reviewed before execution"

---

4. Prerequisites List Too Dense

Location: Quick Start, Prerequisites section
Issue: The prerequisites list is comprehensive but overwhelming for beginners. Mixes critical items with nice-to-haves.

Current: One big bulleted list
Recommended: Separate into:

• Must Have (GitHub account, gh CLI, repository)
• Must Configure (repository permissions, Actions enabled)
• Will Need Later (PAT for Copilot - you'll set this up in Step 3)

---

5. No Visual Confirmation of Success

Location: Quick Start, Step 4
Issue: After running gh aw status, users don't know what a successful output looks like.

Recommendation: Add example output:

✓ daily-team-status: completed successfully in 45s
  Discussion created: https://github.com/user/repo/discussions/123

---

🟢 What Worked Well

1. Step-by-Step Structure

The Quick Start guide follows a clear 1-2-3-4 structure that's easy to follow.

2. Clear CLI Command Examples

Commands are shown with exact syntax: gh aw add githubnext/agentics/daily-team-status --pr

3. "Why Compile?" Callout Box

The concept explanation is good (just needs to be moved earlier).

4. Caution Box at Top

The research demonstrator warning is appropriately prominent and sets expectations.

5. Concrete Example Workflow

Starting with daily-team-status is a good choice - it's simple and produces visible output (a discussion).

---

Priority Recommendations

Immediate Quick Wins (1-2 hours)

1. Add glossary page with key terms: frontmatter, safe outputs, .lock.yml, coding agent, PAT
2. Move "Why Compile?" section to appear BEFORE Step 2
3. Add "If you can't find Copilot Requests permission" troubleshooting to Step 3
4. Add example successful output to Step 4

Short-term Improvements (1 day)

5. Create visual diagram showing markdown → compile → YAML flow
6. Add screenshots for PAT creation process
7. Reorganize prerequisites into Must Have / Must Configure / Will Need Later
8. Add "Common First-Time Errors" section at bottom of Quick Start

Longer-term Enhancements (1 week)

9. Improve home page with concrete use case examples
10. Add workflow structure explanation with labeled diagram
11. Create interactive troubleshooting guide (decision tree style)
12. Add video walkthrough of Quick Start (5-7 minutes)

---

Terminology Gaps Identified

Terms used without definition on first mention:

• frontmatter
• safe outputs
• .lock.yml / compiled workflow
• coding agent vs AI engine
• Personal Access Token (PAT)
• fine-grained permissions
• cron schedule
• workflow_dispatch
• GitHub Actions secret

Action: All these should be defined in glossary and linked on first use.

---

Navigation Structure Assessment

Clear:

• Setup section is well-organized
• Examples are categorized logically
• Reference section is comprehensive

Confusing:

• No clear "Concepts" or "Learn" section for understanding core ideas
• Troubleshooting is under CLI Commands (should be top-level or in Setup)
• "Labs" section has no content/explanation

---

Conclusion

The documentation is well-written and comprehensive, but assumes more knowledge than a true beginner has. The main pain points are:

1. Missing foundational concepts - glossary is critical
2. Token creation confusion - Step 3 needs more help
3. Compilation model - needs earlier, clearer explanation

With the recommended quick wins implemented, the getting started experience would improve significantly. The biggest impact would come from adding the glossary and improving Step 3's troubleshooting.

---

Test Methodology

• Tested on: November 26, 2025
• Approach: Read documentation with "beginner mindset," noting every moment of confusion
• Tools: curl + bash for page analysis
• Coverage: Home, Quick Start, CLI, Authoring, 3 example pages
• Time to complete: ~30 minutes of beginner-paced reading

---

Labels: documentation, user-experience, automated-testing

AI generated by Documentation Noob Tester

[pelikhan]

/plan

[github-actions[bot]]

✅ Agentic Plan Command completed successfully.