[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-04-15

[github-actions[bot]]

Summary

• Date: 2026-04-15
• Pages visited: Home, [/gh-aw/]((localhost/redacted) [/gh-aw/setup/quick-start/]((localhost/redacted) [/gh-aw/setup/cli/]((localhost/redacted)
• Overall impression: The site looks polished and the security story is clearly articulated, but the Quick Start has a big hidden dependency (setting up a PAT token) that could easily block a new user before they write a single workflow. Navigation is clean; "Quick Start" is easy to find.

⚠️ Note: Visual screenshots were unavailable — Playwright could not connect to the dev server from its container (net::ERR_CONNECTION_TIMED_OUT). Page content was captured via curl and is attached as text artifacts.

---

🔴 Critical Issues (Block Getting Started)

1. COPILOT_GITHUB_TOKEN — Hidden Complexity in Step 2

On the Quick Start page, Step 2 instructs users to run gh aw add-wizard which internally requires setting up a fine-grained PAT with the "Copilot Requests" permission — a separate token distinct from the standard GITHUB_TOKEN. This is mentioned briefly in a parenthetical:

"Set up the required secret — COPILOT_GITHUB_TOKEN (a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN)"

For a true beginner, "fine-grained PAT with Copilot Requests permission" is a significant setup step with many sub-steps (create PAT, configure permissions, add as GitHub secret). The Quick Start page links to a separate Authentication page for details but doesn't inline the critical steps.

Recommendation: Either inline a 3-step "create your token" mini-guide in Quick Start, or add a prominent callout box saying "⚠️ This step requires a PAT — takes ~5 min to set up first" before Step 2.

2. add-wizard is a Black Box

Step 2 relies entirely on gh aw add-wizard githubnext/agentics/daily-repo-status — an interactive wizard. The docs say it "takes you through an interactive process" but don't show what the wizard prompts look like or what answers to give. A beginner running this command for the first time has no idea what to expect.

Recommendation: Add a screenshot or text walkthrough of the wizard interaction (e.g., "The wizard will ask: Which engine? [Copilot / Claude / Codex] — select Copilot for this guide").

3. "Frontmatter" Term Used Without Definition at Point of First Use

On the Overview/About Workflows page, the term frontmatter is used prominently:

"The YAML section at the top is called frontmatter — it configures when the workflow runs and what it can do."

This is fine on the Overview page. But the Quick Start page mentions "If you have changed the frontmatter (the YAML configuration block between --- markers)" in Step 4 without linking to a definition. A beginner may not know what --- markers are or how to identify frontmatter in a file.

Recommendation: Link "frontmatter" to the Frontmatter reference or Glossary on its first mention in Quick Start.

---

🟡 Confusing Areas

4. Video with Vague Link Text ("Download the video here")

The Quick Start page embeds a video (<video> tag) and has a fallback link:

"Your browser doesn't support HTML5 video. Download the video here."

The link text "here" is a known accessibility anti-pattern, but more importantly for beginners: if the video fails to load (e.g., slow CI environment), the user gets a broken experience with no description of what the video shows.

Recommendation: Add a text caption like "Video: Installing the extension and adding your first workflow (2 min)" and change the link to "Download video".

5. Navigation Bar Item "Create" is Ambiguous

The top navigation has: Quick Start | Create | Examples | Docs | FAQ | Blog

"Create" is unclear — create what? A workflow? An issue? For a newcomer, "Getting Started" or "Create a Workflow" would be more descriptive.

Recommendation: Rename "Create" to "Create a Workflow" or "Workflow Guide".

6. "Peli's Agent Factory" in Top Nav is Confusing for Newcomers

The persistent top navigation includes "Peli's Agent Factory" — a blog post link named after a person. This is confusing for a first-time visitor who doesn't know who Peli is or why a named "factory" is in the main nav.

Recommendation: Rename to "Workflow Gallery" or "Example Workflows", or move to the Blog section. Consider adding a short tooltip or subtitle on hover.

7. CLI Commands Page: No "Start Here" Guidance

The CLI Commands page lists 30+ commands in a large table. For a beginner, there's no visual hierarchy to distinguish "you need these 3 commands to start" from "these are advanced/rarely-used commands". The "Most Common Commands" table at the top helps, but all 10 commands in it carry equal weight.

Recommendation: Add a "🚀 Just getting started? Run these in order:" callout with gh aw init → gh aw add-wizard → gh aw run as a 3-command getting-started sequence.

8. "Check in Settings → Actions" Without a Direct Link

Prerequisites list: "GitHub Actions enabled — Check in Settings → Actions"

This is vague — "Settings → Actions" could mean various sub-pages. No direct link is provided.

Recommendation: Link directly to the GitHub Actions settings path: https://github.com/{owner}/{repo}/settings/actions.

9. Step 4 (Customize) Assumes Familiarity With Git

Step 4 says:

"Commit and push to your repository."

No commands are provided. A newcomer unfamiliar with git won't know what to run.

Recommendation: Add: git add .github/workflows/ && git commit -m "Add daily-repo-status workflow" && git push

---

🟢 What Worked Well

1. Hero section is clear and compelling — "Repository automation, running the coding agents you know and love, with strong guardrails in GitHub Actions" is immediately descriptive. The "Quick Start with CLI" CTA button is prominent and leads directly to the right page.

2. Security architecture explained up front — The 5 security layers on the home page build trust immediately. For technically-minded users, this answers "is this safe?" before they even try it.

3. Estimated time on Quick Start — "Estimated time: 10 minutes" sets expectations well. The 4-step structure is logical and scannable.

4. Authentication page is comprehensive — Once you find it (/gh-aw/reference/auth/), it explains every engine's secret very clearly with direct PAT creation links. The gh aw secrets set CLI shortcut is excellent.

5. CLI Commands "Most Common Commands" table — The table at the top of the CLI page with 10 key commands is very helpful. The descriptions are concise.

6. Sidebar navigation is well-organized — The left sidebar groups content logically: Setup → Guides → Design Patterns → Reference → Troubleshooting. A beginner can orient themselves quickly.

7. Tip callouts in Quick Start — The "Tip" boxes (e.g., about auth issues, about checking secrets) are well-placed and use good formatting.

---

Recommendations

Quick wins (high impact, low effort):

1. 🔴 Add a "⚠️ Before Step 2: Set up your token" callout box in Quick Start with inline 3-step PAT setup
2. 🟡 Add a wizard walkthrough (text or screenshot) showing the add-wizard prompts
3. 🟡 Rename nav item "Create" → "Create a Workflow"
4. 🟡 Add git add . && git commit -m "..." && git push commands to Step 4
5. 🟡 Change "here" link text on video to "Download video"

Longer-term improvements:
6. Add a "3-command quick start" callout box at the top of the CLI page
7. Consider renaming "Peli's Agent Factory" to "Workflow Gallery" in the nav
8. Link "GitHub Actions enabled" prerequisite to the actual settings URL
9. Cross-link "frontmatter" on first use in Quick Start to the Glossary

---

Page Content Artifacts

Text-based page captures (visual screenshots were unavailable due to network isolation):

📎 homepage-content.txt — artifact aw_home (download from workflow run artifacts)

📎 quickstart-content.txt — artifact aw_qs (download from workflow run artifacts)

📎 cli-content.txt — artifact aw_cli (download from workflow run artifacts)

References: §24456590748

Generated by Documentation Noob Tester · ● 4M · ◷

• expires on Apr 16, 2026, 1:33 PM UTC