Project: Creative Apps - GitHub Agentic Workflow Inspector #57
Description
Track
Creative Apps (GitHub Copilot)
Project Name
GH AW Inspector
GitHub Username
Repository URL
https://github.com/petermefrandsen/gh-aw-inspector
Project Description
GH AW Inspector is a Visual Studio Code extension that brings the GitHub Agentic Workflows CLI (gh aw) directly into the editor.
GitHub Agentic Workflows let developers define AI-driven tasks in plain Markdown files.
The problem: once written, those workflows are opaque blobs — there's no way to understand, inspect, or evaluate them without running the CLI manually.
GH AW Inspector closes that gap.
Key features:
- 🗂️ Workflow Discovery Sidebar - automatically discovers all .md agentic workflow files in .github/workflows/ and lists them in a dedicated Activity Bar panel
- 🔍 Frontmatter Inspector - parses and renders the full gh-aw schema (engine, triggers, tools, permissions, safe outputs) so you can understand a workflow at a glance
▶️ One-Click Evaluation - select any workflow and launch an evaluation powered by the VS Code Copilot Language Model API (vscode.lm), streaming results in real time- 📝 Auto-Generated Reports - each evaluation produces a structured Markdown report covering description, metadata, intent, happy/unhappy path outputs, and prerequisites
The extension is published on both the VS Code Marketplace and Open VSX, includes a full CI pipeline, and has >85% unit test coverage.
Demo Video or Screenshots
Screenshots: https://github.com/petermefrandsen/gh-aw-inspector/tree/main/docs/screenshots
Live demo: https://marketplace.visualstudio.com/items?itemName=PeterMEFrandsen.gh-aw-inspector
Primary Programming Language
TypeScript/JavaScript
Key Technologies Used
- VS Code Extension API (webview panels, sidebar providers, commands)
- GitHub Copilot Language Model API (vscode.lm) — streams evaluation results
- gray-matter — frontmatter parsing for gh-aw workflow Markdown files
- TypeScript (strict mode)
- esbuild — extension bundler
- Mocha + @vscode/test-cli — unit & integration testing (>85% coverage)
- GitHub Actions — CI/CD with auto-release to VS Marketplace & Open VSX
Submission Type
Individual
Team Members
No response
Submission Requirements
- My project meets the track-specific challenge requirements
- My repository includes a comprehensive README.md with setup instructions
- My code does not contain hardcoded API keys or secrets
- I have included demo materials (video or screenshots)
- My project is my own work with proper attribution for any third-party code
- I agree to the Code of Conduct
- I have read and agree to the Disclaimer
- My submission does NOT contain any confidential, proprietary, or sensitive information
- I confirm I have the rights to submit this content and grant the necessary licenses
Quick Setup Summary
- Install the extension from the VS Code Marketplace:
ext install petermefrandsen.gh-aw-inspector - or search "GH Agentic Workflows Inspector" in the Extensions panel - Install prerequisites (if using locally from source):
- git clone https://github.com/petermefrandsen/gh-aw-inspector.git
- cd gh-aw-inspector
- pnpm install
- pnpm run package
- Install the gh-aw CLI extension:
- gh extension install github/gh-aw
- Open a repo that has .github/workflows/*.md agentic workflow files
- Press F5 (dev mode) or use the installed extension — click the GH Agentic Workflows icon in the Activity Bar, select a workflow, and press ▶ Start Evaluation
Technical Highlights
- Copilot Language Model API (vscode.lm): Evaluation results are streamed token-by-token using VS Code's built-in vscode.lm.selectChatModels API, requiring zero API keys or manual token management - it uses the user's existing Copilot subscription seamlessly
- Recursive import resolution: The importResolver utility walks the
imports:frontmatter field recursively, assembles a dependency-ordered context block, and feeds it into the evaluation prompt - so evaluations are aware of shared prompt libraries and reusable workflow fragments - Dual-registry CI/CD: A single GitHub Actions workflow auto-bumps the semver (driven by PR labels), creates a GitHub Release, and publishes simultaneously to both the VS Code Marketplace and Open VSX Registry
- Zero-config workflow discovery: Uses VS Code's RelativePattern + workspace.findFiles to discover *.md workflows without any user configuration, filtering out standard .yml/.yaml GitHub Actions files
- Unit test coverage > 85%: Test suite runs inside the VS Code extension host (real VS Code APIs, no mocks for the host itself), with coverage reported to a live badge via GitHub Gist
Challenges & Learnings
The biggest challenge wasn't the extension logic - it was the CI/CD pipeline, which took more iterations to get right than anything else.
Publishing pipeline rabbit hole (4 PRs to get it working)
The initial plan was simple: a separate publish.yml that triggers when a release is created. In practice, GITHUB_TOKEN-pushed tags do NOT trigger other workflow runs - a GitHub security boundary that isn't prominently documented. This meant the publish job silently never ran.
The fix was merging the publish job into auto-release.yml as a dependent job. Then the next wall: pnpm's hoisted node_modules layout causes npm ls (which vsce runs internally) to report missing devDependencies and abort. Fix: --no-dependencies flag. Two separate issues, four PRs.
Test coverage infrastructure was harder than the tests themselves
Getting @vscode/test-cli, Istanbul coverage output, and the CI badge pipeline to all agree on the same format required fixing: the coverage reporter CLI flags, a corrupted coverage-output.txt caused by a duplicate tee in the Makefile, an incompatible JUnit reporter, and the Gist ID being passed as a full URL instead of just the ID. The actual test logic was straightforward - the plumbing around it was not.
A subtle sidebar bug
The sidebar would get stuck showing a loading state after switching away and back to the view. The fix revealed that the onDidChangeVisibility callback wasn't calling _updateWorkflows - it was simply missing.
Key learning:
In VS Code extension development, the runtime constraints (GITHUB_TOKEN scoping, pnpm/vsce incompatibilities, extension host test lifecycle) are ehere you lose time - not the product features. Plan for that friction upfront.
Contact Information
Country/Region
Denmark