Smoke Test: Claude - 22487258068 #18683
Description
Test Results Summary
Overall Status: PARTIAL — Most tests passed, with 1 failure, 1 partial failure, and 2 skips.
PR Used for Tests: #18669 (chore: bump CLI tool versions)
Run: §22487258068
Timestamp: 2026-02-27T13:05:00Z
Core Tests (#1–10)
| # | Test | Result | Notes |
|---|---|---|---|
| 1 | GitHub MCP Testing | ❌ | Response exceeded 25k token limit (73k chars) |
| 2 | Safe Inputs GH CLI Testing | ✅ | Got 2 PRs: #18671, #18669 |
| 3 | Serena MCP Testing | activate_project succeeded; find_symbol failed with EOF |
|
| 4 | Make Build Testing | ✅ | make build succeeded |
| 5 | Playwright Testing | ✅ | Page title contains "GitHub" |
| 6 | Tavily Web Search Testing | ✅ | 3 results returned for "GitHub Agentic Workflows" |
| 7 | File Writing Testing | ✅ | /tmp/gh-aw/agent/smoke-test-claude-22487258068.txt created |
| 8 | Bash Tool Testing | ✅ | File contents verified via cat |
| 9 | Discussion Interaction Testing | ✅ | Discussion #18673 found; comic-book comment added |
| 10 | Agentic Workflows MCP Testing | ✅ | smoke-claude: active, compiled, engine=claude |
PR Review Tests (#11–17)
| # | Test | Result | Notes |
|---|---|---|---|
| 11 | Update PR Testing | ✅ | PR #18669 body appended with test message |
| 12 | PR Review Comment Testing | ✅ | 2 inline comments added on different files |
| 13 | Submit PR Review Testing | ✅ | Review submitted (COMMENT) |
| 14 | Resolve Review Thread Testing | Skipped: GraphQL thread node IDs not retrievable via safe inputs | |
| 15 | Add Reviewer Testing | ✅ | copilot added as reviewer to PR #18669 |
| 16 | Push to PR Branch Testing | ❌ | Failed: "context is not defined" error from safeoutputs |
| 17 | Close PR Testing | Skipped: no safe test PR to close |
Failure Details
Test #1 — GitHub MCP list_pull_requests token overflow
mcp__github__list_pull_requests response exceeded the 25,000 token limit (73,588 characters). Used safeinputs-gh as fallback for PR data retrieval.
Test #3 — Serena find_symbol EOF errors
activate_project succeeded and created project gh-aw. Subsequent find_symbol calls failed with MCP error 0: connection closed: calling "tools/call": client is closing: EOF. LSP server may need more initialization time.
Test #16 — push_to_pull_request_branch context error
File was committed locally (tmp-smoke-test-22487258068.txt on branch copilot/update-cli-tool-versions-another-one), but push failed with backend error: code=-32603, message=context is not defined.
References:
💥 [THE END] — Illustrated by Smoke Claude
- expires on Feb 27, 2026, 3:10 PM UTC