Smoke Test: Claude - 22524720389 #18868
Description
Summary
Automated smoke test for Claude engine, run §22524720389 on 2026-02-28. PR used for review tests: #18843.
Overall Status: PARTIAL (1 failed, 2 skipped)
Test Results
| # | Test | Status | Notes |
|---|---|---|---|
| 1 | GitHub MCP - last 2 merged PRs | ✅ | Found #18713 ("Update checkout front matter") and #18846 ("Remove dead script stubs") |
| 2 | safeinputs-gh CLI - list 2 PRs | ✅ | Returned #18855 and #18843 |
| 3 | Serena MCP - activate + find_symbol | ❌ | activate_project succeeded; find_symbol failed repeatedly with EOF/connection closed errors |
| 4 | Make build | ✅ | make build succeeded, binary compiled |
| 5 | Playwright - GitHub.com title | ✅ | Title: "GitHub · Change is constant. GitHub keeps you ahead. · GitHub" |
| 6 | Tavily web search | ✅ | 3 results returned for "GitHub Agentic Workflows" |
| 7 | File writing | ✅ | Created /tmp/gh-aw/agent/smoke-test-claude-22524720389.txt |
| 8 | Bash verify file | ✅ | cat confirmed file contents |
| 9 | Discussion interaction | ✅ | Got discussion #18863, comic-book comment added |
| 10 | Agentic Workflows MCP | ✅ | smoke-claude status: active, compiled: Yes, engine: claude |
| 11 | Update PR body | ✅ | Appended test message to PR #18843 |
| 12 | PR review comments | ✅ | 2 inline comments on .changeset file and compiler_activation_jobs.go |
| 13 | Submit PR review | ✅ | COMMENT review submitted on PR #18843 |
| 14 | Resolve review thread | Skipped — review thread node IDs not accessible via available tooling | |
| 15 | Add reviewer | ✅ | copilot added as reviewer to PR #18843 |
| 16 | Push to PR branch | ✅ | smoke-test-22524720389.txt committed and pushed to copilot/update-activation-job-logs |
| 17 | Close PR | Skipped — triggering PR #18843 must not be closed |
Failure Details
Test 3 — Serena find_symbol (❌)
activate_project at /home/runner/work/gh-aw/gh-aw succeeded (Go + TypeScript + Bash detected). Subsequent find_symbol calls failed consistently:
MCP error 0: connection closed: calling "tools/call": client is closing: EOF
The Serena language server appears to start but then closes the connection before responding to symbol queries. This may indicate a language server startup timeout or resource constraint in the CI environment.
References:
💥 [THE END] — Illustrated by Smoke Claude
- expires on Feb 28, 2026, 6:55 PM UTC