-
Notifications
You must be signed in to change notification settings - Fork 268
Closed as not planned
Labels
Description
Summary
Smoke test for Claude engine validation — Run §22526414043
Overall Status: PARTIAL — Test 3 (Serena find_symbol) failed due to MCP connection drop; Test 17 skipped
PR used for review testing: #18754 (copilot/fix-safe-outputs-target-repo-support)
Timestamp: 2026-02-28
Core Test Results (#1–10)
| # | Test | Status | Notes |
|---|---|---|---|
| 1 | GitHub MCP — Review last 2 merged PRs | ✅ | PRs #18876, #18879 found |
| 2 | SafeInputs GH CLI — Query 2 PRs | ✅ | PRs #18878, #18754 returned |
| 3 | Serena MCP — Activate project + find_symbol | ❌ | Project activated; find_symbol failed (MCP EOF/connection closed) |
| 4 | Make Build | ✅ | make build succeeded |
| 5 | Playwright — Navigate to github.com | ✅ | Title: "GitHub · Change is constant..." |
| 6 | Tavily Web Search | ✅ | 3 results for "GitHub Agentic Workflows" |
| 7 | File Writing | ✅ | /tmp/gh-aw/agent/smoke-test-claude-22526414043.txt created |
| 8 | Bash Tool | ✅ | File contents verified via cat |
| 9 | Discussion Interaction | ✅ | Discussion #18881 found, comment added |
| 10 | Agentic Workflows MCP | ✅ | smoke-claude workflow: active, engine: claude |
PR Review Safe Output Test Results (#11–17)
| # | Test | Status | Notes |
|---|---|---|---|
| 11 | Update PR body | ✅ | Appended test message to PR #18754 |
| 12 | PR Review Comments | ✅ | 2 inline comments on .changeset/ and add_labels.cjs |
| 13 | Submit PR Review | ✅ | COMMENT review submitted |
| 14 | Resolve Review Thread | ✅ | Thread PRRT_kwDOPc1QR85xU3uQ resolved |
| 15 | Add Reviewer | ✅ | copilot added as reviewer |
| 16 | Push to PR Branch | ✅ | Smoke test file pushed to copilot/fix-safe-outputs-target-repo-support |
| 17 | Close PR | Skipped — no safe test PR to close |
Test 3 Error Details
Serena activate_project succeeded with response:
Created and activated a new project with name 'gh-aw' at /home/runner/work/gh-aw/gh-aw. Programming languages: go, typescript, bash
However, both find_symbol calls failed:
- First attempt:
MCP error 0: calling "tools/call": EOF - Second attempt (with relative_path):
MCP error 0: connection closed: calling "tools/call": client is closing: EOF - Re-activation attempt:
MCP error 0: connection closed: calling "tools/call": client is closing: EOF
The Serena MCP server appears to crash or disconnect after activate_project returns. This is likely an issue with the Serena MCP server configuration or resource limits in this environment.
Agentic Workflows Status Detail
[{
"workflow": "smoke-claude",
"engine_id": "claude",
"compiled": "No",
"status": "active",
"triggers": {
"pull_request": {"names": ["smoke"], "types": ["labeled"]},
"reaction": "heart",
"schedule": "every 12h",
"status-comment": true,
"workflow_dispatch": null
}
}]References:
💥 [THE END] — Illustrated by Smoke Claude
- expires on Feb 28, 2026, 8:36 PM UTC
Reactions are currently unavailable