Safe Output Health Report - 2026-02-20

[github-actions[bot]]

Daily audit of safe output job health across all agentic workflow runs in the last 24 hours.

Executive Summary

• Period: 2026-02-19T04:22Z – 2026-02-20T04:22Z (last 24 hours)
• Runs Analyzed: 77
• Runs with Safe Output Jobs Executed: 12
• Runs with Safe Output Jobs Skipped (condition not met): 40
• Runs without Safe Output Jobs (no agent output): 25
• Total Safe Output Messages Processed: 26
• Safe Output Messages Succeeded: 21
• Safe Output Messages Failed: 4
• Overall Safe Output Success Rate: 80.8%
• Error Clusters Identified: 4

Safe Output Job Statistics

Job Type	Total Sent	Notes
add_comment	39	Most frequent; 2 failures (permission denied)
update_project	30	2 failures (view limit + cascading temp ID)
create_issue	29	No failures
add_labels	19	No failures
create_pull_request_review_comment	18	No failures
update_pull_request	9	No failures
submit_pull_request_review	9	No failures
remove_labels	8	No failures
create_discussion	7	No failures
push_to_pull_request_branch	6	No failures
dispatch_workflow	6	No failures
send_slack_message	6	No failures
unassign_from_user	5	No failures
create_project_status_update	5	No failures
resolve_pull_request_review_thread	4	No failures
noop	4	Correctly skipped (standalone step)
add_reviewer	4	No failures
update_issue	1	1 failure (context mismatch)
create_pull_request	1	No failures

Most reliable: create_issue, add_labels, create_pull_request_review_comment — all succeeded
Most failures: add_comment (2 failures), update_project (2 failures)

Error Clusters

Cluster 1: Permission Errors on PR Comments

• Count: 2 occurrences
• Affected Job: add_comment
• Affected Workflows: Agent Container Smoke Test (§22210660012), Smoke Temporary ID (§22210660018)
• Target: PR Fix push_to_pull_request_branch generating bad patch on issue_comment follow-up runs #16927 in github/gh-aw
• Event: pull_request on branch copilot/fix-patch-generation-bug

Sample Error:

##[error]Failed to add comment: Resource not accessible by integration
##[error]✗ Message 1 (add_comment) failed: Resource not accessible by integration
```

**Root Cause**: The `GITHUB_TOKEN` provided to the safe output job lacks write permission to comment on PR #16927. This is a known GitHub limitation for workflows triggered by pull requests from branches where the token has restricted scope — the token cannot comment on a PR that the integration hasn't been explicitly granted access to write on.

**Impact**: Both smoke test workflows failed to post their results comment on the PR, meaning PR authors received no inline feedback from these test runs.

---

#### Cluster 2: GitHub Project View Limit (API Constraint)

- **Count**: 1 occurrence  
- **Affected Job**: `update_project` (with embedded view creation)  
- **Affected Workflow**: Smoke Project ([§22210660010](https://github.com/github/gh-aw/actions/runs/22210660010))  
- **Target**: GitHub Project #24068  

**Sample Error:**
```
##[error]Failed to manage project: Views are limited to 50 per project
##[error]Failed to create configured view 1: Smoke Test Board
```

**Root Cause**: The GitHub Projects API enforces a hard limit of 50 views per project. Project #24068 used by the Smoke Project workflow has accumulated 50 views, preventing the smoke test from creating its expected "Smoke Test Board" view.

**Impact**: Smoke Project smoke test partially failed (1/7 messages). The view creation step failed, but 6 other project operations succeeded.

---

#### Cluster 3: Cascading Temporary ID Resolution Failure

- **Count**: 1 occurrence  
- **Affected Job**: `update_project`  
- **Affected Workflow**: Smoke Project ([§22210660010](https://github.com/github/gh-aw/actions/runs/22210660010))  

**Sample Error:**
```
##[error]Failed to manage project: draft_issue_id "aw_draft1" not found in temporary ID map and no draft_title provided for fallback lookup
##[error]✗ Message 4 (update_project) failed: draft_issue_id "aw_draft1" not found in temporary ID map
```

**Root Cause**: This is a cascading failure triggered by Cluster 2. When the initial `update_project` that created the draft issue (with `temporary_id: "aw_draft1"`) failed due to the view limit error, the temporary ID was never registered. A subsequent `update_project` message referencing `draft_issue_id: "aw_draft1"` failed because the ID wasn't in the map. No `draft_title` fallback was provided to allow the system to look up the draft issue by name.

**Impact**: Combined with Cluster 2, these two errors account for 2 of the 4 total failures, both in the same workflow run. The Smoke Project overall still reported as `success` (6/7 messages passed).

---

#### Cluster 4: Context Mismatch for `update_issue` with `triggering` Target

- **Count**: 1 occurrence  
- **Affected Job**: `update_issue`  
- **Affected Workflow**: CI Failure Doctor ([§22210689351](https://github.com/github/gh-aw/actions/runs/22210689351))  
- **Event**: `workflow_run`  

**Sample Error:**
```
##[warning]Target is "triggering" but not running in issue context, skipping update_issue
##[error]✗ Message 3 (update_issue) failed: Target is "triggering" but not running in issue context, skipping update_issue

Root Cause: The CI Failure Doctor is triggered by workflow_run events. The agent attempted to use update_issue with target: "triggering", which resolves the issue from the current triggering context. However, since the workflow is triggered by a CI run — not an issue — there is no "triggering issue" to update. The agent should have used an explicit issue_number in its update_issue output (the issue was #17076).

Impact: Low. The other 2 messages in this run succeeded — create_issue and add_comment both completed successfully. The update_issue failure means that issue #17076 was not closed/updated via safe output, but the new issue was still created correctly.

Root Cause Analysis

Permission Issues

The "Resource not accessible by integration" error for add_comment is a GitHub token scope issue specific to PR #16927. This PR was created from a branch (copilot/fix-patch-generation-bug) and the safe output job's integration token does not have the necessary write permission to comment on it. This may be related to:

1. The PR being from a fork (fork PRs have restricted token access)
2. The GitHub App not being authorized on this specific PR
3. The PR having been merged/closed, revoking access

Both affected smoke test workflows (smoke-test-tools for Agent Container and smoke-temporary-id) targeted the same PR, explaining why both failed simultaneously.

API Limit Issues

The project view limit of 50 is a hard GitHub API constraint for GitHub Projects (V2). The Smoke Project workflow creates a new view on every run. Over time, these views accumulate until the limit is hit. This is an operational maintenance issue requiring manual cleanup of old project views.

Agent Instruction Issues

The update_issue context mismatch in CI Failure Doctor is an agent instruction problem. The agent (claude) was instructed or inferred that it could update the "triggering" issue, but since the workflow is triggered by workflow_run, there is no triggering issue. This is an edge case where the workflow's trigger type (schedule/workflow_run vs. issue events) must match the agent's intent to use target: "triggering".

Recommendations

Critical Issues (Immediate Action Required)

1. Clean Up Project #24068 Views

   • Priority: High
   • Root Cause: Project has hit the 50-view GitHub API limit
   • Recommended Action: Manually delete old/stale views from Project #24068 to free up view slots. Also consider modifying the Smoke Project workflow to reuse an existing view rather than creating a new one each run, or add logic to delete previous smoke test views before creating new ones.
   • Affected: smoke-project workflow

2. Investigate Permission Error for add_comment on PR Fix push_to_pull_request_branch generating bad patch on issue_comment follow-up runs #16927

   • Priority: High
   • Root Cause: GITHUB_TOKEN lacks write access to comment on this PR
   • Recommended Action: Check if PR Fix push_to_pull_request_branch generating bad patch on issue_comment follow-up runs #16927 is from a fork (forks require explicit pull-request-target trigger for write access). Review the smoke test workflow's permissions block to ensure pull-requests: write is configured. If the PR is merged/closed, determine whether the smoke tests need to target a different PR.
   • Affected: smoke-test-tools, smoke-temporary-id workflows

Bug Fixes Required

1. Smoke Project: Reuse or Cleanup Project Views

   • Location: .github/workflows/smoke-project.md
   • Problem: New view created on every run, accumulating until 50-view limit hit
   • Fix: Before creating a new view, check if a view named "Smoke Test Board" already exists and either reuse it or delete it first. Alternatively, add post-run cleanup to delete the created view.

2. CI Failure Doctor: Use Explicit issue_number in update_issue

   • Location: CI Failure Doctor agent prompt / workflow configuration
   • Problem: Agent uses target: "triggering" for update_issue in a workflow_run-triggered workflow where no issue context exists
   • Fix: Update the agent prompt to always provide an explicit issue_number when calling update_issue in CI Failure Doctor, since it runs from workflow_run events. The agent already knows the issue number (it comments on the issue with add_comment).

3. Smoke Project: Provide draft_title Fallback for update_project

   • Location: Smoke Project agent / workflow prompt
   • Problem: When referencing a draft_issue_id via temporary ID, no draft_title fallback is provided, causing cascading failure
   • Fix: When using draft_issue_id: "aw_draft1" in subsequent update_project messages, also include draft_title as a fallback. This allows the system to resolve the draft issue by name even if the temporary ID map lookup fails.

Process Improvements

1. View Lifecycle Management for Project Smoke Tests

   • Current State: Smoke tests create new project views on every run, relying on manual cleanup
   • Proposed: Implement idempotent view management — check for existing "Smoke Test Board" view, delete it if found, then create a new one (or reuse). Set a retention policy.
   • Benefits: Eliminates the 50-view API limit problem permanently

2. Add Fallback Data in Temporary ID References

   • Current State: When agents reference temporary IDs (e.g., draft_issue_id: aw_draft1), they often don't provide fallback lookup data
   • Proposed: Update agent instructions to always include a human-readable fallback field (like draft_title) when referencing temporary IDs that may not be in the map
   • Benefits: Prevents cascading failures when parent operations fail

Work Item Plans

Work Item 1: Fix Project View Accumulation in Smoke Project

• Type: Bug Fix
• Priority: High
• Description: The Smoke Project workflow creates a new project view named "Smoke Test Board" on every run. After 50 views, the GitHub API blocks new view creation. This causes smoke test failures and requires manual intervention.
• Acceptance Criteria:
  • Smoke Project workflow no longer fails due to "Views are limited to 50 per project"
  • Old/stale smoke test views are cleaned up automatically
  • Project view count stays manageable over time
• Technical Approach: Modify the agent prompt or safe output configuration to delete existing "Smoke Test Board" views before creating a new one. Alternatively, query existing views and reuse an existing one.
• Effort: Small
• Dependencies: Access to project #24068 admin to delete existing views manually first

Work Item 2: Fix CI Failure Doctor update_issue Context Error

• Type: Bug Fix
• Priority: Medium
• Description: CI Failure Doctor uses update_issue with target: "triggering" but runs on workflow_run events where no triggering issue exists. The agent already knows the issue number (it creates and comments on it).
• Acceptance Criteria:
  • update_issue in CI Failure Doctor uses explicit issue_number instead of target: "triggering"
  • No "Target is triggering but not running in issue context" errors
• Technical Approach: Update the CI Failure Doctor agent prompt to specify issue_number explicitly in update_issue outputs. The agent can reference the aw_newci temporary ID or pass the actual issue number.
• Effort: Small
• Dependencies: None

Work Item 3: Add draft_title Fallback for Temporary ID References

• Type: Enhancement
• Priority: Medium
• Description: When a temporary ID fails to register (due to parent operation failing), downstream operations that reference it fail with no fallback. Adding a draft_title fallback would allow recovery.
• Acceptance Criteria:
  • Agents consistently include fallback data when referencing temporary IDs
  • System documentation updated to recommend fallback fields
• Technical Approach: Update Smoke Project agent prompt to include draft_title alongside draft_issue_id in update_project messages. Also consider updating safe output handler documentation.
• Effort: Small
• Dependencies: None

Historical Context

This is the first audit of safe output health. No historical baseline exists for trend comparison.

Initial Baseline (2026-02-20):

• Safe output success rate: 80.8% (21/26 messages)
• Most common failure: Permission errors (2 occurrences)
• Most reliable job types: create_issue, add_labels, create_pull_request_review_comment
• Most problematic job types: add_comment (2/39 messages failed), update_project (2/30 messages failed)

Next Steps

• Delete stale views from Project #24068 to restore Smoke Project functionality
• Fix CI Failure Doctor to use explicit issue_number in update_issue
• Update Smoke Project prompt to provide draft_title fallback
• Investigate PR Fix push_to_pull_request_branch generating bad patch on issue_comment follow-up runs #16927 permission issue (check if fork-based, verify token permissions)
• Run next day's audit to establish trend baseline

References:

• §22210660010 — Smoke Project (project view + temp ID errors)
• §22210660012 — Agent Container Smoke Test (permission error)
• §22210689351 — CI Failure Doctor (context mismatch)

AI generated by Safe Output Health Monitor

• expires on Feb 27, 2026, 4:31 AM UTC

[github-actions[bot]]

🤖 Beep boop! The smoke test agent was here! 🚀

I just swooped in, ran a bunch of tests, and I'm happy to report: things exist and are working (probably).

Consider this my digital graffiti: smoke-test-copilot-22211586699 was here ✨

Now back to your regularly scheduled discussions...

📰 BREAKING: Report filed by Smoke Copilot

[github-actions[bot]]

💥 KA-BOOM! 💥

WHOOOOSH! The smoke test agent has ARRIVED! 🦸

⚡ ZAP! Claude engine operational — all systems NOMINAL!

🔥 "With great workflows come great responsibilities." 🔥

✨ POW! Tests 1 through 10: ALL GREEN! ✨

💨 The agent swoops in, validates every tool, and disappears into the void...

🎯 SMOKE TEST COMPLETE — Run §22211583283 — MISSION ACCOMPLISHED! 🎯

💥 [THE END] — Illustrated by Smoke Claude