CI Failure Analysis #428
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: CI Failure Analysis | |
| # Automatically analyze failed CI runs and provide actionable fix suggestions | |
| # Triggers when Markdown Lint, Check Links, or Validate Workflows fail | |
| # | |
| # Testing: Intentionally fail markdownlint (e.g., change - to * in any .md file) | |
| on: | |
| workflow_run: | |
| workflows: | |
| - "Markdown Lint" | |
| - "Check Links" | |
| - "Validate Workflows" | |
| types: [completed] | |
| # Cancel in-progress analysis for same workflow run | |
| concurrency: | |
| group: ${{ github.workflow }}-${{ github.event.workflow_run.id }} | |
| cancel-in-progress: true | |
| jobs: | |
| analyze-failure: | |
| name: Analyze CI Failure | |
| # Only run on failure, skip bot triggers to prevent loops | |
| if: | | |
| github.event.workflow_run.conclusion == 'failure' && | |
| github.event.workflow_run.actor.login != 'dependabot[bot]' && | |
| github.event.workflow_run.actor.login != 'claude[bot]' | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 10 | |
| permissions: | |
| contents: read | |
| pull-requests: write | |
| actions: read | |
| id-token: write | |
| steps: | |
| - name: Analyze failure with Claude | |
| uses: anthropics/claude-code-action@68cfeead1890300cc87935dbe2c023825be87b8a # v1.0.52 | |
| with: | |
| claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} | |
| prompt: | | |
| A CI workflow has failed. Analyze the failure and help the developer fix it. | |
| ## Failed Workflow Details | |
| - **Workflow**: ${{ github.event.workflow_run.name }} | |
| - **Run ID**: ${{ github.event.workflow_run.id }} | |
| - **Run URL**: ${{ github.event.workflow_run.html_url }} | |
| - **Commit SHA**: ${{ github.event.workflow_run.head_sha }} | |
| - **Branch**: ${{ github.event.workflow_run.head_branch }} | |
| - **Actor**: ${{ github.event.workflow_run.actor.login }} | |
| - **Repository**: ${{ github.repository }} | |
| ## Instructions | |
| ### Step 1: Get the failure logs | |
| First, download and examine the workflow run logs: | |
| ```bash | |
| gh run view ${{ github.event.workflow_run.id }} --log | |
| ``` | |
| If that's too verbose, get the job summary first: | |
| ```bash | |
| gh run view ${{ github.event.workflow_run.id }} --json jobs --jq '.jobs[] | select(.conclusion == "failure") | {name: .name, conclusion: .conclusion}' | |
| ``` | |
| ### Step 2: Analyze the failure | |
| Based on which workflow failed, look for specific error patterns: | |
| **For "Markdown Lint" failures:** | |
| - Look for lines like: `filename:line:column MD00X/rule-name message` | |
| - Common issues: wrong list style (should use `-`), wrong header style (should use ATX `#`) | |
| **For "Check Links" (lychee) failures:** | |
| - Look for broken URLs with HTTP status codes (404, 403, etc.) | |
| - Check if the URL has moved or if it's a false positive | |
| **For "Validate Workflows" (actionlint) failures:** | |
| - Look for YAML syntax errors or invalid workflow configurations | |
| - Check for missing required fields, invalid expressions, or deprecated syntax | |
| ### Step 3: Find the associated PR | |
| Try to find the pull request associated with this commit: | |
| ```bash | |
| gh pr list --search "${{ github.event.workflow_run.head_sha }}" --state open --json number,title,url --jq '.[0]' | |
| ``` | |
| If no PR is found (e.g., push to main), note this - you won't be able to comment. | |
| ### Step 4: Post analysis comment | |
| If a PR was found, post a helpful comment with your analysis. The comment should include: | |
| 1. **Summary**: What failed and why (1-2 sentences) | |
| 2. **Failures Found**: List each error with file, line, and explanation | |
| 3. **How to Fix**: Specific commands or code changes | |
| Use this format for the comment: | |
| ```markdown | |
| ## ❌ CI Failure Analysis: {Workflow Name} | |
| **Run**: [#{run_id}]({run_url}) | |
| **Commit**: `{sha:7}` | |
| ### Summary | |
| {Brief explanation of what failed} | |
| ### Failures Found | |
| | File | Line | Issue | | |
| |------|------|-------| | |
| | `{file}` | {line} | {description} | | |
| ### How to Fix | |
| {For markdown failures} | |
| **Option 1**: Auto-fix with markdownlint | |
| ```bash | |
| markdownlint "{file}" --fix | |
| ``` | |
| **Option 2**: Manual fix | |
| {Explain specific change needed} | |
| {For link failures} | |
| - Check if URL has moved and update the link | |
| - Or add to `.lycheeignore` if the link is expected to fail in CI | |
| {For workflow validation failures} | |
| - Show corrected YAML syntax | |
| --- | |
| 🤖 Analyzed by [Claude](https://claude.ai/code) | |
| ``` | |
| Post the comment using: | |
| ```bash | |
| gh pr comment {pr_number} --body "..." | |
| ``` | |
| ### Step 5: Report outcome | |
| After posting (or if no PR found), summarize what you found and what action was taken. | |
| claude_args: | | |
| --model claude-opus-4-6 | |
| --allowedTools "Bash(gh run:*),Bash(gh pr:*)" |