[prompt-clustering] Copilot Agent Prompt Clustering Analysis - Feb 7, 2026

[github-actions[bot]]

Analysis Date: 2026-02-07

Daily NLP-based clustering analysis of copilot agent task prompts to identify patterns and optimization opportunities.

Summary

Analysis Period: 2026-01-21 to 2026-02-07
Total Tasks Analyzed: 984
Clusters Identified: 3
Overall Success Rate: 69.3% (682 merged)
Average Task Size: 511 additions, 396 deletions

Key Findings

• Most Common Task Type: Dependency & Version Updates (553 tasks, 56.2%)
• Highest Success Rate: Dependency & Version Updates (75.6%)
• Lowest Success Rate: Bug Fixes & Error Resolution - Cluster 1 (54.4%)
• Most Complex Tasks: Bug Fixes & Error Resolution - Cluster 2 (1106 avg additions)

View Full Analysis Report

Cluster Overview

Cluster	Theme	Tasks	Success Rate	Avg Additions	Top Keywords
3	Dependency & Version Updates	553	75.6%	407	workflow, test, mcp, url
1	Bug Fixes & Error Resolution	285	54.4%	407	workflow, comments, url, issue
2	Bug Fixes & Error Resolution	146	74.7%	1106	url, fix, reference, workflow

Detailed Cluster Analysis

Cluster 3: Dependency & Version Updates

Statistics:

• Size: 553 tasks (56.2% of total)
• Success Rate: 75.6% (418 merged, 134 closed)
• Average Changes: 407 additions, 457 deletions
• Average Files Changed: 21.9
• Average Commits: 4.2

Characteristic Keywords: workflow, test, mcp, url, github, workflows
TF-IDF Terms: update, workflow, project, campaign, agentic

Example Tasks:

• #11120: Convert safe-outputs MCP server to HTTP transport
• #11567: Update AWF firewall to v0.11.2 with ACT agent container
• #11180: chore: recompile workflows after safe outputs handler changes

Sample Prompt

Run the update command and ensure that the Sentry MCP is updated. It should be upgraded to version 0.27.0, as shown in the dependency pull-request description attached below.
https://github.com/githubnext/gh-aw/pull/11013
``````
</details>

---

#### Cluster 1: Bug Fixes & Error Resolution (Lower Success)

**Statistics**:
- Size: 285 tasks (29.0% of total)
- Success Rate: 54.4% (155 merged, 127 closed)
- Average Changes: 407 additions, 428 deletions
- Average Files Changed: 14.2
- Average Commits: 3.2

**Characteristic Keywords**: workflow, comments, url, issue, discussion, section  
**TF-IDF Terms**: gh, gh aw, aw, workflow, comments

**Example Tasks**:
- [#13182](https://github.com/github/gh-aw/pull/13182): Refactor compiler to use grouped shell redirects (fix SC2129)
- [#11178](https://github.com/github/gh-aw/pull/11178): Add security alert burndown campaign with clustered fixes
- [#12076](https://github.com/github/gh-aw/pull/12076): Refactor large test files (>1500 lines) into focused modules under 1000 lines

<details>
<summary><b>Sample Prompt</b></summary>

``````
If you update the code that checks whether workflows are in sync in the agentic maintenance workflow, and the workflows are not in sync, and the `gh-aw-agent-token` secret is available, then the workflow should automatically create an issue and assign it to `@copilot`.
``````
</details>

---

#### Cluster 2: Bug Fixes & Error Resolution (Higher Complexity)

**Statistics**:
- Size: 146 tasks (14.8% of total)
- Success Rate: 74.7% (109 merged, 37 closed)
- Average Changes: 1106 additions, 101 deletions
- Average Files Changed: 20.6
- Average Commits: 3.2

**Characteristic Keywords**: url, fix, reference, workflow, job, logs  
**TF-IDF Terms**: url, reference, fix, job, logs

**Example Tasks**:
- [#13361](https://github.com/github/gh-aw/pull/13361): [WIP] Fix failing GitHub Actions workflow test
- [#12864](https://github.com/github/gh-aw/pull/12864): Verify CI test failures in run #21537657789
- [#13949](https://github.com/github/gh-aw/pull/13949): Simplify permissions: agent job ALWAYS gets contents:read

<details>
<summary><b>Sample Prompt</b></summary>

``````
format, lint js, fix js tests

---

Full Data Table (First 20 PRs)

PR #	Title	Cluster	Theme	Merged	Additions	Files
11060	Merge maintenance jobs and add comprehensive logging	1	Bug Fixes & Error Resolution	✅	101	3
11059	Install Go toolchain in daily-cli-performance workflow	1	Bug Fixes & Error Resolution	❌	12	2
11885	perf: Move regex compilation to package level	1	Bug Fixes & Error Resolution	✅	33	4
11822	Fix firewall-escape workflow post-issue job syntax error	1	Bug Fixes & Error Resolution	✅	5	2
11811	Remove verification step from copilot-setup-steps workflow	1	Bug Fixes & Error Resolution	✅	20	3
11798	Fix: Increase Daily Team Evolution Insights workflow timeout	1	Bug Fixes & Error Resolution	✅	2	2
11930	Add diagnostic logging and increase timeout for safe-outputs	1	Bug Fixes & Error Resolution	✅	72	7
11927	Optimize install.md for Copilot agent-driven repository setup	1	Bug Fixes & Error Resolution	✅	133	1
11916	Add --push and --create-pull-request flags	1	Bug Fixes & Error Resolution	✅	246	11
13998	Integrate gh-aw-mcpg v0.0.103	1	Bug Fixes & Error Resolution	✅	436	146
14129	Add unit tests for compiler_yaml_main_job.go	1	Bug Fixes & Error Resolution	✅	1070	86
14127	Fix daily-fact workflow action-tag	1	Bug Fixes & Error Resolution	✅	257	87
14120	Fix gh-aw binary availability for user-defined steps	1	Bug Fixes & Error Resolution	✅	77	24
14119	Fix textblob version check in Copilot PR NLP Analysis workflow	1	Bug Fixes & Error Resolution	✅	3	2
11098	Restrict Issue Monster to cookie-labeled issues	1	Bug Fixes & Error Resolution	✅	85	54
11093	Add error handling to agent output ingestion	1	Bug Fixes & Error Resolution	✅	596	3
11067	Add missing get_repository tool to repos toolset	1	Bug Fixes & Error Resolution	✅	2	2
13605	Update gh-aw-mcpg container image to v0.0.96	1	Bug Fixes & Error Resolution	✅	2	1
13576	Use AWF --enable-chroot mode	1	Bug Fixes & Error Resolution	✅	894	158
13717	Add secrets field to workflow schema with validation	1	Bug Fixes & Error Resolution	✅	195	4

Showing first 20 of 984 PRs (full table available in generated report)

Recommendations

Based on the clustering analysis, here are actionable recommendations:

1. Improve Success Rate for Complex Bug Fixes (Cluster 1)

Current State: 54.4% success rate
Issue: These tasks involve complex workflow changes and multiple dependencies
Recommendation:

• Break down complex bug fixes into smaller, focused PRs
• Provide more detailed context about workflow dependencies
• Include specific test cases or validation steps in prompts

2. Optimize Large Code Changes (Cluster 2)

Current State: Average 1106 additions per task
Issue: Large PRs are harder to review and more likely to have issues
Recommendation:

• Split large refactoring tasks into multiple smaller PRs
• Use staged rollout approach for major changes
• Consider using task decomposition in prompts

3. Leverage Successful Patterns from Dependency Updates (Cluster 3)

Current State: 75.6% success rate
Success Factors: Clear, specific prompts with version references
Recommendation:

• Apply similar prompt structure to other task types
• Provide specific references (URLs, PRs, issues) in all prompts
• Include expected outcome in task description

4. Improve Prompt Engineering for Bug Fixes

Observation: Bug fix prompts vary significantly in clarity
Recommendation:

• Use consistent prompt template for bug fixes
• Include: what's broken, expected behavior, relevant logs/errors
• Reference specific workflow runs or test failures

Methodology

This analysis used:

• Data Source: 1000 copilot-created PRs from last 30 days
• NLP Technique: TF-IDF vectorization + K-means clustering
• Features: Task prompts extracted from PR bodies
• Validation: Manual review of cluster themes and samples
• Tools: scikit-learn, pandas, Python NLP libraries

Next Steps

1. Historical Trending: Compare with previous analyses to track improvement
2. Workflow Metrics Integration: Add turn counts and duration data
3. Success Pattern Analysis: Deep dive into high-success-rate prompts
4. Failure Analysis: Investigate common failure patterns in Cluster 1

---

Workflow Run: §21774270699

This is an automated daily analysis. Full report and data available in workflow artifacts.

AI generated by Copilot Agent Prompt Clustering Analysis

• expires on Feb 14, 2026, 4:55 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-02-14T04:55:36.280Z.

Closed by Workflow