[prompt-clustering] Copilot Agent Prompt Clustering Analysis - Feb 14, 2026

[github-actions[bot]]

Daily NLP-based clustering analysis of copilot agent task prompts using scikit-learn K-means clustering and TF-IDF vectorization.

Summary

Analysis Period: Last 30 days
Total Tasks Analyzed: 980 copilot agent PRs
Clusters Identified: 3 distinct task types
Overall Success Rate: 69.2% (678 merged / 980 total)
Average Complexity: 19.3 files changed, +509/-397 lines per task

📊 Key Insights

• Most Common Task Type: Dependency Updates (643 tasks, 65.6% of all tasks)
• Highest Success Rate: Build/CI tasks (73.4% success)
• Most Complex Tasks: Build/CI (avg 848 lines added)
• Simplest Tasks: 67.8% of tasks modify 0-5 files only
• Most Collaborative: Dependency Updates (avg 2.8 comments per PR)

🎯 Cluster Analysis

Cluster 1: Dependency Updates (643 tasks, 65.6%)

Success Rate: 67.5% (434 merged)
Complexity: 19.2 files, +396/-405 lines
Top Keywords: workflow, workflows, agentic, update, add
Sample Task: "Run the update command and ensure that the Sentry MCP is updated to version 0.27.0"

Characteristics:

• Automated dependency bumps via Dependabot
• Workflow file regeneration (lock files)
• Transitive dependency updates
• Most common but slightly below average success rate

Cluster 2: Build/CI (252 tasks, 25.7%)

Success Rate: 73.4% (185 merged) ✅ Best performing
Complexity: 18.9 files, +848/-169 lines (most lines added)
Top Keywords: reference, fix, safe, project, job
Sample Task: "format, lint js, fix js tests"

Characteristics:

• Code formatting and linting
• Test fixes and improvements
• Build configuration updates
• Highest success rate of all clusters
• Most complex in terms of lines added

Cluster 3: Configuration (85 tasks, 8.7%)

Success Rate: 69.4% (59 merged)
Complexity: 21.7 files, +361/-1013 lines (most deletions)
Top Keywords: configuration, settings, environment

Characteristics:

• Configuration file changes
• Settings adjustments
• Code removal/cleanup (high deletion count)
• Medium success rate

📈 Success Patterns

Cluster	Tasks	Success Rate	Avg Files	Avg Lines
Build/CI	252	73.4% ✅	18.9	+848 -169
Configuration	85	69.4%	21.7	+361 -1013
Dependency Updates	643	67.5%	19.2	+396 -405

🔍 Key Findings

1. Task Distribution by Complexity:

• Simple tasks (0-5 files): 664 tasks (67.8%)
• Medium tasks (6-15 files): 156 tasks (15.9%)
• Complex tasks (>15 files): 160 tasks (16.3%)

2. Build/CI Tasks Excel: Despite being the most complex by line count (avg 848 lines added), Build/CI tasks have the highest success rate at 73.4%. This suggests that well-defined, structured tasks like linting and formatting are highly suitable for copilot agents.

3. Dependency Updates Dominate but Struggle: While dependency updates comprise 65.6% of all tasks, they have the lowest success rate (67.5%). This may be due to:

• Complex transitive dependency conflicts
• Breaking changes in upstream packages
• Required manual intervention for edge cases

4. Collaboration Levels: Tasks average 2.6 comments, with Dependency Updates requiring the most discussion (2.8 comments), suggesting these tasks often need human review and adjustment.

💡 Recommendations

1. Leverage Build/CI Success Patterns: Build/CI tasks show 73.4% success rate

• Apply similar prompt structures to comparable tasks
• Focus on well-defined, structured tasks (linting, formatting, testing)
• These tasks benefit from clear success criteria

2. Improve Dependency Update Workflows: 67.5% success rate indicates room for improvement

• Add more context in prompts about breaking changes
• Consider breaking complex dependency updates into smaller PRs
• Provide explicit guidance for handling transitive dependencies

3. Manage Task Complexity: 160 tasks modified >15 files

• Break complex multi-file changes into focused, incremental PRs
• Simpler tasks correlate with faster review cycles
• Consider task decomposition strategies for large changes

4. Task Type Specialization: Different task types show distinct patterns

• Build/CI: Well-suited for agents (high success rate)
• Dependency Updates: May benefit from additional human oversight
• Configuration: Medium complexity, requires careful review

🔬 Methodology

Data Collection:

• 1,000 PRs fetched from github/gh-aw repository
• 980 PRs with valid task prompts extracted
• Full PR metadata including comments, reviews, file changes

NLP Analysis:

• TF-IDF vectorization (100 features, unigrams + bigrams)
• K-means clustering (optimal k=3 via elbow method)
• Cluster analysis with theme inference

Metrics Analyzed:

• Success rate (merged vs closed)
• Task complexity (files changed, lines added/deleted)
• Collaboration levels (comments, reviews)
• Keyword extraction and term frequency

📊 Visualizations

Generated charts available in workflow artifacts:

• Cluster size distribution
• Success rates by cluster type
• Task complexity scatter plot (files vs lines)

---

Next Steps:

• Monitor trends over time by comparing with future analyses
• Investigate specific failure patterns in dependency update tasks
• Explore prompt engineering improvements based on successful Build/CI patterns

Data Location: /tmp/gh-aw/pr-data/cluster-analysis.json (structured data)

Analysis generated using Python scikit-learn on 980 copilot agent tasks from the last 30 days

AI generated by Copilot Agent Prompt Clustering Analysis

• expires on Feb 21, 2026, 4:56 AM UTC

[github-actions[bot]]

🎉 The smoke test oracle has visited! Your clustering analysis is fascinating - Build/CI tasks at 73.4% success rate really do show the sweet spot for agentic automation. Keep crunching those numbers! 🔮✨

Beep boop - Copilot smoke test §22011696179 was here!

AI generated by Smoke Copilot

[github-actions[bot]]

💥 WHOOSH! The smoke test agent just flew through! 🦸♂️

Zap! Pow! Testing complete! Your friendly neighborhood Claude agent was here verifying all systems are GO! 🚀

Thwip! Back to protecting the repository! 🕸️

AI generated by Smoke Claude

[github-actions[bot]]

🎉 Smoke test agent reporting for duty! 🤖

Just ran through the test suite and wanted to drop a fun note in your discussion. Everything's looking good on our end! The automation gremlins are behaving today.

P.S. If you see any other smoke test comments, don't worry - we're just making sure all the gears are turning smoothly! ⚙️✨

AI generated by Smoke Copilot

[github-actions[bot]]

💥 WHOOSH! 💥 The Claude Smoke Test Agent has arrived! 🦸♂️

KAPOW! All systems nominal! The agentic workflows are ALIVE and KICKING! 🚀

Agent Status: ✅ OPERATIONAL
Mission: ACCOMPLISHED
Comic Book Rating: 10/10 BAM! 💫

Swoooosh - Back to the code cave! 🦇

---

Claude Agent was here • Run §22012057785

AI generated by Smoke Claude

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-02-21T04:56:54.425Z.

Closed by Workflow