[prompt-clustering] Copilot Agent Prompt Clustering Analysis - 2026-02-12

[github-actions[bot]]

Daily NLP-based clustering analysis of copilot agent task prompts to identify patterns, success rates, and optimization opportunities.

Summary

Analysis Period: Last 30 days
Total Tasks Analyzed: 1000
Clusters Identified: 7
Overall Success Rate: 69.0%
Total Merged PRs: 690

Quick Insights

• Most Common Pattern: Bugfix Tasks (test, tests, workflow) (236 tasks, 76.3% success)
• Highest Success Rate: Bugfix Tasks (test, tests, workflow) (76.3% success)
• Most Complex: Bugfix Tasks (mcp, server, v0) (38.5 files changed on avg)
• Top Category: Bugfix (653 tasks - 65.3% of all work)

Key Findings

1. Success Rate Variance by Task Type

Success rates vary by 21 percentage points across clusters, indicating task type significantly impacts outcomes:

• Highest: Test tasks (76.3% success) - Focused scope, clear validation
• Lowest: Workflow/issue management tasks (55.4% success) - More ambiguous requirements
• Finding: Well-defined, testable tasks perform significantly better

2. Complexity and Success Correlation

Tasks with higher file change counts don't necessarily have lower success rates:

• MCP server updates: 38.5 files changed, 67.1% success
• Test/CI fixes: 17.7 files changed, 76.3% success
• Finding: Success depends more on task clarity than code volume

3. Task Distribution Shows Clear Patterns

Top 3 categories account for 89.7% of all copilot agent work:

• Bugfix: 65.3% (653 tasks, 68.9% success)
• Update Deps: 14.5% (145 tasks, 66.2% success)
• Config: 9.9% (99 tasks, 69.7% success)

Notably, feature additions and tests show highest success:

• Feature: 88.0% success (25 tasks)
• Test: 85.2% success (27 tasks)

View Detailed Cluster Analysis

Cluster Breakdown

Cluster 4: Bugfix Tasks (test, tests, workflow)

• Size: 236 tasks (23.6% of total)
• Success Rate: 76.3% (180/236 merged)
• Top Keywords: test, tests, workflow, job, ci, cjs, error
• Average Metrics:
  • Commits: 3.7
  • Files Changed: 17.7
  • Comments: 2.2
  • Reviews: 0.7
• Sample Tasks:
  • Fix ephemerals tests after blockquote prefix requirement
  • Improve error messages for invalid target configuration

Why This Works: Clear validation criteria, isolated scope, immediate feedback from test runs.

Cluster 3: Bugfix Tasks (workflows, agentic, github)

• Size: 187 tasks (18.7% of total)
• Success Rate: 75.9% (142/187 merged)
• Top Keywords: workflows, agentic, github, workflow, md, file
• Average Metrics:
  • Commits: 4.0
  • Files Changed: 20.8
  • Comments: 3.3
  • Reviews: 1.1
• Sample Tasks:
  • Update parent issue template for agentic-workflow failures
  • Auto-assign @copilot to workflow sync issues

Why This Works: Domain-specific context (agentic workflows), clear file targets.

Cluster 2: Bugfix Tasks (workflow, issue, workflows) ⚠️ Low Success

• Size: 184 tasks (18.4% of total)
• Success Rate: 55.4% (102/184 merged) ⚠️
• Top Keywords: workflow, issue, workflows, daily, section, gh, gh aw
• Average Metrics:
  • Commits: 2.9
  • Files Changed: 11.2
  • Comments: 0.8
  • Reviews: 0.4
• Sample Tasks:
  • Install Go toolchain in daily-cli-performance workflow
  • Merge maintenance jobs and add comprehensive logging

Why This Struggles: Vague requirements, cross-cutting concerns, multiple potential solutions.

Cluster 6: Bugfix Tasks (mcp, server, v0)

• Size: 167 tasks (16.7% of total)
• Success Rate: 67.1% (112/167 merged)
• Top Keywords: mcp, server, v0, mcp server, tool, gateway, tools
• Average Metrics:
  • Commits: 3.7
  • Files Changed: 38.5 (highest complexity)
  • Comments: 4.2
  • Reviews: 0.6
• Sample Tasks:
  • Update Sentry MCP server to 0.27.0
  • Add interactive engine selection and secret configuration

Note: High complexity but reasonable success - suggests familiarity with MCP patterns helps.

Cluster 1: Bugfix Tasks (project, safe, safe outputs)

• Size: 95 tasks (9.5% of total)
• Success Rate: 73.7% (70/95 merged)
• Top Keywords: project, safe, safe outputs, outputs, output, field
• Average Metrics:
  • Commits: 4.3
  • Files Changed: 15.4
  • Comments: 3.4
  • Reviews: 0.8

Cluster 7: Bugfix Tasks (campaign, security, md)

• Size: 68 tasks (6.8% of total)
• Success Rate: 61.8% (42/68 merged)
• Top Keywords: campaign, security, md, project, workflows, docs

Cluster 5: Config Tasks (node, bin, firewall)

• Size: 63 tasks (6.3% of total)
• Success Rate: 66.7% (42/63 merged)
• Top Keywords: node, bin, firewall, docs, command, gh

Success Rate Comparison

Cluster	Description	Tasks	Success	Commits	Files
4	Test/CI Fixes	236	76.3%	3.7	17.7
3	Agentic Workflow Bugs	187	75.9%	4.0	20.8
1	Safe Outputs	95	73.7%	4.3	15.4
6	MCP Server Updates	167	67.1%	3.7	38.5
5	Config Changes	63	66.7%	3.2	12.3
7	Campaign/Security	68	61.8%	4.8	8.4
2	Workflow/Issue Mgmt	184	55.4% ⚠️	2.9	11.2

Task Category Performance

Category	Count	Success Rate	% of Total
Feature	25	88.0% ✨	2.5%
Test	27	85.2%	2.7%
Refactor	1	100.0%	0.1%
Config	99	69.7%	9.9%
Bugfix	653	68.9%	65.3%
Update Deps	145	66.2%	14.5%
Docs	48	58.3%	4.8%
Other	2	50.0%	0.2%

Actionable Recommendations

1. Optimize for High-Success Patterns 🎯

Tasks with clear validation and isolated scope show 76%+ success:

• Test fixes and CI improvements
• Focused feature additions
• Version updates with clear before/after states

Action: When creating prompts, explicitly define:

• Success criteria (tests pass, build succeeds)
• Scope boundaries (which files/modules)
• Validation steps (how to verify)

2. Improve Low-Performing Task Types ⚠️

Workflow/issue management tasks struggle (55.4% success):

• Vague requirements ("merge maintenance jobs")
• Cross-cutting concerns (multiple files, unclear boundaries)
• Ambiguous completion criteria

Action: Break these into smaller, focused tasks:

• ❌ "Merge maintenance jobs and add comprehensive logging"
• ✅ "Consolidate duplicate job definitions in workflow X"
• ✅ "Add structured logging to maintenance job Y"

3. Leverage Domain Specialization 🔧

MCP server tasks show 67% success despite 38 files changed:

• Suggests pattern recognition helps with complex tasks
• Clear architectural patterns (MCP protocol, server structure)

Action: When working on complex domains:

• Reference existing patterns explicitly
• Point to similar successful PRs
• Provide architectural context upfront

4. Capitalize on Feature/Test Success ✨

Feature additions (88%) and test tasks (85%) show exceptional results:

• Clear requirements and validation
• Well-defined interfaces
• Immediate feedback

Action: Use these task types as templates:

• Frame bugs as "add test that reproduces X, then fix"
• Frame refactors as "implement Y feature with cleaner approach"
• Always include test coverage as acceptance criteria

Data Observations

Commit Patterns:

• Average: 3.8 commits per PR
• Range: 2.9 - 4.8 commits across clusters
• Higher commits correlate with complexity, not success

Review Activity:

• Overall low review counts (0.4 - 1.1 avg)
• Suggests heavy reliance on automated checks
• Human review concentrated on complex changes (MCP: 0.6, Agentic workflows: 1.1)

File Change Distribution:

• Most tasks: 8-21 files changed
• Outlier: MCP updates at 38.5 files (still 67% success)
• File count alone doesn't predict failure

Methodology

Analysis Approach:

1. Extracted 1000 copilot PRs from last 30 days
2. Cleaned and vectorized task prompts using TF-IDF
3. Applied K-means clustering (k=7 optimal via silhouette analysis)
4. Analyzed success rates, complexity metrics, and patterns per cluster
5. Correlated task characteristics with outcomes

Data Sources:

• GitHub PR data (titles, bodies, metadata)
• PR review history and comments
• Commit and file change metrics
• Merge status as success indicator

Limitations:

• Success measured by merge status (doesn't capture code quality)
• 30-day window (may miss seasonal patterns)
• Text-based clustering (may group semantically different tasks with similar wording)

Visualizations Generated

Charts available in analysis artifacts:

• cluster_distribution.png - Task distribution across 7 clusters
• success_rate.png - Success rate by cluster (76% to 55% range)
• category_distribution.png - Task category breakdown (Bugfix dominates)
• complexity_metrics.png - Commits, files, comments by cluster

Next Steps

Immediate Actions:

1. Review Cluster 2 (workflow/issue mgmt) failed PRs to identify common patterns
2. Create prompt templates based on high-success clusters (4, 3)
3. Document best practices for breaking down complex tasks

Future Analysis:

1. Trend analysis: Track cluster success rates over time
2. Temporal patterns: Do certain task types succeed better at different times?
3. Prompt engineering: A/B test structured vs unstructured prompts
4. Failure analysis: Deep dive into common failure modes per cluster

Data Collection:

1. Add workflow turn counts and duration metrics
2. Track token usage per cluster
3. Correlate with human review patterns
4. Monitor prompt template adoption impact

AI generated by Copilot Agent Prompt Clustering Analysis

• expires on Feb 19, 2026, 5:06 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-02-19T05:06:38.942Z.

Closed by Workflow