Open
Conversation
- Scaffold eval directory, rubric schema, and runner skeleton (task .1) - Author priority rubric YAML files for 10-15 skills (task .2) - Build A/B effectiveness eval runner with LLM judge (task .3) - Add CI workflow, baseline regression, and documentation (task .4) - Create offline activation eval dataset and runner (task .5) - Develop size impact and progressive disclosure evals (task .6) - Expand negative controls and confusion matrix tests (task .7)
…tons - Create tests/evals/ directory with _common.py shared infrastructure module - Add config.yaml with per-eval-type regression thresholds, models, retry, cost - Add rubric_schema.yaml documenting rubric contract and validate_rubrics.py enforcer - Create skeleton runners: run_effectiveness.py, run_activation.py, run_size_impact.py, run_confusion_matrix.py - Add compare_baseline.py informational regression detection utility (always exit 0) - Create committed directories with .gitkeep: rubrics/, baselines/, datasets/activation/, datasets/confusion/, datasets/size_impact/ - Add .gitignore for runtime-only directories: results/, reports/ - Add requirements.txt with anthropic and pyyaml only Task: fn-58-offline-skill-evaluation-framework-from.1
- Replace regex-based JSON extraction with json.JSONDecoder.raw_decode scanning for robust nested-object and braces-in-strings handling - Add skill directory existence check in validate_rubrics.py - Guard load_config() against non-dict YAML with ValueError - Use encoding="utf-8" in write_results() file output Task: fn-58-offline-skill-evaluation-framework-from.1
- Remove unused math import from compare_baseline.py - Replace PEP 604 union types with Optional[] for Python 3.9 compat - Add OSError handling in validate_rubrics.py file reading Task: fn-58-offline-skill-evaluation-framework-from.1
Remove unused imports/variables and fix Optional[Path] type narrowing by using explicit resolved_dir: Path assignments. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ness evaluation - Add rubrics for 12 skills: xunit, minimal-apis, efcore-patterns, csharp-coding-standards, csharp-async-patterns, resilience, containers, blazor-patterns, testing-strategy, observability, security-owasp, native-aot - Each rubric has 5 weighted criteria (sum=1.0) with skill-specific descriptions referencing concrete APIs, patterns, and anti-patterns from each SKILL.md - Each rubric has 2 realistic developer test prompts for A/B effectiveness eval - All 12 rubrics pass validate_rubrics.py schema validation - Prioritization: user-invocable code-producing skills with clear A/B testability, high-overlap disambiguation (xunit vs testing-strategy), and specialized knowledge skills (native-aot, resilience, observability) Task: fn-58-offline-skill-evaluation-framework-from.2
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Plan-sync detected drift: compare_baseline.py uses underscore-delimited baseline filenames (effectiveness_baseline.json) but task .4 spec listed hyphenated names. Updated spec to match implementation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add judge_prompt.py with structured JSON judge prompts, progressive retry escalation on parse failure, and response validation - Flesh out run_effectiveness.py skeleton with full generation pipeline: enhanced (skill-injected) vs baseline conditions, seeded A/B randomization, cost tracking with abort limit, and resume/replay via cached generations in results/generations/ - Support --dry-run, --skill, --runs, --seed, --regenerate, --output-dir - Compute per-skill summary statistics (mean, stddev, n, win_rate) Task: fn-58-offline-skill-evaluation-framework-from.3
- Strengthen judge response validation: enforce exact criterion name matching, score range [1-5], no duplicates, require reasoning field - Remove bogus weight normalization in _compute_case_scores that could hide missing criteria - Wrap judge invocation in try/except to prevent run abort on API errors - Validate cached generation shape before use (treat invalid as cache miss) - Add entity_id field to all case records for envelope contract compliance - Add prompt injection defense: instruct judge to ignore instructions inside responses, wrap responses in XML delimiters - Derive case_seed from stable hash of (seed, skill, prompt, run) so A/B assignment is consistent regardless of iteration order - Remove unused Optional import from judge_prompt.py Task: fn-58-offline-skill-evaluation-framework-from.3
- Include model, temperature, and skill body hash in generation cache key to prevent stale reuse when parameters change - Guard _validate_judge_response against empty criteria list - Replace simple XML response delimiters with high-entropy sentinels to prevent delimiter collision from model outputs Task: fn-58-offline-skill-evaluation-framework-from.3
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Plan-sync detected drift: compare_baseline.py already fully implemented (task .4 updated to verify-only), judge_prompt.invoke_judge() API signature documented for task .6. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…sets - Build compressed routing index dynamically from skill frontmatter (id + description, sorted by id, truncated to 120 chars) - Structured JSON response detection via _common.extract_json() with LLM fallback only on parse failures - 55 positive activation cases across 55 unique skills, 18 negative controls - Metrics: TPR, FPR, accuracy, per-skill activation rate, token usage, index char_count, cost - Results include mean/stddev/n when --runs > 1 - --dry-run and --skill filter supported - Per-case details: prompt, expected, actual, detection_method, pass/fail - Reuses 26 prompts from existing copilot-smoke test infrastructure Task: fn-58-offline-skill-evaluation-framework-from.5
- Emit scalar tpr/fpr/accuracy/n fields in summary for compare_baseline.py compatibility alongside rich stats dicts - Keep detection_method as "parse_failure" on negative controls where fallback is not actually executed - Include fallback cost in per-case cost tracking so compute_metrics() reports accurate totals - Defensively normalize descriptions (type check, collapse newlines) - Remove unused run_results_per_case variable - Fallback now checks expected_skills + acceptable_skills Task: fn-58-offline-skill-evaluation-framework-from.5
- Initialize fallback_cost before branching to prevent UnboundLocalError on structured-success path - Return input/output tokens from detect_activation_fallback and accumulate in per-case token counts for accurate reporting - Mark negative-control parse failures as non-passing to prevent artificially low FPR from non-compliant model responses - Move per-skill activation rates from summary to artifacts to avoid polluting compare_baseline.py's entity iteration namespace - Remove unused fallback_input/fallback_output declarations Task: fn-58-offline-skill-evaluation-framework-from.5
- Base TP/FP/TN/FN in compute_metrics() on the passed field so negative parse failures and API errors are counted as FP, not TN - Normalize fallback JSON boolean parsing to handle string "false" instead of relying on bool() which treats non-empty strings as True - Sort fallback targets for deterministic iteration order - Add explicit encoding="utf-8" to dataset file reads Task: fn-58-offline-skill-evaluation-framework-from.5
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…osure validation - Implement full L6 runner (run_size_impact.py) with three conditions: Full (complete SKILL.md body), Summary (deterministic extraction), and Baseline (no skill content) - Add deterministic summary extraction algorithm: strip frontmatter, extract Scope section, strip code fences and cross-refs, concatenate - Add sibling file testing (fourth condition) with explicit allowlist and max_sibling_bytes cap in candidates.yaml - Create candidates.yaml with 10 skills spanning small/medium/large tiers, including 2 with sibling files - Support pairwise LLM judge comparisons via judge_prompt.invoke_judge() - Record exact bytes and token count per condition per case - Support --dry-run, --skill filter, --regenerate, resume/replay via generation caching Task: fn-58-offline-skill-evaluation-framework-from.6
- Fix load_siblings() byte accounting: use raw bytes for truncation to avoid splitting multi-byte UTF-8 and ensure accurate byte counts - Store all condition_sizes on every case record (not just compared pair) to satisfy "exact bytes/tokens per condition per case" acceptance - Split per-comparison cost into cost_judge and cost_generation_allocated to avoid double-counting shared generations across comparisons - Align tier thresholds with task .6 spec: <2KB/2-5KB/>5KB (was <5KB/5-12KB/>12KB) - Update candidates.yaml tier comments to match new thresholds - Validate candidate shape in load_candidates() with per-entry warnings - Add CRLF normalization in extract_summary() for cross-platform regex Task: fn-58-offline-skill-evaluation-framework-from.6
- Fix sibling byte accounting: load_siblings() now returns formatted byte count (including delimiters) for exact injection consistency - Derive full_siblings condition_sizes from exact injected string - Adjust tier thresholds to <5KB/5-15KB/>15KB to match actual repo data (no skills <2KB exist in the 131-skill catalog) - Add dotnet-csharp-code-smells as 3rd sibling-tested candidate (now 3 skills with siblings, 11 total candidates across 3 tiers) - Validate siblings list element types and max_sibling_bytes in load_candidates() with per-field type checks - Add CRLF normalization to classify_size_tier() matching extract_summary() pattern - Add standard run fields (model, judge_model, run_id, seed, condition_sizes, conditions_present) to error case records Task: fn-58-offline-skill-evaluation-framework-from.6
- Derive condition_sizes from exact injected blobs (including wrappers) for summary and full_siblings conditions - Rename token fields to tokens_estimated throughout results to clearly indicate approximate nature; update task spec acceptance accordingly - Update task spec tier thresholds to match implementation (<5KB/5-15KB/>15KB) with explanation of why original <2KB/2-5KB/>5KB was adjusted - Add path traversal rejection in load_candidates() validation and defense-in-depth check in load_siblings() - Use errors="replace" for non-truncated sibling UTF-8 decode - Add expected_tier advisory field to each candidate in candidates.yaml for machine-readable size metadata Task: fn-58-offline-skill-evaluation-framework-from.6
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Bounded sibling reads: read only needed bytes instead of full file - Symlink escape prevention in sibling loading - Indented code fence stripping in summary extraction - Cache schema version in generation hash keys - Remove unused local variables Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create confusion_matrix.jsonl with 36 prompts across 7 domain groups (testing, security, data, performance, api, cicd, blazor) - Create negative_controls_expanded.jsonl with 18 prompts (non-.NET + temptation prompts that overlap with disallowed skill domains) - Flesh out run_confusion_matrix.py skeleton into full L4 runner with group-scoped routing index, NxN confusion matrices, cross-activation flagging (>20%), low-discrimination detection, findings report, and per-group mean/stddev/n statistics - Support --dry-run, --group, --model, --runs, --seed, --output-dir CLI - Uses structured JSON response approach from task .5 Task: fn-58-offline-skill-evaluation-framework-from.7
- Lock matrix axes to declared DOMAIN_GROUPS skills (stable NxN across runs), track out-of-group predictions separately - Remove out-of-group acceptable_skills from confusion_matrix.jsonl (cm-security-003, cm-api-003, cm-api-004) - Add prompt-level low-discrimination findings for multi_activation cases, with case IDs and activated skill lists - Add cross-activation finding example_case_ids (top 3) - Validate --group name against DOMAIN_GROUPS, print clear error - Add skill existence validation (warn on missing skills in groups) - Remove unused total_cases variable in compute_cross_activation_rates Task: fn-58-offline-skill-evaluation-framework-from.7
- Make skill validation fail-fast: abort run with clear error if any DOMAIN_GROUPS skill lacks a description (missing from skills/) - Record skipped cases with classification=skipped_no_index instead of silently continuing, ensuring coverage gaps are visible - Include out_of_group_count in serialized matrices and add index_violation_rate metric to cross-activation rates - Add never-activated skill findings (skills never predicted in any case within their group) - Track out_of_group_count separately from cross_activation_rate Task: fn-58-offline-skill-evaluation-framework-from.7
- Add index_violation findings when model predicts skills outside the group-scoped routing index - Write invalid-run results envelope on missing-skills abort so scheduled automation sees explicit failure instead of stale results Task: fn-58-offline-skill-evaluation-framework-from.7
Pyright flagged full_skill_count as not accessed. Use _ discard pattern since only full_index_text is needed for negative control routing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace Python hash() with SHA-256 stable hash for deterministic cross-process sampling (hash() is randomized per PYTHONHASHSEED) - Use effective capped group count (not raw args.limit) for confusion negative-control proportional limiting - Support both --limit=N and --limit N forms in run_suite.sh Task: fn-60-run-eval-suite-against-skills-and-fix.1
…t arg - Set meta["aborted"] = True in confusion runner's missing-skills early-abort branch before writing result JSON - Add guard in run_suite.sh for dangling --limit with no value Task: fn-60-run-eval-suite-against-skills-and-fix.1
…ess tracking - Inventory 18 result files across 4 eval types, validate backends and case counts - Analyze L3 activation: TPR=69% raw (92.7% excl errors), 5 actual routing issues - Analyze L4 confusion: all 7 groups passing quality bar (60-100% accuracy) - Analyze L5 effectiveness: all 12 skills above 50% win rate on clean run - Analyze L6 size impact: 51.7% full win rate (below 55%), 1 baseline sweep (xunit) - Write triage.md with prioritized fix batches for tasks .3 (5 routing) and .4 (4 content) - Initialize eval-progress.json with per-dimension status tracking (routing_status + content_status) - 21 skill entries with eval type coverage, run IDs, and failure mode notes - Zero CLI calls used (reused existing valid post-.7 results) Task: fn-60-run-eval-suite-against-skills-and-fix.2
- Change results/ to results/* to allow per-file exceptions - Add !results/triage.md exception so triage report is tracked - Add the triage.md analysis document created by task .2 Task: fn-60-run-eval-suite-against-skills-and-fix.2
- Add P0 infra reliability section clarifying CLI timeout impact on metrics - Define quality bar gating policy: exclude detection_method:error from metrics - Normalize run-id format to prefixed form (activation_xxx, effectiveness_xxx) - Define L6 denominator precisely (full_vs_baseline comparisons, ties in denom) - Add verification command table for task .3 routing fixes - Add skill ID verification note (all 21 confirmed in skills/ directory) Task: fn-60-run-eval-suite-against-skills-and-fix.2
…essaging-patterns, architecture-patterns - dotnet-container-deployment: remove explicit Kubernetes keyword to reduce false positives on pure K8s prompts (neg-004) - dotnet-messaging-patterns: add .NET + MassTransit/Service Bus qualifiers to reduce false positives on generic messaging prompts (neg-018) - dotnet-architecture-patterns: add layered patterns keyword to fix false negative on architecture prompts (act-005 now passes) Batch 1 of routing fixes for fn-60.3. Budget: 11666 chars (was 11671, -5). Task: fn-60-run-eval-suite-against-skills-and-fix.3
…ilience - dotnet-system-commandline: lead with System.CommandLine 2.0, add RootCommand/Option<T>/tab completion keywords to fix false negative on CLI prompts (act-012 now passes) - dotnet-resilience: lead with Polly v8, add rate limiter/HttpClient keywords to fix false negative on resilience prompts (act-017 now passes) - Update eval-progress.json: mark all 5 routing-fix skills as fixed with targeted re-run evidence Batch 2 of routing fixes for fn-60.3. Budget: 11665 chars (was 11666, -1). Task: fn-60-run-eval-suite-against-skills-and-fix.3
Set fixed_by and fixed_at for dotnet-system-commandline and dotnet-resilience to reference the batch 2 commit (92fb8b8). Task: fn-60-run-eval-suite-against-skills-and-fix.3
Address review feedback: rephrase dotnet-resilience and dotnet-system-commandline descriptions to use third-person declarative verb-led style per routing style guide. - dotnet-resilience: "Configures Polly v8 resilience..." - dotnet-system-commandline: "Builds System.CommandLine 2.0..." Budget: 11673 chars (OK, under 12000). Task: fn-60-run-eval-suite-against-skills-and-fix.3
Address review feedback: - dotnet-container-deployment: update run_id to post-fix run (5fd7c96e) where neg-004 passes, replacing pre-fix run (1ef0e510) - dotnet-messaging-patterns: revert to needs-fix since neg-018 still activates this skill; further refine description to use verb-led style and emphasize Azure Service Bus/.NET specificity - Update notes with accurate verification evidence Task: fn-60-run-eval-suite-against-skills-and-fix.3
The _transitions contract states ".3 sets routing_status to fixed", but dotnet-messaging-patterns remains needs-fix (neg-018 still fails). Remove ".3" from fixed_tasks and reset fixed_by/fixed_at to null to avoid misleading downstream task .5 which uses fixed_tasks to decide what needs re-verification. Task: fn-60-run-eval-suite-against-skills-and-fix.3
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9843bddd4c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
tests/evals/run_confusion_matrix.py
Outdated
Comment on lines
922
to
923
| if classification in ("single_activation", "multi_activation"): | ||
| passed = activated_skills[0] in all_valid |
There was a problem hiding this comment.
The confusion eval is defined as a single-skill disambiguation task (Select ONLY the single most relevant skill), but this pass/fail logic treats multi_activation as a pass whenever the first returned skill is valid. In practice, responses like ["expected-skill", "other-skill"] will be counted as correct, which inflates group accuracy and masks the exact ambiguity this eval is supposed to detect. Multi-activation cases should be marked failed (or scored separately) rather than accepted based on index 0.
Useful? React with 👍 / 👎.
…valuation - Added task fn-62 to refactor `_execute_cli()` for improved subprocess timeout management using `subprocess.Popen` and configurable timeouts via `config.yaml`. - Introduced a timeout shim in `run_suite.sh` for per-runner wall-clock timeouts, ensuring suite progression even if a runner hangs. - Created task fn-63 to enhance evaluation coverage with multi-label scoring, auto-generating activation test cases for uncovered skills, and retiring the confusion matrix runner. - Expanded effectiveness rubrics and fixed routing issues across multiple batches, ensuring all skills are evaluated and documented. - Conducted full-coverage verification for all skills, saving baselines and ensuring compliance with quality metrics.
- dotnet-xunit: 17.5KB -> 8.7KB by removing verbose v2 compat comments, ILogger integration section, redundant code examples, and condensing analyzer rules. Retains core patterns (Fact/Theory, fixtures, key principles, agent gotchas). - dotnet-csharp-coding-standards: 12.2KB -> 5.7KB by removing verbose correct/avoid code blocks, Type Design section (duplicates dotnet-solid-principles), Knowledge Sources section, and condensing code style rules into concise bullet points. Retains naming table, critical style rules, CancellationToken, XML docs. Both skills had L6 size_impact failures where baseline (description-only) outperformed full body content. Reducing noise/signal ratio should let body add value over what the model already knows. Task: fn-60-run-eval-suite-against-skills-and-fix.4
- dotnet-windbg-debugging: 2.9KB -> 4.4KB by replacing generic quick-start file references with concrete diagnostic workflows (crash dump analysis, hang/deadlock, high CPU, memory pressure) including actual WinDbg/SOS commands and a structured report template. - dotnet-ado-patterns: 3.3KB -> 5.9KB by inlining key YAML pipeline patterns (step templates, extends templates, variable groups, conditional insertion, triggers) from examples.md. Body was previously just Agent Gotchas with a pointer to examples.md -- too little actionable content for the model to benefit from. Both skills had L6 size_impact failures where summary outperformed full body. Adding concrete technical content should improve body signal density. Task: fn-60-run-eval-suite-against-skills-and-fix.4
Update content_status to "fixed" for 4 skills addressed in task .4: - dotnet-xunit: trimmed 17.5KB -> 8.7KB (baseline sweep fix) - dotnet-csharp-coding-standards: trimmed 12.2KB -> 5.7KB (baseline wins fix) - dotnet-windbg-debugging: restructured 2.9KB -> 4.4KB (signal density) - dotnet-ado-patterns: restructured 3.3KB -> 5.9KB (signal density) Each skill has fixed_tasks=[".4"], fixed_by set to batch commit, content_status="fixed". Verification re-runs require non-nested Claude session (eval runners cannot invoke CLI from within Claude). Removed empty result file from failed nested-session eval attempt. Task: fn-60-run-eval-suite-against-skills-and-fix.4
- dotnet-xunit: restore minimal analyzer section (package + 3 key rules
+ editorconfig suppression) to match scope/description promise
- dotnet-ado-patterns: add pipeline decorators subsection to match scope
bullet that claims decorator coverage
- dotnet-windbg-debugging: add symbol preflight section (.symfix, .reload,
lm verification) before diagnostic workflows
- dotnet-csharp-coding-standards: restore routing-strength activation
guidance ("do not wait for explicit user wording")
- eval-progress.json: clarify all notes that verification re-runs are
deferred to task .5, remove bogus empty run_id
Task: fn-60-run-eval-suite-against-skills-and-fix.4
- Computed quality bar metrics from valid result runs (excluding error cases per policy) - L3 Activation: PASS (TPR=92.7%, FPR=13.3%, Accuracy=91.1% after error exclusion) - L4 Confusion: PASS (all 7 groups >=60% accuracy, cross-activation <=35%, neg controls 88.9%) - L5 Effectiveness: PASS (all skills >=50% win rate, micro-avg 87.5%) - L6 Size Impact: PENDING_REVERIFICATION (pre-fix baseline sweeps fixed in .4, external re-run needed) - Verified .3 routing fixes: architecture-patterns, system-commandline, resilience, container-deployment now verified - Documented messaging-patterns as variance exception (neg-018 deliberately ambiguous) - Created verify-content-fixes.sh for external .4 L5/L6 re-verification - Removed 7 invalid nested-session result files (0 structured detections from CLAUDECODE constraint) - Updated eval-progress.json with verified statuses and quality bar summary Task: fn-60-run-eval-suite-against-skills-and-fix.5
- Replace verification_run with explicit verification_runs map citing concrete run IDs (activation_3925edef, confusion_e3f1b006, etc.) - Reword ".5 sweep" language to cite specific run IDs and commits for each verified skill, avoiding over-claiming - Add attempted_tasks/attempted_commits provenance to messaging-patterns exception entry for auditability - Add CLAUDECODE env guard to verify-content-fixes.sh for fail-fast when run inside nested Claude Code session - Append validate-skills.sh + validate-marketplace.sh to verification script to cover full .5 acceptance checklist Task: fn-60-run-eval-suite-against-skills-and-fix.5
- Update _transitions to reflect actual .3/.4/.5 promotion semantics - Clarify L6 summary: 1 baseline sweep (xunit) + 1 zero-full-wins with tie (coding-standards), matching triage.md definitions Task: fn-60-run-eval-suite-against-skills-and-fix.5
- L5 summary notes pre-.4 status and pending re-verification for edited skills - Soften messaging-patterns provenance claim to match recorded run_ids - Rephrase .4 skill notes to avoid implying verified status - Clarify overall_status transition contract to match actual roll-up usage - Quote $0 in verify-content-fixes.sh for path safety Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…n re-verification - Change overall_status contract to worst-of across dimensions (not highest) - Fix dotnet-resilience overall_status from verified to passing (content is passing) - Clarify .5 spec: confusion re-verification only if skill is in confusion dataset Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Change 'variance exception' to 'routing exception' in .5 done summary - Add activation_1ef0e510 pre-fix run_id to container-deployment for audit trail Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The --mcp-config "{}" flag caused the claude CLI to return empty stdout
with exit code 0, because the {} JSON object lacks the expected
mcpServers key. --strict-mcp-config alone is sufficient to block MCP
server loading during eval runs.
Also adds _wrap_for_login_shell() to ensure CLI tools have access to
PATH and auth tokens when invoked from subprocess environments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Full-coverage baseline results for all 4 eval types, generated with backend=claude, model=haiku, judge=haiku, seed=42. Source result files: - activation_1a21d627-bbfd-445c-b2c6-3066bd8be7cd.json - confusion_80dd5aab-78a0-40bb-a0bd-85c7ba3eef88.json - effectiveness_f648b036-c39e-4e57-8c40-bc197438109f.json - size_impact_27906ad7-5a1c-47d7-b3c0-e6eaa0a8d5af.json Quality bars verified (all PASS): - L3 Activation: TPR=100%, FPR=16.67%, Accuracy=95.89% - L4 Confusion: 100% accuracy all 7 groups, 100% negative controls - L5 Effectiveness: all 12 skills >=83.3% win rate, 0% error rate - L6 Size Impact: no baseline sweeps, all skills n>=2 Coverage: full dataset (no --limit), 3 runs for effectiveness/size_impact. compare_baseline.py verified: loads all baselines, produces output. Unblocks fn-58.4 (CI regression gates). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ation mapping - Created task fn-64-consolidate-131-skills-into-20-broad.1 for a comprehensive skill audit and consolidation mapping. - Developed detailed approach for reading and consolidating 131 SKILL.md files into ~20 broad skills. - Established acceptance criteria to ensure all skills are mapped and documented. feat(consolidation): Delete eval harness and update CI gates - Added task fn-64-consolidate-131-skills-into-20-broad.10 to delete eval harness and update CI gates. - Updated CI validation scripts and remapped smoke tests to reflect new consolidated skill structure. - Regenerated baselines and updated documentation accordingly. feat(consolidation): Consolidate core language skills - Introduced task fn-64-consolidate-131-skills-into-20-broad.2 to consolidate core language skills into dotnet-csharp, dotnet-debugging, and dotnet-project-setup. - Created SKILL.md and references for each consolidated skill, ensuring all source directories are removed. feat(consolidation): Consolidate testing skills - Added task fn-64-consolidate-131-skills-into-20-broad.3 to create a consolidated dotnet-testing skill directory. - Merged all relevant testing-related skills and ensured framework-specific skills are handled separately. feat(consolidation): Consolidate API and data skills - Created task fn-64-consolidate-131-skills-into-20-broad.4 for merging API and EF Core skills into dotnet-api and dotnet-efcore. - Developed SKILL.md and references for each consolidated skill, removing old directories. feat(consolidation): Consolidate UI framework skills - Introduced task fn-64-consolidate-131-skills-into-20-broad.5 to consolidate UI framework skills into dotnet-blazor, dotnet-uno, dotnet-maui, and dotnet-desktop. - Ensured all relevant testing skills are absorbed into their respective frameworks. feat(consolidation): Consolidate DevOps skills - Added task fn-64-consolidate-131-skills-into-20-broad.6 to create consolidated skill directories for DevOps skills. - Merged relevant skills into dotnet-cicd-gha, dotnet-cicd-ado, dotnet-containers, and dotnet-packaging. feat(consolidation): Consolidate build and performance skills - Created task fn-64-consolidate-131-skills-into-20-broad.7 for merging build and performance skills into dotnet-performance, dotnet-aot, dotnet-build, and dotnet-cli-apps. feat(consolidation): Consolidate cross-cutting skills - Introduced task fn-64-consolidate-131-skills-into-20-broad.8 to handle cross-cutting skills and remaining unmapped skills. - Created SKILL.md and references for dotnet-security, dotnet-observability, and dotnet-docs. feat(consolidation): Update agents and advisor for consolidated skills - Added task fn-64-consolidate-131-skills-into-20-broad.9 to update all agent definitions and the dotnet-advisor routing catalog. - Ensured all references to old skill names are replaced with new consolidated names.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new epic, "Run Eval Suite Against Skills and Fix to Quality Bar," which closes the loop on the previously implemented offline evaluation framework by running the full eval suite against all skills, analyzing failures, making targeted fixes, and establishing baseline results for regression tracking. It also includes infrastructure improvements for authentication and skill restore safety, and updates dependencies and status for related epics.
Key changes include:
New Epic and Task Structure for Skill Evaluation and Improvement
.flow/.checkpoint-fn-60-run-eval-suite-against-skills-and-fix.jsoncontaining the full specification, quality bar thresholds, scope, approach, and a detailed breakdown of tasks for running evals, analyzing results, fixing skills, rerunning, and saving baselines. This epic ensures all skills are evaluated and improved to meet defined quality standards, with clear acceptance criteria and risk mitigation strategies.Infrastructure and Workflow Enhancements
get_client()to support SDK-based auth discovery (not just theANTHROPIC_API_KEYenv var), and documents a git-based skill restore mechanism to safely revert manual changes to skill files.Completion and Dependency Updates
.flow/epics/fn-36-library-api-compatibility-skills.json..flow/epics/fn-58-offline-skill-evaluation-framework-from.jsonfor the "Offline Skill Evaluation Framework from dotnet-skills-evals," marking it as "done" and updating its dependency to require the completion of the new evaluation-and-fix epic.These changes collectively enable a robust, iterative evaluation and improvement process for the skill catalog, ensuring ongoing quality and providing a foundation for future CI integration.