|
| 1 | +--- |
| 2 | +name: skill-feedback-collection |
| 3 | +description: > |
| 4 | + Collect structured feedback about skill usage after completing a coding task. |
| 5 | + Activate at the end of any session where one or more SKILL.md files were |
| 6 | + loaded. Captures agent signals (gaps, errors, corrections, human interventions) |
| 7 | + and brief human input, then submits via the intent feedback CLI. |
| 8 | +metadata: |
| 9 | + version: '1.0' |
| 10 | + category: meta-tooling |
| 11 | + output_artifacts: |
| 12 | + - intent-feedback.json |
| 13 | +--- |
| 14 | + |
| 15 | +# Skill Feedback Collection |
| 16 | + |
| 17 | +Run this at the end of any session where you loaded one or more SKILL.md files. |
| 18 | +The goal is to capture what worked, what didn't, and what was missing — so skill |
| 19 | +maintainers can improve future versions. |
| 20 | + |
| 21 | +--- |
| 22 | + |
| 23 | +## Phase 1 — Automated Signal Collection |
| 24 | + |
| 25 | +Review your own session transcript. No human interaction needed yet. |
| 26 | + |
| 27 | +### 1a: Gap detection |
| 28 | + |
| 29 | +Identify moments where the skill was silent and you had to bridge the gap |
| 30 | +yourself — via code reading, search, trial-and-error, or general knowledge. |
| 31 | + |
| 32 | +For each gap, note: |
| 33 | +- What you needed to do |
| 34 | +- What the skill should have told you |
| 35 | +- How you solved it (code reading, web search, guessing) |
| 36 | + |
| 37 | +### 1b: Error/correction tracking |
| 38 | + |
| 39 | +Identify moments where the skill prescribed an approach that produced an error. |
| 40 | + |
| 41 | +For each error, note: |
| 42 | +- What the skill said to do |
| 43 | +- The error or incorrect behavior that resulted |
| 44 | +- The fix you applied |
| 45 | + |
| 46 | +### 1c: Human intervention events |
| 47 | + |
| 48 | +Identify moments where the human clarified, corrected, or overrode your approach. |
| 49 | + |
| 50 | +For each intervention, note: |
| 51 | +- What you were doing when the human intervened |
| 52 | +- What the human said or changed |
| 53 | +- Whether the skill could have prevented this |
| 54 | + |
| 55 | +### 1d: Step duration anomalies |
| 56 | + |
| 57 | +Identify steps that consumed disproportionate effort compared to their apparent |
| 58 | +complexity. These signal that the skill should provide a template, snippet, or |
| 59 | +more detailed guidance. |
| 60 | + |
| 61 | +--- |
| 62 | + |
| 63 | +## Phase 2 — Human Interview |
| 64 | + |
| 65 | +Ask the human up to 4 questions. Keep it brief — skip questions if the session |
| 66 | +already provided clear answers. Respect if they decline. |
| 67 | + |
| 68 | +1. "Was anything unclear about what was happening during the task?" |
| 69 | +2. "Did anything feel frustrating or take longer than expected?" |
| 70 | +3. "Were you uncertain about the output quality at any point?" |
| 71 | +4. "Anything you'd want done differently next time?" |
| 72 | + |
| 73 | +Derive `userRating` from overall sentiment: |
| 74 | +- Mostly positive → `good` |
| 75 | +- Mixed signals → `mixed` |
| 76 | +- Mostly negative → `bad` |
| 77 | + |
| 78 | +If the human gives an explicit rating, use that instead. |
| 79 | + |
| 80 | +--- |
| 81 | + |
| 82 | +## Phase 3 — Build the Payload |
| 83 | + |
| 84 | +Construct one JSON payload per skill used. The schema must match exactly: |
| 85 | + |
| 86 | +```json |
| 87 | +{ |
| 88 | + "skill": "<skill name from SKILL.md frontmatter>", |
| 89 | + "package": "<npm package name that contains the skill>", |
| 90 | + "skillVersion": "<metadata.version from SKILL.md frontmatter, or library_version>", |
| 91 | + "task": "<one-sentence summary of what the human asked you to do>", |
| 92 | + "whatWorked": "<patterns/instructions from the skill that were accurate and helpful>", |
| 93 | + "whatFailed": "<errors from 1b — what the skill got wrong>", |
| 94 | + "missing": "<gaps from 1a — what the skill should have covered>", |
| 95 | + "selfCorrections": "<fixes you applied from 1b + interventions from 1c>", |
| 96 | + "userRating": "good | mixed | bad", |
| 97 | + "userComments": "<optional — direct quotes or paraphrased human input from Phase 2>" |
| 98 | +} |
| 99 | +``` |
| 100 | + |
| 101 | +### Field derivation guide |
| 102 | + |
| 103 | +| Field | Source | |
| 104 | +|-------|--------| |
| 105 | +| `skill` | Frontmatter `name` field of the SKILL.md you loaded | |
| 106 | +| `package` | The npm package the skill lives in (e.g. `@tanstack/query-intent`) | |
| 107 | +| `skillVersion` | Frontmatter `metadata.version` or `library_version` | |
| 108 | +| `task` | Summarize the human's original request in one sentence | |
| 109 | +| `whatWorked` | List skill sections/patterns that were correct and useful | |
| 110 | +| `whatFailed` | From 1b — skill instructions that produced errors | |
| 111 | +| `missing` | From 1a — gaps where the skill was silent | |
| 112 | +| `selfCorrections` | From 1b fixes + 1c human interventions, combined | |
| 113 | +| `userRating` | From Phase 2 sentiment analysis or explicit rating | |
| 114 | +| `userComments` | From Phase 2 answers, keep brief | |
| 115 | + |
| 116 | +### Example |
| 117 | + |
| 118 | +```json |
| 119 | +{ |
| 120 | + "skill": "tanstack-query/core", |
| 121 | + "package": "@anthropic/tanstack-query-intent", |
| 122 | + "skillVersion": "1.0", |
| 123 | + "task": "Add optimistic updates to a mutation with rollback on error", |
| 124 | + "whatWorked": "Setup pattern was correct. onMutate/onError/onSettled lifecycle was accurate.", |
| 125 | + "whatFailed": "Cache key format used array syntax that doesn't match v5 — had to switch to queryOptions pattern.", |
| 126 | + "missing": "No guidance on TypeScript generics for mutation variables. Had to read source.", |
| 127 | + "selfCorrections": "Fixed cache key format. Human corrected the rollback type to include undefined.", |
| 128 | + "userRating": "mixed", |
| 129 | + "userComments": "The optimistic update pattern was helpful but the cache key thing wasted 10 minutes." |
| 130 | +} |
| 131 | +``` |
| 132 | + |
| 133 | +If multiple skills were loaded, produce one payload per skill. Only include |
| 134 | +skills that were actually used during the session — skip any that were loaded |
| 135 | +but never referenced. |
| 136 | + |
| 137 | +--- |
| 138 | + |
| 139 | +## Phase 4 — Submit |
| 140 | + |
| 141 | +1. Save the payload to `intent-feedback.json` in the project root. |
| 142 | + If multiple skills, save as a JSON array. |
| 143 | + |
| 144 | +2. Submit: |
| 145 | + ```bash |
| 146 | + npx intent feedback --submit --file intent-feedback.json |
| 147 | + ``` |
| 148 | + |
| 149 | +3. If the submission succeeds, delete `intent-feedback.json`. |
| 150 | + |
| 151 | +4. If it fails, tell the human and leave the file for manual retry. |
0 commit comments