Skip to content

Commit 8f2b159

Browse files
KyleAMathewsclaude
andcommitted
add feedback collection skill and list output hint
New meta-skill guides agents to collect structured feedback after using SKILL.md files. The `list` command in both CLIs now shows a feedback hint pointing agents to the instructions and submit command. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 28483ea commit 8f2b159

File tree

4 files changed

+165
-0
lines changed

4 files changed

+165
-0
lines changed
Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
---
2+
name: skill-feedback-collection
3+
description: >
4+
Collect structured feedback about skill usage after completing a coding task.
5+
Activate at the end of any session where one or more SKILL.md files were
6+
loaded. Captures agent signals (gaps, errors, corrections, human interventions)
7+
and brief human input, then submits via the intent feedback CLI.
8+
metadata:
9+
version: '1.0'
10+
category: meta-tooling
11+
output_artifacts:
12+
- intent-feedback.json
13+
---
14+
15+
# Skill Feedback Collection
16+
17+
Run this at the end of any session where you loaded one or more SKILL.md files.
18+
The goal is to capture what worked, what didn't, and what was missing — so skill
19+
maintainers can improve future versions.
20+
21+
---
22+
23+
## Phase 1 — Automated Signal Collection
24+
25+
Review your own session transcript. No human interaction needed yet.
26+
27+
### 1a: Gap detection
28+
29+
Identify moments where the skill was silent and you had to bridge the gap
30+
yourself — via code reading, search, trial-and-error, or general knowledge.
31+
32+
For each gap, note:
33+
- What you needed to do
34+
- What the skill should have told you
35+
- How you solved it (code reading, web search, guessing)
36+
37+
### 1b: Error/correction tracking
38+
39+
Identify moments where the skill prescribed an approach that produced an error.
40+
41+
For each error, note:
42+
- What the skill said to do
43+
- The error or incorrect behavior that resulted
44+
- The fix you applied
45+
46+
### 1c: Human intervention events
47+
48+
Identify moments where the human clarified, corrected, or overrode your approach.
49+
50+
For each intervention, note:
51+
- What you were doing when the human intervened
52+
- What the human said or changed
53+
- Whether the skill could have prevented this
54+
55+
### 1d: Step duration anomalies
56+
57+
Identify steps that consumed disproportionate effort compared to their apparent
58+
complexity. These signal that the skill should provide a template, snippet, or
59+
more detailed guidance.
60+
61+
---
62+
63+
## Phase 2 — Human Interview
64+
65+
Ask the human up to 4 questions. Keep it brief — skip questions if the session
66+
already provided clear answers. Respect if they decline.
67+
68+
1. "Was anything unclear about what was happening during the task?"
69+
2. "Did anything feel frustrating or take longer than expected?"
70+
3. "Were you uncertain about the output quality at any point?"
71+
4. "Anything you'd want done differently next time?"
72+
73+
Derive `userRating` from overall sentiment:
74+
- Mostly positive → `good`
75+
- Mixed signals → `mixed`
76+
- Mostly negative → `bad`
77+
78+
If the human gives an explicit rating, use that instead.
79+
80+
---
81+
82+
## Phase 3 — Build the Payload
83+
84+
Construct one JSON payload per skill used. The schema must match exactly:
85+
86+
```json
87+
{
88+
"skill": "<skill name from SKILL.md frontmatter>",
89+
"package": "<npm package name that contains the skill>",
90+
"skillVersion": "<metadata.version from SKILL.md frontmatter, or library_version>",
91+
"task": "<one-sentence summary of what the human asked you to do>",
92+
"whatWorked": "<patterns/instructions from the skill that were accurate and helpful>",
93+
"whatFailed": "<errors from 1b — what the skill got wrong>",
94+
"missing": "<gaps from 1a — what the skill should have covered>",
95+
"selfCorrections": "<fixes you applied from 1b + interventions from 1c>",
96+
"userRating": "good | mixed | bad",
97+
"userComments": "<optional — direct quotes or paraphrased human input from Phase 2>"
98+
}
99+
```
100+
101+
### Field derivation guide
102+
103+
| Field | Source |
104+
|-------|--------|
105+
| `skill` | Frontmatter `name` field of the SKILL.md you loaded |
106+
| `package` | The npm package the skill lives in (e.g. `@tanstack/query-intent`) |
107+
| `skillVersion` | Frontmatter `metadata.version` or `library_version` |
108+
| `task` | Summarize the human's original request in one sentence |
109+
| `whatWorked` | List skill sections/patterns that were correct and useful |
110+
| `whatFailed` | From 1b — skill instructions that produced errors |
111+
| `missing` | From 1a — gaps where the skill was silent |
112+
| `selfCorrections` | From 1b fixes + 1c human interventions, combined |
113+
| `userRating` | From Phase 2 sentiment analysis or explicit rating |
114+
| `userComments` | From Phase 2 answers, keep brief |
115+
116+
### Example
117+
118+
```json
119+
{
120+
"skill": "tanstack-query/core",
121+
"package": "@anthropic/tanstack-query-intent",
122+
"skillVersion": "1.0",
123+
"task": "Add optimistic updates to a mutation with rollback on error",
124+
"whatWorked": "Setup pattern was correct. onMutate/onError/onSettled lifecycle was accurate.",
125+
"whatFailed": "Cache key format used array syntax that doesn't match v5 — had to switch to queryOptions pattern.",
126+
"missing": "No guidance on TypeScript generics for mutation variables. Had to read source.",
127+
"selfCorrections": "Fixed cache key format. Human corrected the rollback type to include undefined.",
128+
"userRating": "mixed",
129+
"userComments": "The optimistic update pattern was helpful but the cache key thing wasted 10 minutes."
130+
}
131+
```
132+
133+
If multiple skills were loaded, produce one payload per skill. Only include
134+
skills that were actually used during the session — skip any that were loaded
135+
but never referenced.
136+
137+
---
138+
139+
## Phase 4 — Submit
140+
141+
1. Save the payload to `intent-feedback.json` in the project root.
142+
If multiple skills, save as a JSON array.
143+
144+
2. Submit:
145+
```bash
146+
npx intent feedback --submit --file intent-feedback.json
147+
```
148+
149+
3. If the submission succeeds, delete `intent-feedback.json`.
150+
151+
4. If it fails, tell the human and leave the file for manual retry.

packages/intent/src/cli.ts

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,12 @@ async function cmdList(args: string[]): Promise<void> {
167167
console.log()
168168
}
169169

170+
console.log(`Feedback:`)
171+
console.log(` After completing your task, collect feedback on skill usage.`)
172+
console.log(` Instructions: node_modules/@tanstack/intent/meta/feedback-collection/SKILL.md`)
173+
console.log(` Submit: npx intent feedback --submit --file intent-feedback.json`)
174+
console.log()
175+
170176
if (result.warnings.length > 0) {
171177
console.log(`Warnings:`)
172178
for (const w of result.warnings) console.log(` ⚠ ${w}`)

packages/intent/src/init.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@ const INTENT_BLOCK = `## Intent Skills
2020
This project uses TanStack Intent. Run \`npx intent list\` to discover
2121
available AI coding skills. Before working with a library that has skills,
2222
read the relevant SKILL.md file at the path shown in the list output.
23+
After completing a task that used skills, collect and submit feedback
24+
following the instructions in the list output.
2325
`
2426

2527
const DEFAULT_CONFIG: IntentProjectConfig = {

packages/intent/src/intent-library.ts

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,12 @@ async function cmdList(): Promise<void> {
145145
console.log()
146146
}
147147

148+
console.log(`Feedback:`)
149+
console.log(` After completing your task, collect feedback on skill usage.`)
150+
console.log(` Instructions: node_modules/@tanstack/intent/meta/feedback-collection/SKILL.md`)
151+
console.log(` Submit: npx intent feedback --submit --file intent-feedback.json`)
152+
console.log()
153+
148154
if (result.warnings.length > 0) {
149155
console.log(`Warnings:`)
150156
for (const w of result.warnings) console.log(` ⚠ ${w}`)

0 commit comments

Comments
 (0)