Skip to content

Commit 273c802

Browse files
lutterlutter
committed
docs: Update gnd-cli-expansion spec, commit plan for it, too
1 parent 037af74 commit 273c802

File tree

2 files changed

+392
-28
lines changed

2 files changed

+392
-28
lines changed

‎docs/plans/gnd-cli-expansion.md‎

Lines changed: 346 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,346 @@
1+
# Plan: Extend gnd with graph-cli functionality
2+
3+
## Overview
4+
5+
Extend the existing `gnd` (Graph Node Dev) CLI to be a **drop-in replacement** for the TypeScript-based `graph-cli`. The goal is identical CLI behavior, flags, output format, and byte-for-byte identical AssemblyScript code generation.
6+
7+
**Spec document**: `docs/specs/gnd-cli-expansion.md`
8+
9+
**Scope**:
10+
11+
- Ethereum protocol only (other protocols are future work)
12+
- All graph-cli commands except `local` and `node` subcommand
13+
- Match latest graph-cli version (0.98.x)
14+
15+
**Current state of gnd** (`~/code/graph-node/gnd/`):
16+
17+
- ~700 lines of Rust across 3 files
18+
- Single command with flags (no subcommands)
19+
- Already uses clap, imports graph crates
20+
- Runs graph-node in dev mode with file watching
21+
22+
## Status Summary (Updated 2026-01-19)
23+
24+
**Overall Progress**: ~95% complete (implementation done, testing/docs remaining)
25+
26+
| Phase | Status | Notes |
27+
|-------|--------|-------|
28+
| Phase 1: CLI Restructure | ✅ Complete | |
29+
| Phase 2: Infrastructure | ✅ Complete | |
30+
| Phase 3: Simple Commands | ✅ Complete | |
31+
| Phase 4: Code Generation | ✅ Complete | 11 verification fixtures passing |
32+
| Phase 5: Migrations | ✅ Complete | |
33+
| Phase 6: Build Command | 🟡 Needs Testing | Manual end-to-end verification |
34+
| Phase 7: Deploy Command | 🟡 Needs Testing | Manual Graph Node test |
35+
| Phase 8: Init Command | 🟡 Needs Testing | Implemented, manual verification pending |
36+
| Phase 9: Add Command | ✅ Complete | |
37+
| Phase 10: Publish Command | 🟡 Needs Testing | Manual end-to-end verification |
38+
| Phase 11: Test Command | 🟡 Needs Testing | Manual Matchstick test |
39+
| Phase 12: Testing & Polish | 🟡 In Progress | Port TS CLI tests, write docs |
40+
41+
**Total LOC**: ~15,000 lines of Rust (158 unit tests, 12 verification tests passing)
42+
43+
## Git Workflow
44+
45+
**Branch**: All work should be committed to the `gnd-cli` branch.
46+
47+
**Commit discipline**:
48+
49+
- Commit work in small, reviewable chunks
50+
- Each commit should be self-contained and pass all checks
51+
- Prefer many small commits over few large ones
52+
- Each commit message should clearly describe what it does
53+
54+
**Before each commit**:
55+
56+
```bash
57+
just format
58+
just lint
59+
just test-unit # only run the tests in gnd/
60+
```
61+
62+
- MANDATORY: Work must be committed, a task is only done when work is committed
63+
- MANDATORY: Make sure to follow the commit discipline above
64+
65+
## Commands
66+
67+
| Command | Description | Complexity |
68+
| ------------- | ----------------------------------- | ----------------------- |
69+
| `gnd dev` | Existing gnd functionality | Done (restructure only) |
70+
| `gnd codegen` | Generate AssemblyScript types | High |
71+
| `gnd build` | Compile to WASM | Medium |
72+
| `gnd deploy` | Deploy to Graph Node | Medium |
73+
| `gnd init` | Scaffold new subgraph | High |
74+
| `gnd add` | Add datasource to existing subgraph | Medium |
75+
| `gnd create` | Register subgraph name | Low |
76+
| `gnd remove` | Unregister subgraph name | Low |
77+
| `gnd auth` | Set deploy key | Low |
78+
| `gnd publish` | Publish to decentralized network | Medium |
79+
| `gnd test` | Run Matchstick tests | Low (shell out) |
80+
| `gnd clean` | Remove build artifacts | Low |
81+
82+
**Not implemented** (documented differences):
83+
84+
- `gnd local` - Use existing test infrastructure
85+
- `gnd node` - Use graphman for node management
86+
- `--uncrashable` flag - Float Capital third-party feature
87+
88+
## Reusable from graph-node
89+
90+
| Feature | Location | Status |
91+
| ------------------- | ------------------------------------- | ---------------------------- |
92+
| Manifest parsing | `graph/src/data/subgraph/mod.rs` | Ready |
93+
| Manifest validation | `graph/src/data/subgraph/mod.rs` | Ready (may need refactoring) |
94+
| GraphQL schema | `graph/src/schema/input/` | Ready |
95+
| ABI parsing | `ethabi` + chain/ethereum wrappers | Ready |
96+
| IPFS client | `graph/src/ipfs/` | Ready |
97+
| Link resolution | `graph/src/components/link_resolver/` | Ready |
98+
| File watching | `gnd/src/watcher.rs` | Ready |
99+
| CLI framework | `clap` (already in gnd) | Ready |
100+
101+
## Module Structure
102+
103+
```
104+
gnd/src/
105+
├── main.rs # Entry point, clap setup
106+
├── lib.rs
107+
├── commands/
108+
│ ├── mod.rs
109+
│ ├── dev.rs # Existing gnd functionality
110+
│ ├── codegen.rs
111+
│ ├── build.rs
112+
│ ├── deploy.rs
113+
│ ├── init.rs
114+
│ ├── add.rs
115+
│ ├── create.rs
116+
│ ├── remove.rs
117+
│ ├── auth.rs
118+
│ ├── publish.rs
119+
│ ├── test.rs
120+
│ └── clean.rs
121+
├── codegen/
122+
│ ├── mod.rs
123+
│ ├── schema.rs # Entity class generation
124+
│ ├── abi.rs # ABI binding generation
125+
│ └── template.rs # Template binding generation
126+
├── scaffold/
127+
│ ├── mod.rs
128+
│ ├── manifest.rs # Generate subgraph.yaml
129+
│ ├── schema.rs # Generate schema.graphql
130+
│ └── mapping.rs # Generate mapping.ts
131+
├── migrations/
132+
│ ├── mod.rs
133+
│ └── ... # One module per migration version
134+
├── compiler/
135+
│ ├── mod.rs
136+
│ └── asc.rs # Shell out to asc
137+
├── services/
138+
│ ├── mod.rs
139+
│ ├── etherscan.rs # Etherscan API client
140+
│ ├── sourcify.rs # Sourcify API client
141+
│ ├── registry.rs # Network registry client
142+
│ └── graph_node.rs # Graph Node JSON-RPC client
143+
├── config/
144+
│ ├── mod.rs
145+
│ ├── auth.rs # ~/.graphprotocol/ management
146+
│ └── networks.rs # networks.json handling
147+
├── output/
148+
│ ├── mod.rs
149+
│ └── spinner.rs # Progress/spinner output (match TS CLI exactly)
150+
└── watcher.rs # Existing file watcher
151+
```
152+
153+
## Dependencies to Add
154+
155+
| Crate | Purpose |
156+
| ----------- | ------------------------------------------- |
157+
| `inquire` | Interactive prompts (init) |
158+
| `minijinja` | Template rendering (scaffold) |
159+
| `indicatif` | Progress bars and spinners |
160+
| `reqwest` | HTTP client (Etherscan, Sourcify, registry) |
161+
162+
## TODO List
163+
164+
### Phase 1: CLI Restructure
165+
166+
- [x] Create `gnd-cli` branch
167+
- [x] Restructure main.rs with clap subcommands
168+
- [x] Move existing gnd logic to `commands/dev.rs`
169+
- [x] Add empty command stubs for all commands
170+
- [x] Verify `gnd dev` works exactly as before
171+
- [x] Add `--version` output showing both gnd and graph-cli versions
172+
173+
### Phase 2: Infrastructure
174+
175+
- [x] Implement `output/spinner.rs` matching TS CLI output format exactly
176+
- [x] Implement `config/auth.rs` for `~/.graph-cli.json` management (in commands/auth.rs)
177+
- [x] Implement `config/networks.rs` for networks.json parsing
178+
- [x] Add reqwest dependency and basic HTTP client setup (in services/graph_node.rs)
179+
- [x] Implement `services/registry.rs` for @pinax/graph-networks-registry (implemented as `NetworksRegistry` in services/contract.rs)
180+
181+
### Phase 3: Simple Commands
182+
183+
- [x] Implement `gnd clean` command
184+
- [x] Implement `gnd auth` command
185+
- [x] Implement `gnd create` command
186+
- [x] Implement `gnd remove` command
187+
- [x] Add tests for simple commands
188+
189+
### Phase 4: Code Generation (High Priority, High Risk)
190+
191+
- [x] Study TS CLI codegen output in detail
192+
- [x] Implement `codegen/schema.rs` - entity classes from GraphQL
193+
- [x] Implement `codegen/typescript.rs` - AST builders for TypeScript/AssemblyScript
194+
- [x] Implement `codegen/types.rs` - type conversion utilities
195+
- [x] Implement `codegen/abi.rs` - ABI bindings
196+
- [x] Implement `codegen/template.rs` - template bindings
197+
- [x] Integrate prettier formatting (shell out)
198+
- [x] Implement `commands/codegen.rs` with all flags
199+
- [x] Implement `--watch` mode using existing file watcher
200+
- [x] Create snapshot tests comparing output to TS CLI (11 fixtures from graph-cli validation tests)
201+
- [x] Verify byte-for-byte identical output - tests pass with documented known differences:
202+
- Int8 import always included (gnd simplicity)
203+
- Trailing commas in multi-line constructs (gnd style)
204+
- 2D array accessors use correct `toStringMatrix()` (gnd fixes graph-cli bug)
205+
206+
### Phase 5: Migrations
207+
208+
- [x] Study TS CLI migrations in `/packages/cli/src/migrations/`
209+
- [x] Implement migration framework in `migrations/mod.rs`
210+
- [x] Implement each migration version (0.0.1 → current)
211+
- [x] Add `--skip-migrations` flag support
212+
- [x] Test migrations with old manifest versions
213+
214+
### Phase 6: Build Command
215+
216+
- [x] Implement `compiler/asc.rs` - shell out to asc
217+
- [x] Implement `commands/build.rs` with all flags (~1084 lines, full build pipeline)
218+
- [x] Handle network file resolution
219+
- [x] Implement `--watch` mode
220+
- [x] Implement optional IPFS upload with deduplication
221+
- [ ] Test build output matches TS CLI (NEEDS HUMAN: manual end-to-end verification)
222+
223+
### Phase 7: Deploy Command
224+
225+
- [x] Implement `services/graph_node.rs` - JSON-RPC client (~258 lines, create/remove/deploy)
226+
- [x] Implement `commands/deploy.rs` with all flags (~239 lines)
227+
- [x] Support all deploy targets (local, studio, hosted service) - defaults to Subgraph Studio
228+
- [x] Handle access token / deploy key authentication
229+
- [ ] Test deployment to local Graph Node (NEEDS HUMAN: run services + manual test)
230+
231+
### Phase 8: Init Command
232+
233+
- [x] Add `inquire` dependency for interactive prompts
234+
- [x] Implement `services/contract.rs` - ABI fetching (Etherscan, Blockscout, Sourcify) (~730 lines)
235+
- [x] Implement networks registry loading
236+
- [x] Implement `scaffold/manifest.rs` (~271 lines)
237+
- [x] Implement `scaffold/schema.rs` (~206 lines)
238+
- [x] Implement `scaffold/mapping.rs` (~205 lines)
239+
- [x] Implement `commands/init.rs` with all flags (~1060 lines including --from-subgraph)
240+
- [x] Implement --from-example mode
241+
- [x] Implement --from-contract mode (uses ContractService for ABI fetching)
242+
- [x] Add interactive prompts when required options are missing
243+
- [x] Implement --from-subgraph mode (fetches manifest from IPFS, extracts immutable entities)
244+
- [ ] Test scaffold output matches TS CLI (NEEDS HUMAN: manual end-to-end verification)
245+
246+
### Phase 9: Add Command
247+
248+
- [x] Implement `commands/add.rs` (~773 lines, fully functional)
249+
- [x] Reuse scaffold components for actual implementation
250+
- [x] ABI validation, entity generation, mapping creation, manifest updates
251+
- [x] Test adding datasource to existing subgraph (unit tests for helper functions)
252+
- [x] Comprehensive error handling
253+
254+
### Phase 10: Publish Command
255+
256+
- [x] Implement `commands/publish.rs` (~230 lines, fully functional)
257+
- [x] Build subgraph and upload to IPFS (reuses build command)
258+
- [x] Open browser to webapp URL with query params (same approach as graph-cli)
259+
- [x] Support --ipfs-hash flag to skip build
260+
- [x] Support --subgraph-id, --api-key for updating existing subgraphs
261+
- [x] Support --protocol-network (arbitrum-one, arbitrum-sepolia)
262+
- [x] Interactive prompt before opening browser
263+
- [ ] Test publish workflow (NEEDS HUMAN: manual end-to-end verification)
264+
265+
### Phase 11: Test Command
266+
267+
- [x] Implement Matchstick binary download/detection
268+
- [x] Implement `commands/test.rs` (~254 lines, shell out to Matchstick)
269+
- [x] Handle Docker mode (Dockerfile generation on demand)
270+
- [x] Recompilation flag support
271+
- [ ] Test with actual Matchstick tests (NEEDS HUMAN: install matchstick + run end-to-end)
272+
273+
### Phase 12: Testing & Polish
274+
275+
- [ ] Port all tests from TS CLI test suite not already covered by gnd tests
276+
- Location: `/home/lutter/code/subgraphs/graph-cli/packages/cli/tests/`
277+
- Review existing coverage and identify gaps
278+
- [x] Add snapshot tests for code generation scenarios (11 fixtures from graph-cli validation tests)
279+
- [x] Add tests for overloaded events/functions in ABI codegen (disambiguation)
280+
- [x] Add tests for simple array fields in schema codegen
281+
- [x] Add tests for nested array types (`[[String]]` etc.) - schema codegen refactored to track list depth
282+
- [x] Add tests for tuple/struct types in ABI codegen (functions, events, nested, arrays)
283+
- [x] Add tests for array types in ABI codegen (array params in events, 2D matrices)
284+
- [x] Codegen verification test framework (gnd/tests/codegen_verification.rs)
285+
- [x] Add comprehensive tests for prompt module (network completer, source type)
286+
- [x] Add comprehensive tests for init command (interactive mode detection)
287+
- [ ] Ensure all edge cases are covered (ongoing)
288+
- [ ] Documentation:
289+
- [ ] CLI usage docs (README with command reference, examples, common workflows)
290+
- [ ] Migration guide from graph-cli to gnd (differences, compatibility notes)
291+
- [x] Shell completions (bash, elvish, fish, powershell, zsh via clap_complete)
292+
293+
## Key Decisions
294+
295+
| Decision | Choice | Rationale |
296+
| ---------------- | ---------------------------- | ------------------------------------------- |
297+
| Binary name | `gnd` | Existing name, avoid confusion with `graph` |
298+
| Protocol support | Ethereum only | Simplify initial scope |
299+
| Compatibility | Drop-in replacement | Same flags, output, exit codes |
300+
| Code generation | Byte-for-byte identical | Snapshot testable, no surprises |
301+
| Formatting | Shell out to prettier | Guarantees identical output |
302+
| Compilation | Shell out to asc | Same as TS CLI |
303+
| Testing | Shell out to Matchstick | Same as TS CLI |
304+
| Debug logging | RUST_LOG | Rust standard, not DEBUG env var |
305+
| Validation | Use graph-node's | Refactor if needed, single source of truth |
306+
| Network registry | Fetch at runtime | Registry contents change over time |
307+
| Error messages | Same info, format may differ | Convey same information |
308+
| TS CLI bugs | Fix them | Don't replicate bugs |
309+
310+
## Risk Assessment
311+
312+
| Risk | Severity | Mitigation |
313+
| --------------------------- | -------- | ------------------------------------ |
314+
| Code generation fidelity | High | Snapshot tests against TS CLI output |
315+
| Migration correctness | High | Test with real old manifests |
316+
| Breaking existing gnd users | Medium | Keep `gnd dev` behavior identical |
317+
| `asc` version compat | Medium | Version detection, clear errors |
318+
| Network API changes | Low | Error handling, clear messages |
319+
320+
## TS CLI References
321+
322+
Key files to study in `/home/lutter/code/subgraphs/graph-cli/packages/cli/src/`:
323+
324+
| Feature | TS CLI Location |
325+
| ---------------- | ----------------------------------- |
326+
| Commands | `commands/*.ts` |
327+
| Type generator | `type-generator.ts` |
328+
| Schema codegen | `codegen/schema.ts` |
329+
| ABI codegen | `protocols/ethereum/codegen/abi.ts` |
330+
| Template codegen | `codegen/template.ts` |
331+
| Compiler | `compiler/index.ts` |
332+
| Migrations | `migrations/` |
333+
| Scaffold | `scaffold/` |
334+
| Spinner/output | `command-helpers/spinner.ts` |
335+
| Auth | `command-helpers/auth.ts` |
336+
| Network config | `command-helpers/network.ts` |
337+
| Tests | `../tests/cli/` |
338+
339+
## Effort Estimate
340+
341+
**~3-4 person-months** for full implementation because:
342+
343+
- Code generation must be byte-for-byte identical (requires careful study)
344+
- All migrations must be implemented
345+
- Full test coverage needed
346+
- But we reuse: manifest parsing, validation, IPFS client, file watching, CLI framework

0 commit comments

Comments
 (0)