Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
4487423
feat: RECALL — persistent seek materialization cache (v10.4.0)
flyingrobots Feb 10, 2026
fe30e51
fix: address PR review feedback for RECALL seek cache
flyingrobots Feb 10, 2026
332beb3
fix: default crypto adapter, join() .elements bug & coverage tests (v…
flyingrobots Feb 10, 2026
f51700c
docs: fix ARCHITECTURE.md markdown formatting
flyingrobots Feb 10, 2026
4ee8ea9
fix: accept ECONNRESET in 413 body-limit test
flyingrobots Feb 10, 2026
5758970
fix: move @git-stunts/git-cas to optionalDependencies
flyingrobots Feb 10, 2026
dfd9933
fix: setSeekCache TypeScript signature accepts null
flyingrobots Feb 10, 2026
c407a1f
fix: self-heal corrupted seek cache entries on deserialize failure
flyingrobots Feb 10, 2026
63675a7
fix: true LRU eviction in CasSeekCacheAdapter
flyingrobots Feb 10, 2026
8946fea
chore: zero TypeScript errors in source files (Stage B)
flyingrobots Feb 10, 2026
269d789
fix: suppress ESLint warnings for ignored scripts in pre-commit hook
flyingrobots Feb 10, 2026
06a4505
perf: optimize pre-push hook — single Docker build, parallel lint+typ…
flyingrobots Feb 10, 2026
a29f112
chore: zero TypeScript errors in test files (Stage C)
flyingrobots Feb 10, 2026
2820067
chore: TS policy enforcement — ban @ts-ignore, tag wildcard casts (B3…
flyingrobots Feb 11, 2026
d8bb054
fix: add type stub for optional @git-stunts/git-cas dependency
flyingrobots Feb 11, 2026
be9a00a
fix: require Node 22, fire-and-forget seek cache, fast pre-push
flyingrobots Feb 11, 2026
44b02d0
perf: remove Docker from pre-push hook — run locally only
flyingrobots Feb 11, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,16 @@ jobs:
cache: 'npm'
- run: npm install
- run: npm run lint
- name: TypeScript
run: npm run typecheck
- name: TS policy
run: npm run typecheck:policy

test-node:
runs-on: ubuntu-latest
strategy:
matrix:
node: [20, 22]
node: [22]
steps:
- uses: actions/checkout@v4
- name: Run unit + integration tests
Expand Down
10 changes: 8 additions & 2 deletions .github/workflows/release-pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@ jobs:
steps:
- uses: actions/checkout@v4

- name: Setup Node 20
- name: Setup Node 22
uses: actions/setup-node@v4
with:
node-version: "20"
node-version: "22"
cache: npm

- name: Install
Expand All @@ -26,6 +26,12 @@ jobs:
- name: Lint
run: npm run lint --if-present

- name: TypeScript
run: npm run typecheck

- name: TS policy
run: npm run typecheck:policy

- name: Test
run: npm run test:local --if-present

Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,10 @@ jobs:
with:
ref: ${{ github.event.workflow_run.head_branch }}

- name: Setup Node 20
- name: Setup Node 22
uses: actions/setup-node@v4
with:
node-version: "20"
node-version: "22"
cache: npm

- name: Install
Expand Down
40 changes: 34 additions & 6 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
WarpGraph is a graph database built on Git. It uses a patch-based CRDT model where Git commits represent patch objects containing graph operations, with commit messages encoding patch metadata and parent relationships linking patch history.

This architecture enables:

- Content-addressable storage with built-in deduplication
- Git's proven durability and integrity guarantees
- Standard Git tooling compatibility
Expand All @@ -21,13 +22,15 @@ The codebase follows hexagonal architecture to isolate domain logic from infrast
- **Domain services** contain pure business logic with injected dependencies

This enables:

- Easy testing via mock adapters
- Swappable infrastructure (different Git implementations, logging backends)
- Clear separation of concerns

### Domain-Driven Design

The domain layer models the graph database concepts:

- `GraphNode` - Immutable value object representing a node
- `WarpGraph` - Node CRUD operations (the main API class)
- `TraversalService` - Graph algorithms (BFS, DFS, shortest path)
Expand All @@ -40,6 +43,7 @@ The domain layer models the graph database concepts:
### Dependency Injection

All services accept their dependencies via constructor options:

- Persistence adapters
- Loggers
- Clocks
Expand All @@ -50,10 +54,10 @@ This enables testing with mocks and flexible runtime configuration.
## Layer Diagram

```text
+-------------------------------------------------------------+
+--------------------------------------------------------------+
| WarpGraph | <- Main API
| (WarpGraph.js) |
+-------------------------------------------------------------+
+--------------------------------------------------------------+
| Supporting Services |
| +---------------+ +--------------------+ |
| | IndexRebuild | | TraversalService | |
Expand All @@ -63,11 +67,11 @@ This enables testing with mocks and flexible runtime configuration.
| | HealthCheck | | BitmapIndex | | BitmapIndex | |
| | Service | | Builder | | Reader | |
| +-------------+ +---------------+ +--------------------+ |
| +---------------+ +---------------+ +--------------------+ |
| +---------------+ +--------------------+ |
| | GitLogParser | | Streaming | |
| | | | BitmapIndexBuilder | |
| +---------------+ +--------------------+ |
+-------------------------------------------------------------+
+--------------------------------------------------------------+
| Ports |
| +-------------------+ +---------------------------+ |
| | GraphPersistence | | IndexStoragePort | |
Expand All @@ -76,7 +80,7 @@ This enables testing with mocks and flexible runtime configuration.
| +-------------------+ +---------------------------+ |
| | LoggerPort | | ClockPort | |
| +-------------------+ +---------------------------+ |
+-------------------------------------------------------------+
+--------------------------------------------------------------+
| Adapters |
| +-------------------+ +---------------------------+ |
| | GitGraphAdapter | | ConsoleLogger | |
Expand All @@ -86,7 +90,7 @@ This enables testing with mocks and flexible runtime configuration.
| | PerformanceClock | |
| | GlobalClock | |
| +-------------------+ |
+-------------------------------------------------------------+
+--------------------------------------------------------------+
```

## Directory Structure
Expand Down Expand Up @@ -137,6 +141,7 @@ src/
### Main API: WarpGraph

The main entry point (`WarpGraph.js`) provides:

- Direct graph database API
- `open()` factory for managed mode with automatic durability
- Batch API for efficient bulk writes
Expand All @@ -148,6 +153,7 @@ The main entry point (`WarpGraph.js`) provides:
#### WarpGraph

Core node operations:

- `createNode()` - Create a single node
- `createNodes()` - Bulk creation with placeholder references (`$0`, `$1`)
- `readNode()` / `getNode()` - Retrieve node data
Expand All @@ -160,6 +166,7 @@ Message validation enforces size limits (default 1MB) and non-empty content.
#### IndexRebuildService

Orchestrates index creation:

- **In-memory mode**: Fast, O(N) memory, single serialization pass
- **Streaming mode**: Memory-bounded, flushes to storage periodically

Expand All @@ -168,6 +175,7 @@ Supports cancellation via `AbortSignal` and progress callbacks.
#### TraversalService

Graph algorithms using O(1) bitmap lookups:

- `bfs()` / `dfs()` - Traversal generators
- `ancestors()` / `descendants()` - Transitive closures
- `findPath()` - Any path between nodes
Expand All @@ -179,6 +187,7 @@ Graph algorithms using O(1) bitmap lookups:
- `commonAncestors()` - Find shared ancestors of multiple nodes

All traversals support:

- `maxNodes` / `maxDepth` limits
- Cancellation via `AbortSignal`
- Direction control (forward/reverse)
Expand All @@ -188,11 +197,13 @@ All traversals support:
Roaring bitmap-based indexes for O(1) neighbor lookups:

**Builder**:

- `registerNode()` - Assign numeric ID to SHA
- `addEdge()` - Record parent/child relationship
- `serialize()` - Output sharded JSON structure

**Reader**:

- `setup()` - Configure with shard OID mappings
- `getParents()` / `getChildren()` - O(1) lookups
- Lazy loading with LRU cache for bounded memory
Expand All @@ -201,6 +212,7 @@ Roaring bitmap-based indexes for O(1) neighbor lookups:
#### StreamingBitmapIndexBuilder

Memory-bounded variant of BitmapIndexBuilder:

- Flushes bitmap data to storage when threshold exceeded
- SHA-to-ID mappings remain in memory (required for consistency)
- Merges chunks at finalization via bitmap OR operations
Expand All @@ -210,6 +222,7 @@ Memory-bounded variant of BitmapIndexBuilder:
#### GraphPersistencePort

Git operations contract:

- `commitNode()` - Create commit pointing to empty tree
- `showNode()` / `getNodeInfo()` - Retrieve commit data
- `logNodesStream()` - Stream commit history
Expand All @@ -223,19 +236,22 @@ Also includes blob/tree operations for index storage.
#### IndexStoragePort

Index persistence contract:

- `writeBlob()` / `readBlob()` - Blob I/O
- `writeTree()` / `readTreeOids()` - Tree I/O
- `updateRef()` / `readRef()` - Index ref management

#### LoggerPort

Structured logging contract:

- `debug()`, `info()`, `warn()`, `error()` - Log levels
- `child()` - Create scoped logger with inherited context

#### ClockPort

Timing abstraction:

- `now()` - High-resolution timestamp (ms)
- `timestamp()` - ISO 8601 wall-clock time

Expand All @@ -244,6 +260,7 @@ Timing abstraction:
#### GitGraphAdapter

Implements both `GraphPersistencePort` and `IndexStoragePort`:

- Uses `@git-stunts/plumbing` for git command execution
- Retry logic with exponential backoff for transient errors
- Input validation to prevent command injection
Expand Down Expand Up @@ -302,11 +319,13 @@ SHA: 4b825dc642cb6eb9a060e54bf8d69288fbee4904
This is the well-known SHA of an empty Git tree, automatically available in every repository.

**How it works:**

- **Data**: Stored in commit message (arbitrary payload up to 1MB default)
- **Edges**: Commit parent relationships (directed, multi-parent supported)
- **Identity**: Commit SHA (content-addressable)

**Benefits:**

- Introduces no files into the repository working tree
- Content-addressable with automatic deduplication
- Git's proven durability and integrity (SHA verification)
Expand All @@ -331,6 +350,7 @@ index-tree/
```

**Shard envelope format:**

```json
{
"version": 2,
Expand All @@ -340,6 +360,7 @@ index-tree/
```

**Meta shard content:**

```json
{
"00a1b2c3d4e5f6789...": 0,
Expand All @@ -348,6 +369,7 @@ index-tree/
```

**Edge shard content:**

```json
{
"00a1b2c3d4e5f6789...": "OjAAAAEAAAAAAAEAEAAAABAAAA=="
Expand All @@ -369,12 +391,14 @@ Git garbage collection (GC) prunes commits that are not reachable from any ref.
### Modes

#### Managed Mode (Default)

In managed mode, WarpGraph guarantees durability for all writes.
- Every write operation updates the graph ref (or creates an anchor commit).
- Reachability from the ref is maintained automatically.
- Users do not need to manage refs or call sync manually.

#### Manual Mode

In manual mode, WarpGraph provides no automatic ref management.
- Writes create commits but do not update refs.
- User is responsible for calling `sync()` to persist reachability.
Expand All @@ -386,6 +410,7 @@ In manual mode, WarpGraph provides no automatic ref management.
Anchor commits are the mechanism used to maintain reachability for disconnected graphs (e.g., disjoint roots or imported history).

#### The Problem

In a linear history, every new commit points to the previous tip, maintaining a single chain reachable from the ref. However, graph operations can create disconnected commits:
- Creating a new root node (no parents).
- Merging unrelated graph histories.
Expand All @@ -394,6 +419,7 @@ In a linear history, every new commit points to the previous tip, maintaining a
If the ref simply moves to the new commit, the old history becomes unreachable and will be GC'd.

#### The Solution

An anchor commit is a special infrastructure commit that:
- Has multiple **parents**: The previous ref tip AND the new commit(s).
- Has an **Empty Tree** (like all WarpGraph nodes).
Expand Down Expand Up @@ -444,6 +470,7 @@ However, anchors do impact **Materialization** (scanning Git history to build st
### Sync Algorithm (V7)

In V7 Multi-Writer mode:

1. Each writer maintains their own ref (`refs/.../writers/<id>`), pointing to a chain of **Patch Commits**.
2. **Durability** is ensured because writes update these refs.
3. **Global Reachability** (optional) is maintained via `syncCoverage()`, which creates an **Octopus Anchor** commit pointed to by `refs/.../coverage/head`. This anchor has all writer tips as parents, ensuring they aren't GC'd even if individual writer refs are deleted (e.g. during a clone).
Expand Down Expand Up @@ -495,6 +522,7 @@ for await (const node of graph.iterateNodes({
```

Supported operations:

- `iterateNodes()`
- `rebuildIndex()`
- All traversal methods (BFS, DFS, shortest path, etc.)
56 changes: 56 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,62 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [10.4.2] — 2026-02-10 — TS policy enforcement (B3)

### Added

- **`scripts/ts-policy-check.js`**: Standalone policy checker that walks `src/`, `bin/`, `scripts/` and enforces two rules: (1) no `@ts-ignore` — use `@ts-expect-error` instead, (2) every inline `@type {*}` / `@type {any}` cast must carry a `// TODO(ts-cleanup): reason` tag.
- **`typecheck:policy` npm script**: Runs the policy checker (`node scripts/ts-policy-check.js`).
- **CI enforcement**: Policy check step added to both `ci.yml` (lint job) and `release-pr.yml` (preflight job), after the existing TypeScript step.
- **Pre-push hook**: Policy check runs in parallel with lint and typecheck.
- **BATS test timing**: `STARTING TEST` / `ENDED TEST` instrumentation in BATS helpers for diagnosing slow tests.

### Changed

- **Node.js >= 22.0.0**: Minimum engine bumped from 20 to 22, matching `@git-stunts/git-cas` requirement. CI matrix, release workflows, and documentation updated accordingly.
- **`@git-stunts/git-cas`**: Moved from `optionalDependencies` to `dependencies` now that Node 22 is the minimum.
- **Seek cache write is fire-and-forget**: `WarpGraph.materialize({ ceiling })` no longer awaits the persistent cache write — the CLI exits immediately after emitting output instead of blocking on background I/O (~30s → <1s for seek commands).
- **CLI uses `process.exit()`**: Ensures the process terminates promptly after emitting output, preventing fire-and-forget I/O from holding the event loop open.
- **Pre-push hook**: Removed BATS E2E tests (now CI-only) to keep pre-push fast.
- **`@ts-ignore` → `@ts-expect-error`** across 3 source files and 4 test files. `@ts-expect-error` is strictly better: it errors when the suppression becomes unnecessary.
- **~108 wildcard casts tagged** with `// TODO(ts-cleanup): reason` across ~30 source files in `src/`, `bin/`, and `scripts/`. Categorized reasons: `needs options type`, `type error`, `narrow port type`, `type patch array`, `type CLI payload`, `type http callback`, `type sync protocol`, `type lazy singleton`, `type observer cast`, and others.
- **`TYPESCRIPT_ZERO.md`**: B3 (Policy enforcement) marked complete.

## [10.4.1] — 2026-02-10 — Default crypto & join() fix

### Added

- **`defaultCrypto.js`** (`src/domain/utils/defaultCrypto.js`): Domain-local default crypto adapter wrapping `node:crypto` directly, following the `defaultCodec.js` / `defaultClock.js` pattern. Completes the BULKHEAD port injection pattern — all ports now have domain-local defaults.

### Fixed

- **`WarpGraph.join()`**: Replaced 4 references to non-existent `.elements.size` on ORSet with `orsetElements(...).length`. The `join()` happy path was always throwing a TypeError.

### Changed

- **`WarpGraph` constructor**: `this._crypto` now falls back to `defaultCrypto` when no crypto adapter is injected (same pattern as `this._codec = codec || defaultCodec`).
- **`BitmapIndexBuilder`**, **`StreamingBitmapIndexBuilder`**, **`BitmapIndexReader`**: Removed `if (!crypto) { return null; }` null guards from `computeChecksum`. Checksums are now always computed.
- **`BitmapIndexReader._validateShard`**: Removed `actualChecksum !== null &&` guard — checksum validation now always runs.
- **`StateSerializerV5.computeStateHashV5`**: Removed `crypto ? ... : null` ternary — always returns a hash string.

## [10.4.0] — 2026-02-09 — RECALL: Seek Materialization Cache

Caches materialized `WarpStateV5` at each visited ceiling tick as content-addressed blobs via `@git-stunts/git-cas`, enabling near-instant restoration for previously-visited ticks during seek exploration. Blobs are loose Git objects subject to Git GC (default prune expiry ~2 weeks, configurable) unless pinned to a vault.

### Added

- **`SeekCachePort`** (`src/ports/SeekCachePort.js`): Abstract port for seek materialization cache with `get`, `set`, `has`, `keys`, `delete`, `clear` methods.
- **`CasSeekCacheAdapter`** (`src/infrastructure/adapters/CasSeekCacheAdapter.js`): Git-CAS backed adapter with rich index metadata (treeOid, createdAt, ceiling, frontierHash, sizeBytes, codec, schemaVersion), LRU eviction (default max 200 entries), self-healing on read miss (removes dead entries when blobs are GC'd), and retry loop for transient write failures. **Requires Node >= 22.0.0** (inherited from `@git-stunts/git-cas`).
- **`seekCacheKey`** (`src/domain/utils/seekCacheKey.js`): Deterministic cache key builder producing `v1:t<ceiling>-<sha256hex>` keys. Uses SHA-256 via `node:crypto` with no fallback.
- **`buildSeekCacheRef`** in `RefLayout.js`: Builds `refs/warp/<graph>/seek-cache` ref path for the cache index.
- **`WarpGraph.open({ seekCache })`** / **`graph.setSeekCache(cache)`**: Optional `SeekCachePort` for persistent seek cache injection. Cache is checked after in-memory miss and stored after full materialization in `_materializeWithCeiling`.
- **`--clear-cache` flag** on `git warp seek`: Purges the persistent seek cache.
- **`--no-persistent-cache` flag** on `git warp seek`: Bypasses persistent cache for a single invocation (useful for full provenance access or performance testing).
- **Provenance degradation guardrails**: `_provenanceDegraded` flag on WarpGraph, set on persistent cache hit. `patchesFor()` and `materializeSlice()` throw `E_PROVENANCE_DEGRADED` with clear instructions to re-seek with `--no-persistent-cache`.
- **`SeekCachePort` export** from main entry point (`index.js`) and TypeScript definitions (`index.d.ts`).
- **Unit tests** (`test/unit/domain/seekCache.test.js`, 16 tests): Cache key determinism, WarpGraph integration with mock cache (hit/miss/error/degradation), provenance guardrails.
- **ROADMAP milestone RECALL** (v10.4.0): 6 tasks, all closed.

## [10.3.2] — 2026-02-09 — Seek CLI fixes & demo portability

### Added
Expand Down
3 changes: 1 addition & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,9 @@ npm test -- <pattern> # Run specific tests

# Multi-runtime test matrix (Docker)
npm run test:node22 # Node 22: unit + integration + BATS CLI
npm run test:node20 # Node 20: unit + integration + BATS CLI
npm run test:bun # Bun: API integration tests
npm run test:deno # Deno: API integration tests
npm run test:matrix # All four runtimes in parallel
npm run test:matrix # All runtimes in parallel
```

### No-Coordination Invariant
Expand Down
Loading