Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
name: CI

on:
push:
branches: [main]
pull_request:
branches: [main]

jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
cache: npm
- run: npm ci
- run: npm run lint

test-unit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
cache: npm
- run: npm ci
- run: npm test

test-docker:
runs-on: ubuntu-latest
needs: [lint, test-unit]
strategy:
fail-fast: false
matrix:
runtime: [node, bun, deno]
include:
- runtime: node
integration_cmd: npx vitest run test/integration
- runtime: bun
integration_cmd: bunx vitest run test/integration
- runtime: deno
integration_cmd: deno run -A npm:vitest run test/integration
steps:
- uses: actions/checkout@v4
- name: Build test-${{ matrix.runtime }}
run: docker compose build test-${{ matrix.runtime }}
- name: Unit tests (${{ matrix.runtime }})
run: docker compose run --rm test-${{ matrix.runtime }}
- name: Integration tests (${{ matrix.runtime }})
run: docker compose run --rm test-${{ matrix.runtime }} ${{ matrix.integration_cmd }}
38 changes: 36 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,48 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]
## [Unreleased] — M2 Boomerang (v1.2.0)

### Added
- None.
- `CasService.restore()` — reconstruct files from manifests with per-chunk SHA-256 integrity verification.
- `ContentAddressableStore.restoreFile()` — facade method that restores and writes to disk.
- `readTree()` on `GitPersistencePort` / `GitPersistenceAdapter` — parse Git trees via `ls-tree`.
- `STREAM_ERROR` wrapping — stream failures during `store()` surface as `CasError('STREAM_ERROR')` with `{ chunksWritten }` metadata.
- `MISSING_KEY` error code — `restore()` now fails fast when manifest is encrypted but no decryption key is provided.
- CLI: `git cas store`, `git cas tree`, `git cas restore` subcommands via `bin/git-cas.js`.
- Integration test suite (59 tests) running against real Git bare repos inside Docker.
- `commander` dependency for CLI.

### Changed
- `readBlob()` now normalises `Uint8Array` from plumbing into `Buffer` for codec/crypto compatibility.
- `readTree()` uses `git ls-tree -z` (NUL-delimited output) for safe parsing of filenames with leading/trailing spaces.

### Fixed
- Fuzz tests in stream-error suite now fail explicitly if `store()` does not throw.
- ROADMAP: resolved inconsistent CLI signatures for `git cas tree` (`--slug` vs `--manifest`).

### Security
- None.

## [1.1.0] — M1 Bedrock

### Added
- `CryptoPort` interface and `NodeCryptoAdapter` — extracted all `node:crypto` usage from the domain layer.
- `CasService.store()` — accepts `AsyncIterable<Buffer>` sources (renamed from `storeFile`).
- Multi-stage Dockerfile (Node 22, Bun, Deno) with `docker-compose.yml` for per-runtime testing.
- BATS parallel test runner (`test/platform/runtimes.bats`).
- Devcontainer setup (`.devcontainer/`) with all three runtimes + BATS.
- Encryption key validation (`INVALID_KEY_TYPE`, `INVALID_KEY_LENGTH` error codes).
- Encryption round-trip unit tests (110 tests including fuzz).
- Empty file (0-byte) edge case tests.
- Error-path unit tests for constructors and core failures.
- Deterministic test digest helper (`digestOf`).

### Changed
- `CasService` domain layer has zero `node:*` imports — all platform dependencies injected via ports.
- Constructor requires `crypto` and `codec` params (no defaults); facade supplies them.
- Facade `storeFile()` now opens the file and delegates to `CasService.store()`.

### Fixed
- None.

Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ CMD ["bunx", "vitest", "run", "test/unit"]
# --- Deno ---
FROM denoland/deno:latest AS deno
USER root
RUN apt-get update && apt-get install -y git && rm -rf /var/lib/apt/lists/*
RUN apt-get update && apt-get install -y git nodejs npm && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY . .
RUN deno install --allow-scripts
Expand Down
24 changes: 19 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ We use the object database.
- **Optional AES-256-GCM encryption** store secrets without leaking plaintext into the ODB.
- **Manifests** a tiny explicit index of chunks + metadata (JSON/CBOR).
- **Tree output** generates standard Git trees so assets snap into commits cleanly.
- **Full round-trip** store, tree, and restore — get your bytes back, verified.

**Use it for:** binary assets, build artifacts, model weights, data packs, secret bundles, weird experiments, etc.

Expand All @@ -43,19 +44,32 @@ const manifest = await cas.storeFile({
});

// Turn the manifest into a Git tree OID
const treeOid = cas.createTree({ manifest });
const treeOid = await cas.createTree({ manifest });

// Now you can point a ref/commit at that tree like a normal Git artifact.
// Restore later — get your bytes back, integrity-verified
await cas.restoreFile({ manifest, outputPath: './restored.png' });
```

## CLI (git plugin)

`git-cas` installs as a Git subcommand:

```bash
git cas store ./image.png --slug my-image
git cas tree --slug my-image
git cas restore <tree-oid> --out ./image.png
# Store a file — prints manifest JSON
git cas store ./image.png --slug my-image

# Store and get a tree OID directly
git cas store ./image.png --slug my-image --tree

# Create a tree from an existing manifest
git cas tree --manifest manifest.json

# Restore from a tree OID
git cas restore <tree-oid> --out ./restored.png

# Encrypted round-trip (32-byte raw key file)
git cas store ./secret.bin --slug vault --key-file ./my.key --tree
git cas restore <tree-oid> --out ./decrypted.bin --key-file ./my.key
```

## Why not Git LFS?
Expand Down
152 changes: 147 additions & 5 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ This roadmap is structured as:
3. **Contracts** — Return/throw semantics for all public methods
4. **Version Plan** — Table mapping versions to milestones
5. **Milestone Dependency Graph** — ASCII diagram
6. **Milestones & Task Cards** — 7 milestones, 24 tasks (uniform task card template)
6. **Milestones & Task Cards** — 7 milestones, 26 tasks (uniform task card template)

---

Expand Down Expand Up @@ -42,6 +42,7 @@ Single registry of all error codes used across the codebase. Each code is a stri
| `INVALID_KEY_TYPE` | Encryption key is not a Buffer. | Task 1.3 |
| `INTEGRITY_ERROR` | Decryption auth-tag verification failed (wrong key, tampered ciphertext, or tampered tag), or chunk digest mismatch on restore. | Exists (decrypt); extended by Task 1.6, Task 2.1 |
| `STREAM_ERROR` | Read stream failed during `storeFile`. Partial chunks may have been written to Git ODB (unreachable; handled by `git gc`). Meta includes `{ chunksWritten: <number> }`. | Task 2.4 |
| `MISSING_KEY` | Encryption key required to restore encrypted content but none was provided. | Task 2.1 |
| `TREE_PARSE_ERROR` | `git ls-tree` output could not be parsed into valid entries. | Task 2.2 |
| `MANIFEST_NOT_FOUND` | No manifest entry (e.g. `manifest.json` / `manifest.cbor`) found in the Git tree. | Task 4.1 |
| `GIT_ERROR` | Underlying Git plumbing command failed. Wraps the original error from the plumbing layer. | Task 2.2, Task 4.1 |
Expand Down Expand Up @@ -106,14 +107,29 @@ Return and throw semantics for every public method (current and planned).
- **Algorithms:** `pbkdf2` (default), `scrypt` — both Node.js built-ins.
- **Throws:** Standard Node.js crypto errors on invalid parameters.

### CLI: `git cas store <file> --slug <slug> [--key-file <path>]` *(planned — Task 2.5)*
- **Output:** Prints manifest JSON to stdout. If `--tree` is passed, prints only the Git tree OID instead.
- **Exit 0:** Store succeeded.
- **Exit 1:** Store failed (error message to stderr).

### CLI: `git cas tree --manifest <path>` *(planned — Task 2.5)*
- **Output:** Prints Git tree OID to stdout.
- **Exit 0:** Tree created.
- **Exit 1:** Invalid manifest or Git error (message to stderr).

### CLI: `git cas restore <tree-oid> --out <path> [--key-file <path>]` *(planned — Task 2.6)*
- **Output:** Writes restored file to `--out` path.
- **Exit 0:** Restore succeeded, prints bytes written to stdout.
- **Exit 1:** Integrity error, missing manifest, or I/O error (message to stderr).

---

## 4) Version Plan

| Version | Milestone | Codename | Theme |
|--------:|-----------|----------|-------|
| v1.1.0 | M1 | Bedrock | Foundation hardening |
| v1.2.0 | M2 | Boomerang| File retrieval round trip |
| v1.2.0 | M2 | Boomerang| File retrieval round trip + CLI |
| v1.3.0 | M3 | Launchpad| CI/CD pipeline |
| v1.4.0 | M4 | Compass | Lifecycle management |
| v1.5.0 | M5 | Sonar | Observability |
Expand Down Expand Up @@ -152,13 +168,13 @@ M3 Launchpad (v1.3.0) M4 Compass (v1.4.0)
| # | Codename | Theme | Version | Tasks | ~LoC | ~Hours |
|---:|--------------|----------------------------|:-------:|------:|-------:|------:|
| M1 | Bedrock | Foundation hardening | v1.1.0 | 7 | ~475 | ~6.5h |
| M2 | Boomerang | File retrieval round trip | v1.2.0 | 4 | ~295 | ~9.5h |
| M2 | Boomerang | File retrieval round trip + CLI | v1.2.0 | 6 | ~435 | ~14h |
| M3 | Launchpad | CI/CD pipeline | v1.3.0 | 2 | ~110 | ~4h |
| M4 | Compass | Lifecycle management | v1.4.0 | 3 | ~180 | ~5.5h |
| M5 | Sonar | Observability | v1.5.0 | 2 | ~210 | ~5.5h |
| M6 | Cartographer | Documentation | v1.6.0 | 3 | ~750 | ~10h |
| M7 | Horizon | Advanced features | v2.0.0 | 3 | ~450 | ~17h |
| | **Total** | | | **24**| **~2,470** | **~58h** |
| | **Total** | | | **26**| **~2,610** | **~62.5h** |

---

Expand Down Expand Up @@ -535,7 +551,7 @@ As a maintainer, I want error conditions covered by tests so regressions in vali
---

# M2 — Boomerang (v1.2.0)
**Theme:** Complete store→retrieve round trip.
**Theme:** Complete store→retrieve round trip + CLI.

---

Expand Down Expand Up @@ -761,6 +777,132 @@ As a developer, I want storeFile to fail safely on stream errors so partial stor

---

## Task 2.5: CLI scaffold + `store` and `tree` subcommands

**User Story**
As a developer, I want `git cas store` and `git cas tree` commands so I can use CAS from the terminal without writing Node scripts.

**Requirements**
- R1: Add `bin/git-cas.js` entry point (Git discovers `git-cas` on PATH for `git cas` subcommands).
- R2: Add `"bin": { "git-cas": "./bin/git-cas.js" }` to `package.json`.
- R3: Use a lightweight CLI framework (e.g., `commander`) for subcommand routing.
- R4: `git cas store <file> --slug <slug> [--key-file <path>] [--tree]`:
- Reads the file, calls `storeFile()`.
- Prints manifest JSON to stdout by default.
- If `--tree` is passed, also calls `createTree()` and prints tree OID.
- `--key-file` reads a 32-byte raw key from a file for encryption.
- R5: `git cas tree --manifest <path>`:
- Reads a manifest JSON from file/stdin, calls `createTree()`.
- Prints tree OID to stdout.
- R6: Exit 0 on success, exit 1 on error with message to stderr.
- R7: `--cwd` flag to set Git working directory (defaults to `.`).

**Acceptance Criteria**
- AC1: `npx git-cas store ./test.txt --slug test` prints manifest JSON.
- AC2: `npx git-cas store ./test.txt --slug test --tree` prints tree OID.
- AC3: `npx git-cas tree --manifest manifest.json` prints tree OID.
- AC4: Invalid arguments produce helpful usage message and exit 1.
- AC5: `--key-file` with valid 32-byte file encrypts successfully.
- AC6: `--key-file` with wrong-size file exits 1 with clear error.

**Scope**
- In scope: CLI scaffold, store subcommand, tree subcommand, key-file reading.
- Out of scope: `restore` subcommand (Task 2.6), shell completions, config files.

**Est. Complexity (LoC)**
- Prod: ~80
- Tests: ~30
- Total: ~110

**Est. Human Working Hours**
- ~3h

**Test Plan**
- Golden path:
- store a file via CLI → valid manifest JSON on stdout.
- store with `--tree` → tree OID on stdout.
- tree from manifest file → tree OID on stdout.
- Failures:
- missing file → exit 1 with error.
- missing `--slug` → exit 1 with usage message.
- bad key file → exit 1 with INVALID_KEY_LENGTH/TYPE error.
- Edges:
- 0-byte file store.
- manifest piped via stdin (if supported).
- Fuzz/stress:
- None (thin wrapper over tested API).

**Definition of Done**
- DoD1: `bin/git-cas.js` exists with store and tree subcommands.
- DoD2: `package.json` declares bin entry.
- DoD3: `npx git-cas --help` prints usage.
- DoD4: Integration smoke test passes against real Git repo.

**Blocking**
- Blocks: Task 2.6

**Blocked By**
- Blocked by: None

---

## Task 2.6: CLI `restore` subcommand

**User Story**
As a developer, I want `git cas restore <tree-oid> --out <path>` so I can retrieve stored assets from the terminal.

**Requirements**
- R1: `git cas restore <tree-oid> --out <path> [--key-file <path>]`:
- Reads the tree, extracts the manifest, restores the file to `--out`.
- Prints bytes written to stdout on success.
- `--key-file` supplies decryption key for encrypted assets.
- R2: Exit 0 on success, exit 1 on error (INTEGRITY_ERROR, MANIFEST_NOT_FOUND, etc.) with message to stderr.
- R3: Requires `restoreFile()` (Task 2.1) and `readManifest()` or equivalent tree-reading capability.

**Acceptance Criteria**
- AC1: `npx git-cas restore <oid> --out ./restored.txt` writes correct file.
- AC2: Encrypted asset with `--key-file` restores correctly.
- AC3: Wrong key exits 1 with INTEGRITY_ERROR message.
- AC4: Invalid tree OID exits 1 with clear error.

**Scope**
- In scope: restore subcommand wired to restoreFile API.
- Out of scope: Streaming output to stdout, partial restore, resume.

**Est. Complexity (LoC)**
- Prod: ~30
- Tests: ~20
- Total: ~50

**Est. Human Working Hours**
- ~1.5h

**Test Plan**
- Golden path:
- store → tree → restore → byte-compare original.
- encrypted store → tree → restore with key → byte-compare.
- Failures:
- wrong key → exit 1 INTEGRITY_ERROR.
- nonexistent tree OID → exit 1.
- missing `--out` → exit 1 with usage.
- Edges:
- 0-byte file round-trip via CLI.
- Fuzz/stress:
- None (thin wrapper over tested API).

**Definition of Done**
- DoD1: `restore` subcommand added to `bin/git-cas.js`.
- DoD2: Full CLI round-trip (store → tree → restore) documented and tested.
- DoD3: README CLI section is now accurate and deliverable.

**Blocking**
- Blocks: None

**Blocked By**
- Blocked by: Task 2.1, Task 2.5

---

# M3 — Launchpad (v1.3.0)
**Theme:** Automated quality gates and release process.

Expand Down
Loading