Skip to content

Conversation

@corylanou
Copy link
Collaborator

Summary

  • Add new directory replication guide at content/guides/directory/
  • Add directory configuration section to config reference
  • Add Configuration guides section to guides index (includes File and Directory guides)

Documents the directory replication feature from benbjohnson/litestream#738, enabling replication of entire directories of SQLite databases for multi-tenant applications.

Closes #95

Test plan

  • Markdown linting passes
  • Links resolve correctly in local dev server
  • Content matches existing style/voice

@YourTechBud
Copy link

Hi @corylanou, any particular reason that this doesn't auto discover new sqlite databases created after litestream has started?

@corylanou
Copy link
Collaborator Author

Great question! The current implementation (from PR #738) was designed as a simpler first step that scans directories at startup only.

Dynamic discovery is already implemented in PR #827, which adds a watch: true config option using fsnotify. With that feature, you'd configure it like:

dbs:
  - dir: /var/lib/app/tenants
    pattern: "*.db"
    recursive: true
    watch: true          # enables real-time discovery
    replica:
      type: s3
      bucket: my-backup-bucket

Once PR #827 is merged, the docs will be updated to include the dynamic watching capability.

@YourTechBud
Copy link

YourTechBud commented Dec 11, 2025

This is just incredible. Thanks for the explanation. I appreciate it.

Add documentation for the directory replication feature that allows
replicating entire directories of SQLite databases for multi-tenant
applications.

- Add new guide at content/guides/directory/
- Add directory configuration section to config reference
- Add Configuration guides section to guides index

Closes #95
@corylanou corylanou force-pushed the docs/95-directory-replication branch from 79981ce to e7e78ab Compare December 13, 2025 14:45
- Fix YAML field name: `directory` → `dir` (matches Litestream config)
- Add `{{< since version="0.5.0" >}}` annotation to directory guide
- Document `watch` option with link to Directory Watcher guide
- Fix `pattern` as required field (not optional with default)
- Update "Discovery timing" to reference watch option
- Add "Next steps" section linking to Directory Watcher
- Add cross-reference from Directory Watcher to Directory guide

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@corylanou
Copy link
Collaborator Author

Documentation Review & Testing Report

This PR was reviewed using automated testing against the Litestream binary to verify all configuration examples and commands work correctly.

Testing Environment

  • Litestream: Built from main branch (commit 8561d2a)
  • Test directory: /tmp/litestream-review/

Critical Issue Found & Fixed

Issue: All YAML config examples used directory: but Litestream expects dir:

🔴 Test with original directory: field (FAILED)
dbs:
  - directory: /var/lib/tenants
    pattern: "*.db"
    recursive: true
    replica:
      type: s3
      bucket: backups
      path: tenants

Command:

litestream replicate -config /tmp/litestream-review/config-directory.yml

Output:

2025/12/13 08:49:19 ERROR failed to run error="database config #1: must specify either 'path' or 'dir'"
🟢 Test with corrected dir: field (PASSED)
dbs:
  - dir: /tmp/litestream-review/tenants
    pattern: "*.db"
    recursive: true
    replica:
      type: file
      path: /tmp/litestream-review/backups

Command:

litestream replicate -config /tmp/litestream-review/config-dir.yml

Output:

time=2025-12-13T08:49:37.284-06:00 level=INFO msg=litestream version="(development build)"
time=2025-12-13T08:49:37.285-06:00 level=INFO msg="found databases in directory" dir=/tmp/litestream-review/tenants count=1 watch=false
time=2025-12-13T08:49:37.286-06:00 level=INFO msg="initialized db" path=/tmp/litestream-review/tenants/test.db
time=2025-12-13T08:49:37.286-06:00 level=INFO msg="replicating to" type=file sync-interval=1s path=/tmp/litestream-review/backups/test.db

Recursive Directory Structure Test

Verified that the directory structure is preserved in replica paths as documented.

🟢 Recursive replication test (PASSED)

Setup:

mkdir -p /tmp/litestream-review/customers/enterprise
mkdir -p /tmp/litestream-review/customers/standard
sqlite3 /tmp/litestream-review/customers/enterprise/acme.db "PRAGMA journal_mode=WAL; CREATE TABLE t(id INTEGER);"
sqlite3 /tmp/litestream-review/customers/standard/initech.db "PRAGMA journal_mode=WAL; CREATE TABLE t(id INTEGER);"

Config:

dbs:
  - dir: /tmp/litestream-review/customers
    pattern: "*.db"
    recursive: true
    replica:
      type: file
      path: /tmp/litestream-review/backups-recursive

Output:

time=2025-12-13T08:50:13.673-06:00 level=INFO msg="found databases in directory" dir=/tmp/litestream-review/customers count=2 watch=false
time=2025-12-13T08:50:13.673-06:00 level=INFO msg="replicating to" type=file path=/tmp/litestream-review/backups-recursive/enterprise/acme.db
time=2025-12-13T08:50:13.673-06:00 level=INFO msg="replicating to" type=file path=/tmp/litestream-review/backups-recursive/standard/initech.db

Resulting backup structure:

/tmp/litestream-review/backups-recursive/
├── enterprise/
│   └── acme.db/ltx/...
└── standard/
    └── initech.db/ltx/...

Restore Command Test

🟢 Restore from subdirectory path (PASSED)

Command:

litestream restore -o /tmp/litestream-review/acme-restored.db \
  file:///tmp/litestream-review/backups-recursive/enterprise/acme.db

Verification:

sqlite3 /tmp/litestream-review/acme-restored.db ".tables"
# Output: _litestream_lock  _litestream_seq   t

Source Code Verification

Confirmed the correct field name from Litestream source (cmd/litestream/main.go):

type DBConfig struct {
    Path      string `yaml:"path"`
    Dir       string `yaml:"dir"`       // Directory to scan for databases
    Pattern   string `yaml:"pattern"`   // File pattern to match
    Recursive bool   `yaml:"recursive"` // Scan subdirectories recursively
    Watch     bool   `yaml:"watch"`     // Enable directory monitoring
    // ...
}

Fixes Applied

  1. ✅ Changed directory:dir: in all 11 YAML examples
  2. ✅ Added {{< since version="0.5.0" >}} version annotation
  3. ✅ Documented watch option with link to Directory Watcher guide
  4. ✅ Fixed pattern as required (not optional with default)
  5. ✅ Updated "Discovery timing" consideration to mention watch: true
  6. ✅ Added "Next steps" section linking to Directory Watcher guide
  7. ✅ Added cross-reference from Directory Watcher guide back to this guide

Post-Fix Validation

npm run lint:markdown  # ✅ Passed
litestream replicate -config <fixed-config>  # ✅ Passed

All documentation now accurately reflects the Litestream configuration format and has been validated against the actual binary.

@corylanou corylanou merged commit d02845f into main Dec 13, 2025
1 check passed
@corylanou corylanou deleted the docs/95-directory-replication branch December 13, 2025 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document directory replication configuration

3 participants