Conversation
Add a Turso storage driver using go-libsql embedded replica connector, gated behind a `turso` build tag (CGO required). The driver reuses the existing ent ORM layer via EntDriver embedding, matching the postgres and sqlite driver patterns. Consolidate the duplicated `newStorageDriver()` logic from 6 build-tagged files across serve/proxy/api commands into a shared `pkg/storage/factory` package with a single build-tag pair. This also fixes the turso build variant to include SQLite as a fallback (previously it skipped sqlite entirely). - pkg/storage/turso: new driver with embedded replica support - pkg/storage/factory: shared driver selection (postgres > turso > sqlite > inmemory) - pkg/config: turso DSN, auth token, sync interval, local path flags - cmd/tapes/serve/*: thin delegates to factory.NewDriver()
| func (c *apiCommander) newStorageDriver() (storage.Driver, error) { | ||
| return factory.NewDriver(context.Background(), factory.Params{ | ||
| PostgresDSN: c.postgresDSN, | ||
| SQLitePath: c.sqlitePath, | ||
| TursoDSN: c.tursoDSN, | ||
| TursoAuthToken: c.tursoAuthToken, | ||
| TursoLocalPath: c.tursoLocalPath, | ||
| TursoSyncInterval: c.tursoSyncInterval, | ||
| Logger: c.logger, | ||
| }) | ||
| } |
There was a problem hiding this comment.
This is something I'd been meaning to get to: we have something similar for the vector drivers in pkg/vector/utils/new.go which could probably get a similar "factory" refactor.
There was a problem hiding this comment.
Agreed — addressed as part of this PR. The storage factory now uses a single factory.go with all drivers (postgres, turso, sqlite, inmemory). The vector factory at pkg/vector/utils/new.go was also split into build-tagged variants (new.go for default chroma-only, new_sqlitevec.go for sqlite-vec support).
pkg/storage/factory/factory.go
Outdated
| @@ -0,0 +1,58 @@ | |||
| //go:build !turso | |||
There was a problem hiding this comment.
This is going to create a really complicated build matrix (already complicated since we need libsqlite to build and bundle the CGO libs for normal Sqlite).
Why does this need its own build tag?
If Turso and SQLite support can't live next to eachother, then we we'd likely need to support one or the other, not both.
There was a problem hiding this comment.
Confirmed — they can't coexist. go-libsql and mattn/go-sqlite3 both bundle their own copy of the SQLite C library, causing duplicate symbol errors at link time.
Addressed by consolidating the storage layer onto go-libsql:
- SQLite storage driver now uses
go-libsql(sql.Open("libsql", "file:"+path)) instead ofmattn/go-sqlite3 - Turso driver no longer needs a build tag — it's always available
- Merged
factory.goandfactory_turso.gointo a single file, no build tags - Gated
sqlitevecbehind//go:build sqlitevecsince it's the only thing that still needsmattn/go-sqlite3(viasqlite-vec-go-bindings)
go-libsql and mattn/go-sqlite3 both bundle their own copy of the SQLite C library, causing duplicate symbol errors at link time. The conflict extends to sqlite-vec-go-bindings which also depends on mattn/go-sqlite3.
Replace mattn/go-sqlite3 with go-libsql in the SQLite storage driver
and migrate tests. This eliminates the C symbol conflict between the
two libraries and removes the need for a turso build tag.
- SQLite driver now uses go-libsql ("libsql" database/sql driver)
- Turso driver no longer requires //go:build turso
- Merge factory.go and factory_turso.go into single factory.go
- Gate sqlitevec behind //go:build sqlitevec (it still needs
mattn/go-sqlite3 via sqlite-vec-go-bindings)
- Split vector utils/new.go into tagged variants
Include qdrant vector driver from main in both build-tagged variants of vector utils.
2 participants
Summary
pkg/storage/turso) using go-libsql embedded replica connectorgo-libsqlas the single SQLite C library, replacingmattn/go-sqlite3in the SQLite storage driver — this eliminates the C symbol conflict and removes the need for atursobuild tagnewStorageDriver()logic from 6 build-tagged command files into a sharedpkg/storage/factorypackagesqlitevecbehind asqlitevecbuild tag (it still depends onmattn/go-sqlite3viasqlite-vec-go-bindings)Test plan
TAPES_STORAGE_TURSO_DSN+TAPES_STORAGE_TURSO_AUTH_TOKEN)go build ./...) compiles and works (includes turso + sqlite via go-libsql)go build -tags sqlitevec ./...compiles (includes sqlite-vec vector driver)