A fast, lightweight vector database with SQL-powered payload filtering
VectorXLite is a high-performance vector database built on SQLite with HNSW-based approximate nearest neighbor search. It combines the power of vector similarity with the flexibility of SQL for metadata filtering, making it ideal for AI/ML applications, semantic search, and recommendation systems.
VectorXLite adapts to your needs with three distinct deployment modes:
|
In-process library Direct Rust library integration with zero network overhead. Best for:
Architecture: |
gRPC Server Language-agnostic server for remote access. Best for:
Architecture: |
Raft Cluster High-availability cluster with consensus. Best for:
Architecture: |
| Feature | Embedded | Standalone | Distributed |
|---|---|---|---|
| Language | Rust only | Any (gRPC) | Any (gRPC) |
| Network | None | TCP/gRPC | TCP/gRPC + Raft |
| Setup | Add dependency | Start server | Start cluster |
| Availability | Single process | Single server | Multi-node HA |
| Consistency | Local ACID | Single node | Raft consensus |
| Latency | Sub-millisecond | ~1-5ms | ~5-20ms |
| Use Case | Apps, tests | Services | Production |
- HNSW Algorithm - Sub-millisecond similarity search on millions of vectors
- Multiple Distance Functions - Cosine, L2 (Euclidean), Inner Product
- Flexible Dimensions - Support for vectors of any dimension
- SQL Filtering - Full SQL support for complex metadata queries
- JSON Support - Store and query JSON payloads
- JOINs & Aggregations - Complex relational queries on payloads
- ACID Transactions - Atomic operations with rollback support
- Snapshot Support - Point-in-time backups and recovery
- No Orphans - Guaranteed consistency between vectors and payloads
- Connection Pooling - Concurrent access with r2d2
- Persistent Storage - File-backed or in-memory modes
- Optimized Indexing - Fast inserts and searches
use vector_xlite::{VectorXLite, customizer::SqliteConnectionCustomizer, types::*};
use r2d2::Pool;
use r2d2_sqlite::SqliteConnectionManager;
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Create in-memory database
let manager = SqliteConnectionManager::memory();
let pool = Pool::builder()
.connection_customizer(SqliteConnectionCustomizer::new())
.build(manager)?;
let db = VectorXLite::new(pool)?;
// Create collection
let config = CollectionConfigBuilder::default()
.collection_name("products")
.vector_dimension(384)
.distance(DistanceFunction::Cosine)
.payload_table_schema(
"CREATE TABLE products (rowid INTEGER PRIMARY KEY, name TEXT, price REAL)"
)
.build()?;
db.create_collection(config)?;
// Insert vector
let point = InsertPoint::builder()
.collection_name("products")
.id(1)
.vector(vec![0.1; 384])
.payload_insert_query("INSERT INTO products VALUES (?1, 'Headphones', 99.99)")
.build()?;
db.insert(point)?;
// Search
let search = SearchPoint::builder()
.collection_name("products")
.vector(vec![0.15; 384])
.top_k(10)
.payload_search_query("SELECT * FROM products WHERE price < 150")
.build()?;
let results = db.search(search)?;
println!("Results: {:?}", results);
Ok(())
}# Start the gRPC server
cargo run -p vector_xlite_server --release
# Use Go client
import "github.com/your-org/vectorxlite-go-client/client"
client, _ := client.NewVectorXLiteClient("localhost:50051")Full Guide → (Coming soon)
# Start 3-node cluster with Raft consensus
cd distributed/cluster
./scripts/start_cluster.sh
# Use cluster client
client, _ := cluster.NewClusterClient("localhost:5002")Full Guide → (Coming soon)
[dependencies]
vector_xlite = "1.2"
r2d2 = "0.8"
r2d2_sqlite = "0.31"# Server
cargo build --release -p vector_xlite_server
# Go Client
go get github.com/your-org/vectorxlite-go-client# Build cluster
cd distributed/cluster
make buildvector-db-rs/
├── embedded/ # Embedded library mode
│ ├── core/ # Core Rust library
│ ├── examples/ # Rust examples
│ └── docs/ # Embedded mode docs
│
├── standalone/ # Standalone server mode (coming soon)
│ ├── server/ # gRPC server
│ ├── clients/ # Go, Rust, Python clients
│ └── examples/ # Client examples
│
├── distributed/ # Distributed cluster mode (coming soon)
│ ├── cluster/ # Raft-based cluster
│ ├── clients/ # Cluster clients
│ └── examples/ # Cluster examples
│
├── proto/ # Protocol buffer definitions
├── tests/ # Integration tests
├── docs/ # Comprehensive documentation
└── scripts/ # Build and deployment scripts
# Run all examples
cargo run -p embedded-examples --release
# Output:
# Inserted complex story points
# Search Results: [...]# Run all tests
cargo test --release
# Run integration tests
cargo test -p vector_xlite_tests --release┌─────────────────────────────────────────┐
│ Your Application │
├─────────────────────────────────────────┤
│ VectorXLite Library │
├──────────────────┬──────────────────────┤
│ HNSW Index │ SQLite │
└──────────────────┴──────────────────────┘
┌─────────────┐
│ Client │
│ (Any Lang) │
└──────┬──────┘
│ gRPC
┌──────▼──────────────────────────────────┐
│ VectorXLite gRPC Server (Rust) │
├──────────────────┬──────────────────────┤
│ HNSW Index │ SQLite │
└──────────────────┴──────────────────────┘
┌─────────────┐
│ Client │
└──────┬──────┘
│ gRPC
┌──────▼──────────────────────────────────┐
│ Cluster Proxy (Go + Raft) │
│ Leader ◄─── Consensus ───► Follower │
└──────┬──────────────────────┬───────────┘
│ │
┌──────▼──────┐ ┌──────▼──────┐
│ VectorXLite │ │ VectorXLite │
│ Server │ │ Server │
└─────────────┘ └─────────────┘
| Use Case | Description | Recommended Mode |
|---|---|---|
| RAG for LLMs | Retrieval-Augmented Generation | Embedded or Standalone |
| Semantic Search | Find documents by meaning | Any mode |
| Recommendation | Similar item suggestions | Embedded or Standalone |
| Image Search | Visual similarity | Standalone |
| Production System | High availability required | Distributed |
| Microservices | Multi-language services | Standalone |
| Operation | Embedded | Standalone | Distributed |
|---|---|---|---|
| Insert (1000 vectors) | ~50ms | ~100ms | ~200ms |
| Search (top 10) | <1ms | ~2ms | ~10ms |
| Throughput | 20k ops/s | 10k ops/s | 5k ops/s |
| Dataset Size | Limited by RAM | Limited by disk | Distributed |
Benchmarks on: Rust 1.70, Ubuntu 22.04, 16-core CPU, 384-dim vectors
- Embedded Mode Guide - In-process library usage
- Standalone Mode Guide - gRPC server setup (coming soon)
- Distributed Mode Guide - Cluster deployment (coming soon)
- API Reference - Full API documentation
- Architecture - System design (coming soon)
Contributions are welcome! Please see our Contributing Guide.
# Clone the repository
git clone https://github.com/uttom-akash/vector-db-rs
cd vector-db-rs
# Build all modes
make build-all
# Run all tests
make test-all- Embedded mode with SQLite + HNSW
- Snapshot support for backups
- Atomic transactions
- Distributed cluster with Raft consensus
- Complete standalone mode migration
- Complete distributed mode migration
- Python client library
- Observability stack (Prometheus, Grafana)
- Kubernetes deployment guides
- Performance optimizations
- Rust: 1.70 or later
- SQLite: 3.35 or later (with extension loading enabled)
- Go: 1.20+ (for distributed mode)
- Platforms: Linux, macOS, Windows
This project is licensed under the MIT OR Apache-2.0 License - see the LICENSE file for details.
Built with:
- Rust - Systems programming language
- SQLite - Embedded database
- HNSW - Approximate nearest neighbor search
- Raft - Distributed consensus (HashiCorp implementation)
- gRPC - Remote procedure calls
VectorXLite - Fast, flexible, and reliable vector search