Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .golangci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,7 @@ formatters:
settings:
goimports:
local-prefixes:
- github.com/chenyanchen/db
- github.com/chenyanchen/kv
exclusions:
generated: lax
paths:
Expand Down
181 changes: 136 additions & 45 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,45 +1,163 @@
# db
# kv

Generic key-value storage abstraction library for Go with composable implementations.

## Features

- Generic interfaces using Go generics (`KV[K, V]` and `BatchKV[K, V]`)
- Composable/stackable implementations
- Context-aware operations
- Multiple cache implementations with different performance characteristics

## Installation

```bash
go get github.com/chenyanchen/db
go get github.com/chenyanchen/kv
```

## Implementations
## Core Interfaces

The library provides two generic interfaces for key-value operations:

```go
// KV represents a key-value storage for single-key operations.
type KV[K comparable, V any] interface {
Get(ctx context.Context, k K) (V, error)
Set(ctx context.Context, k K, v V) error
Del(ctx context.Context, k K) error
}

// BatchKV represents a key-value storage for batch operations.
type BatchKV[K comparable, V any] interface {
Get(ctx context.Context, keys []K) (map[K]V, error)
Set(ctx context.Context, kvs map[K]V) error
Del(ctx context.Context, keys []K) error
}

// ErrNotFound is returned when a key is not found.
var ErrNotFound = errors.New("not found")
```

## Implementing Your Own KV

Implement the `kv.KV` interface to integrate any storage backend:

```go
type databaseKV struct {
db *sql.DB
}

func (s *databaseKV) Get(ctx context.Context, id int) (*User, error) {
var user User
err := s.db.QueryRowContext(ctx, "SELECT id, name FROM users WHERE id = ?", id).
Scan(&user.ID, &user.Name)
if errors.Is(err, sql.ErrNoRows) {
return nil, kv.ErrNotFound
}
return &user, err
}

func (s *databaseKV) Set(ctx context.Context, id int, user *User) error {
_, err := s.db.ExecContext(ctx,
"INSERT INTO users (id, name) VALUES (?, ?) ON DUPLICATE KEY UPDATE name = ?",
id, user.Name, user.Name)
return err
}

func (s *databaseKV) Del(ctx context.Context, id int) error {
_, err := s.db.ExecContext(ctx, "DELETE FROM users WHERE id = ?", id)
return err
}
```

## Built-in Implementations

### cachekv

In-memory cache implementations:

| Implementation | Description | Use Case |
| ---------------------------- | -------------------------------- | --------------------------- |
| `NewRWMutex()` | Simple RWMutex-protected map | Low concurrency workloads |
| `NewSharded(numShards)` | Sharded map with per-shard locks | High concurrency workloads |
| `NewLRU(size, onEvict, ttl)` | LRU cache with optional TTL | Bounded cache with eviction |

### layerkv
## Composition

The power of `kv.KV` comes from composing implementations together.

### layerkv - Cache + Store Layers

Composes cache and store layers with configurable write strategies:
Compose a cache layer with a persistent store:

```go
// Default: Write-invalidate (deletes from cache on Set)
kv, _ := layerkv.New(cache, store)
cache, _ := cachekv.NewLRU[int, *User](1000, nil, 0)
store := &databaseKV{db: db}

// Write-through (updates cache on Set)
kv, _ := layerkv.New(cache, store, layerkv.WithWriteThrough())
// Cache-aside pattern: checks cache first, falls back to store
userKV, _ := layerkv.New(cache, store)

// With write-through: updates cache on Set instead of invalidating
userKV, _ := layerkv.New(cache, store, layerkv.WithWriteThrough())
```

### singleflightkv
### singleflightkv - Request Deduplication

Deduplicates concurrent requests for the same key.
Prevent duplicate concurrent requests for the same key:

```go
store := &databaseKV{db: db}
userKV, _ := singleflightkv.New(store)
```

### Custom Wrappers - Telemetry Example

Create your own wrapper to add cross-cutting concerns:

```go
type telemetry[K comparable, V any] struct {
next kv.KV[K, V]
record func(operation string, duration time.Duration)
}

func (t telemetry[K, V]) Get(ctx context.Context, k K) (V, error) {
start := time.Now()
v, err := t.next.Get(ctx, k)
t.record("Get", time.Since(start))
return v, err
}

func (t telemetry[K, V]) Set(ctx context.Context, k K, v V) error {
start := time.Now()
err := t.next.Set(ctx, k, v)
t.record("Set", time.Since(start))
return err
}

func (t telemetry[K, V]) Del(ctx context.Context, k K) error {
start := time.Now()
err := t.next.Del(ctx, k)
t.record("Del", time.Since(start))
return err
}
```

### Full Composition Example

Combine multiple layers for a production-ready setup:

```go
// 1. Database backend (your implementation)
dbKV := &databaseKV{db: db}

// 2. Protect database with request deduplication
dbKV, _ = singleflightkv.New(dbKV)

// 3. Add telemetry to database
dbWithMetrics := NewTelemetry(dbKV, dbRecorder)

// 4. LRU cache layer
cache, _ := cachekv.NewLRU[int, *User](1000, nil, time.Minute*5)

// 5. Add telemetry to cache
cacheWithMetrics := NewTelemetry(cache, cacheRecorder)

// 6. Compose: cache + store with write-through
userKV, _ := layerkv.New(cacheWithMetrics, dbWithMetrics, layerkv.WithWriteThrough())
```

## Benchmarks

Expand Down Expand Up @@ -69,33 +187,6 @@ BenchmarkLayerKV_SetThenGet_WriteInvalidate-10 25039428 48 ns/op
BenchmarkLayerKV_SetThenGet_WriteThrough-10 40282930 30 ns/op (1.6x faster)
```

## Usage

```go
package main

import (
"context"
"github.com/chenyanchen/db/cachekv"
"github.com/chenyanchen/db/layerkv"
)

func main() {
ctx := context.Background()

// Simple sharded cache for high concurrency
cache := cachekv.NewSharded[string, string](32)
cache.Set(ctx, "key", "value")
v, _ := cache.Get(ctx, "key")

// Layered cache with database backend
lru, _ := cachekv.NewLRU[string, string](1000, nil, 0)
store := &myDatabaseKV{} // implements db.KV[string, string]
layered, _ := layerkv.New(lru, store, layerkv.WithWriteThrough())
layered.Get(ctx, "key") // checks cache first, then store
}
```

## License

MIT
2 changes: 1 addition & 1 deletion batchkv.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package db
package kv

import "context"

Expand Down
10 changes: 5 additions & 5 deletions cachekv/batchkv.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,21 +5,21 @@ import (
"errors"
"fmt"

"github.com/chenyanchen/db"
kv "github.com/chenyanchen/kv"
)

// cacheBatchKV is a struct that contains a cache and a source BatchKV.
// It is used to cache batch operations.
//
// Important: cacheBatchKV are not guaranteed to get all the values of keys, it guarantees no error.
type cacheBatchKV[K comparable, V any] struct {
cache db.KV[K, V]
cache kv.KV[K, V]

source db.BatchKV[K, V]
source kv.BatchKV[K, V]
}

// NewBatch creates a new cacheBatchKV instance with the given source and options.
func NewBatch[K comparable, V any](cache db.KV[K, V], source db.BatchKV[K, V]) *cacheBatchKV[K, V] {
func NewBatch[K comparable, V any](cache kv.KV[K, V], source kv.BatchKV[K, V]) *cacheBatchKV[K, V] {
return &cacheBatchKV[K, V]{
cache: cache,
source: source,
Expand All @@ -46,7 +46,7 @@ func (c *cacheBatchKV[K, V]) Get(ctx context.Context, keys []K) (map[K]V, error)
continue
}

if errors.Is(err, db.ErrNotFound) {
if errors.Is(err, kv.ErrNotFound) {
misses = append(misses, key)
continue
}
Expand Down
4 changes: 2 additions & 2 deletions cachekv/lru.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import (
"github.com/hashicorp/golang-lru/v2/expirable"
"github.com/hashicorp/golang-lru/v2/simplelru"

"github.com/chenyanchen/db"
kv "github.com/chenyanchen/kv"
)

type lruKV[K comparable, V any] struct {
Expand All @@ -33,7 +33,7 @@ func (c *lruKV[K, V]) Get(ctx context.Context, k K) (V, error) {
if ok {
return v, nil
}
return v, db.ErrNotFound
return v, kv.ErrNotFound
}

func (c *lruKV[K, V]) Set(ctx context.Context, k K, v V) error {
Expand Down
4 changes: 2 additions & 2 deletions cachekv/rwmutextkv.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"context"
"sync"

"github.com/chenyanchen/db"
kv "github.com/chenyanchen/kv"
)

type rwMutexKV[K comparable, V any] struct {
Expand All @@ -22,7 +22,7 @@ func (s *rwMutexKV[K, V]) Get(ctx context.Context, k K) (V, error) {

v, ok := s.m[k]
if !ok {
return v, db.ErrNotFound
return v, kv.ErrNotFound
}
return v, nil
}
Expand Down
4 changes: 2 additions & 2 deletions cachekv/shardedkv.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"context"
"hash/maphash"

"github.com/chenyanchen/db"
kv "github.com/chenyanchen/kv"
)

const defaultShardCount = 32
Expand Down Expand Up @@ -44,7 +44,7 @@ func (s *shardedKV[K, V]) Get(ctx context.Context, k K) (V, error) {
shard.mu.RUnlock()

if !ok {
return v, db.ErrNotFound
return v, kv.ErrNotFound
}
return v, nil
}
Expand Down
6 changes: 3 additions & 3 deletions cachekv/shardedkv_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"

"github.com/chenyanchen/db"
kvpkg "github.com/chenyanchen/kv"
)

func TestShardedKV_Get(t *testing.T) {
Expand All @@ -18,7 +18,7 @@ func TestShardedKV_Get(t *testing.T) {

// Test not found
_, err := kv.Get(ctx, "missing")
require.ErrorIs(t, err, db.ErrNotFound)
require.ErrorIs(t, err, kvpkg.ErrNotFound)

// Test found
require.NoError(t, kv.Set(ctx, "key1", "value1"))
Expand Down Expand Up @@ -54,7 +54,7 @@ func TestShardedKV_Del(t *testing.T) {

// Should be not found
_, err := kv.Get(ctx, "key1")
require.ErrorIs(t, err, db.ErrNotFound)
require.ErrorIs(t, err, kvpkg.ErrNotFound)

// Delete non-existent is fine
require.NoError(t, kv.Del(ctx, "missing"))
Expand Down
Loading