feat: add S3-compatible storage support (MinIO)

.env.example

@@ -21,3 +21,12 @@ GOOGLE_CLIENT_SECRET=
 # Facebook: https://developers.facebook.com/apps/
 FACEBOOK_CLIENT_ID=
 FACEBOOK_CLIENT_SECRET=
+
+# --- S3-compatible storage (optional) ---
+# Use with: docker compose --profile s3 up minio
+# MinIO defaults: http://localhost:9900 (S3 API), minioadmin/minioadmin
+S3_ENDPOINT=
+S3_ACCESS_KEY_ID=
+S3_SECRET_ACCESS_KEY=
+S3_BUCKET=
+S3_USE_SSL=true

CONTRIBUTING.md

@@ -11,6 +11,7 @@ mails/
 ├── cmd/mails/           # Application entry point
 ├── internal/            # Private application packages
 │   ├── auth/            # OAuth2 login (GitHub, Google, Facebook)
+│   ├── storage/         # Blob store (FS or S3) for user data
 │   ├── user/            # User management, UUIDv7 IDs
 │   ├── account/         # Per-user email account CRUD
 │   ├── model/           # Shared data types
@@ -20,7 +21,7 @@ mails/
 │   │   ├── gmail/       # Gmail API sync
 │   │   └── pst/         # PST/OST file import (go-pst library)
 │   ├── search/
-│   │   ├── eml/         # .eml file parser
+│   │   ├── eml/         # .eml file parser, CID inline image extraction
 │   │   ├── index/       # DuckDB + Parquet index
 │   │   └── vector/      # Qdrant similarity search
 │   └── web/             # HTTP router, handlers, middleware
@@ -51,14 +52,15 @@ mails/
 Packages follow a strict dependency hierarchy to avoid import cycles:
 
 ```
-cmd → internal/web → internal/sync → internal/model
-                   → internal/auth
+cmd → internal/web → internal/sync   → internal/model
+                   → internal/auth   → internal/storage
                    → internal/user
                    → internal/account
                    → internal/search
 ```
 
-- Left packages may depend on right packages
+- Left packages may depend on right packages.
+- `internal/storage` (BlobStore) is used by auth, user, account, sync, and search for FS or S3-backed user data.
 - Right packages **MUST NOT** depend on left packages
 - Sub-packages at the same level use interfaces to avoid circular imports
 
@@ -121,9 +123,18 @@ go test -race ./...
 
 # Run specific package tests
 go test ./internal/search/eml/
+
+# Run e2e tests (requires GreenMail + Qdrant + Ollama)
+docker compose --profile test up -d greenmail
+go test -tags e2e -v ./tests/e2e/
+
+# Run S3 storage integration tests (requires MinIO)
+docker compose --profile s3 up -d minio
+S3_ENDPOINT=http://localhost:9900 S3_ACCESS_KEY_ID=minioadmin S3_SECRET_ACCESS_KEY=minioadmin \
+  S3_BUCKET=mails-test S3_USE_SSL=false go test -v ./internal/storage/
 ```
 
-Use `testing.T` and table-driven tests. Mock external services (IMAP, POP3, APIs).
+Use `testing.T` and table-driven tests. Mock external services (IMAP, POP3, APIs). Integration tests skip when required services (S3, GreenMail) are unavailable.
 
 ## Templates (Gitea-style)
 
@@ -225,16 +236,17 @@ Based on [Google JavaScript Style Guide](https://google.github.io/styleguide/jsg
 
 ## User Data Layout
 
-Each user's data lives under `users/{uuidv7}/`:
+Each user's data lives under `users/{uuidv7}/`. When S3 env vars are set (`S3_ENDPOINT`, `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY`), the following are stored in S3; otherwise on the local filesystem:
 
-| Path                             | Purpose                               |
-| -------------------------------- | ------------------------------------- |
-| `user.json`                      | User metadata (name, email, provider) |
-| `accounts.yml`                   | Email account configurations          |
-| `sync.sqlite`                    | Sync jobs, UIDs, state                |
-| `logs/{job-id}.jsonl`            | Structured sync logs                  |
-| `{domain}/{local}/`              | Downloaded .eml files                 |
-| `{domain}/{local}/index.parquet` | Search index per account              |
+| Path                             | Purpose                               | Storage      |
+| -------------------------------- | ------------------------------------- | ------------ |
+| `user.json`                      | User metadata (name, email, provider) | FS or S3     |
+| `accounts.yml`                   | Email account configurations          | FS or S3     |
+| `sessions.json`                 | Session store (root of users dir)     | FS or S3     |
+| `sync.sqlite`                    | Sync jobs, UIDs, state                | Local only   |
+| `logs/{job-id}.jsonl`            | Structured sync logs                  | Local only   |
+| `{domain}/{local}/*.eml`         | Downloaded .eml files                 | FS or S3     |
+| `{domain}/{local}/index.parquet` | Search index per account              | Local only   |
 
 ### Email Storage
 
@@ -266,11 +278,19 @@ See [docs/DOCKER.md](docs/DOCKER.md) for tini, runtime dependencies, and build d
 # Development
 docker compose up
 
+# With S3 (MinIO) for user data storage
+docker compose --profile s3 up -d minio
+export S3_ENDPOINT=http://localhost:9900 S3_ACCESS_KEY_ID=minioadmin S3_SECRET_ACCESS_KEY=minioadmin
+docker compose up
+
 # Production build
 docker compose -f docker-compose.yml up -d
 
 # Run tests
 docker compose run --rm mails go test ./...
+
+# Run e2e tests (GreenMail for IMAP/POP3)
+docker compose --profile test up -d greenmail
 ```
 
 ## API Reference
@@ -337,3 +357,4 @@ All API endpoints require authentication (session cookie or `Authorization: Bear
 - **IMAP connection refused:** Check host, port, and SSL settings. Gmail requires an App Password (not regular password).
 - **Search returns 0 results:** Run reindex after syncing new emails.
 - **SQLite busy:** Increase `_busy_timeout` or reduce concurrent sync jobs.
+- **S3/MinIO connection failed:** When using MinIO, set `S3_USE_SSL=false` and ensure `S3_ENDPOINT` includes the scheme (e.g. `http://localhost:9900`). Run MinIO with `docker compose --profile s3 up minio`.

README.md

@@ -79,6 +79,11 @@ docker pull ghcr.io/eslider/mail-archive:v1.0.1
 | `QDRANT_URL`             | —                       | Qdrant gRPC address for similarity search |
 | `OLLAMA_URL`             | —                       | Ollama API URL for embeddings             |
 | `EMBED_MODEL`            | `all-minilm`            | Embedding model name                      |
+| `S3_ENDPOINT`            | —                       | S3-compatible storage endpoint (e.g. MinIO) |
+| `S3_ACCESS_KEY_ID`       | —                       | S3 access key                             |
+| `S3_SECRET_ACCESS_KEY`   | —                       | S3 secret key                             |
+| `S3_BUCKET`              | `mails`                 | S3 bucket name                            |
+| `S3_USE_SSL`             | `true`                  | Use HTTPS for S3 endpoint                 |
 
 ### OAuth Setup (Optional)
 
@@ -94,6 +99,8 @@ To enable OAuth login, configure one or more providers:
 
 ## User Data Layout
 
+When S3 env vars are set, `user.json`, `accounts.yml`, `sessions.json`, and `.eml` files are stored in S3. SQLite and Parquet stay on the local filesystem.
+
 ```
 users/
   019c56a4-a9ef-79bd-b53a-ef7a080d9c90/
@@ -117,6 +124,7 @@ users/
 cmd/mails/         → Entry point, CLI (serve, fix-dates, version)
 internal/
   auth/            → OAuth2 (GitHub, Google, Facebook), sessions
+  storage/         → Blob store (FS or S3) for user data
   user/            → User storage (users/{uuid}/)
   account/         → Email account CRUD (accounts.yml)
   model/           → Shared types (User, Account, SyncJob)
@@ -153,6 +161,10 @@ go test ./...
 docker compose --profile test up -d greenmail
 go test -tags e2e -v ./tests/e2e/
 
+# Run S3 storage integration tests (requires MinIO)
+docker compose --profile s3 up -d minio
+S3_ENDPOINT=http://localhost:9900 S3_ACCESS_KEY_ID=minioadmin S3_SECRET_ACCESS_KEY=minioadmin S3_BUCKET=mails-test S3_USE_SSL=false go test -v ./internal/storage/
+
 # Docker dev mode (auto-rebuild on changes)
 docker compose watch
 ```
@@ -235,11 +247,23 @@ web/static/
     sw-register.js                   # Service worker registration
 ```
 
-## Todo
+## S3 Storage (Optional)
 
-- **Storage backend** — S3 (AWS/Minio) or pluggable local filesystem
+Store user data on S3 when `S3_ENDPOINT` and credentials are set:
+
+```bash
+docker compose --profile s3 up -d minio
+export S3_ENDPOINT=http://localhost:9900
+export S3_ACCESS_KEY_ID=minioadmin
+export S3_SECRET_ACCESS_KEY=minioadmin
+export S3_BUCKET=mails
+export S3_USE_SSL=false
+./mails serve
+```
+
+## Todo
 
-See (TODO's)[TODO.md]
+See [TODO.md](TODO.md)
 
 ## Ideas
 

cmd/mails/main.go

@@ -19,6 +19,7 @@ import (
 
 	"github.com/eslider/mails/internal/account"
 	"github.com/eslider/mails/internal/auth"
+	"github.com/eslider/mails/internal/storage"
 	"github.com/eslider/mails/internal/sync"
 	"github.com/eslider/mails/internal/user"
 	"github.com/eslider/mails/internal/web"
@@ -74,27 +75,38 @@ Environment:
 
   QDRANT_URL          Qdrant gRPC address for similarity search
   OLLAMA_URL          Ollama API URL for embeddings
-  EMBED_MODEL         Ollama embedding model (default: all-minilm)`)
+  EMBED_MODEL         Ollama embedding model (default: all-minilm)
+
+  S3_ENDPOINT         S3-compatible storage (e.g. MinIO)
+  S3_ACCESS_KEY_ID    S3 access key
+  S3_SECRET_ACCESS_KEY S3 secret key
+  S3_BUCKET           S3 bucket (default: mails)
+  S3_USE_SSL          Use HTTPS for S3 (default: true)`)
 }
 
 func runServe() {
 	listenAddr := envOr("LISTEN_ADDR", ":8090")
 	dataDir := envOr("DATA_DIR", "./users")
 	baseURL := envOr("BASE_URL", "http://localhost:8090")
 
+	blobStore, err := storage.NewBlobStore(dataDir)
+	if err != nil {
+		log.Fatalf("Failed to init blob store: %v", err)
+	}
+
 	// Initialize stores.
-	userStore, err := user.NewStore(dataDir)
+	userStore, err := user.NewStore(dataDir, blobStore)
 	if err != nil {
 		log.Fatalf("Failed to init user store: %v", err)
 	}
 
-	sessionStore, err := auth.NewSessionStore(dataDir)
+	sessionStore, err := auth.NewSessionStore(dataDir, blobStore)
 	if err != nil {
 		log.Fatalf("Failed to init session store: %v", err)
 	}
 
-	accountStore := account.NewStore(dataDir)
-	syncService := sync.NewService(dataDir, accountStore)
+	accountStore := account.NewStore(dataDir, blobStore)
+	syncService := sync.NewService(dataDir, accountStore, blobStore)
 
 	// Configure OAuth providers.
 	var ghCfg, glCfg, fbCfg *auth.ProviderConfig
@@ -137,6 +149,7 @@ func runServe() {
 		Auth:       providers,
 		Sync:       syncService,
 		UsersDir:   dataDir,
+		BlobStore:  blobStore,
 		QdrantURL:  envOr("QDRANT_URL", ""),
 		OllamaURL:  envOr("OLLAMA_URL", ""),
 		EmbedModel: envOr("EMBED_MODEL", "all-minilm"),

docker-compose.yml

@@ -31,6 +31,12 @@ services:
       QDRANT_URL: "http://127.0.0.1:6334"
       OLLAMA_URL: "http://172.17.0.1:11434"
       EMBED_MODEL: "all-minilm"
+      # S3-compatible storage (optional, e.g. MinIO)
+      S3_ENDPOINT: "${S3_ENDPOINT:-}"
+      S3_ACCESS_KEY_ID: "${S3_ACCESS_KEY_ID:-}"
+      S3_SECRET_ACCESS_KEY: "${S3_SECRET_ACCESS_KEY:-}"
+      S3_BUCKET: "${S3_BUCKET:-}"
+      S3_USE_SSL: "${S3_USE_SSL:-true}"
     network_mode: host
     # docker compose watch — rebuild on changes (uses .dockerignore implicitly)
     develop:
@@ -46,6 +52,23 @@ services:
         max-file: "3"
         labels: "service"
 
+  # S3-compatible object storage (MinIO) — optional
+  minio:
+    image: minio/minio:latest
+    container_name: minio
+    restart: unless-stopped
+    command: server /data
+    environment:
+      MINIO_ROOT_USER: "${MINIO_ROOT_USER:-minioadmin}"
+      MINIO_ROOT_PASSWORD: "${MINIO_ROOT_PASSWORD:-minioadmin}"
+    volumes:
+      - minio_data:/data
+    ports:
+      - "9900:9000"   # S3 API (host 9900 to avoid conflicts)
+      - "9901:9001"   # Web console
+    profiles:
+      - s3
+
   # Vector DB for similarity search (optional)
   qdrant:
     image: qdrant/qdrant:latest
@@ -73,3 +96,4 @@ services:
 
 volumes:
   qdrant_storage:
+  minio_data:

go.mod

@@ -3,6 +3,9 @@ module github.com/eslider/mails
 go 1.24.4
 
 require (
+	github.com/aws/aws-sdk-go-v2 v1.41.1
+	github.com/aws/aws-sdk-go-v2/credentials v1.19.7
+	github.com/aws/aws-sdk-go-v2/service/s3 v1.96.0
 	github.com/emersion/go-message v0.18.2
 	github.com/go-chi/chi/v5 v5.2.1
 	github.com/google/uuid v1.6.0
@@ -20,6 +23,15 @@ require (
 require (
 	cloud.google.com/go/compute/metadata v0.7.0 // indirect
 	github.com/apache/arrow-go/v18 v18.1.0 // indirect
+	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.4 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.17 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.17 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.17 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.4 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.8 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.17 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.17 // indirect
+	github.com/aws/smithy-go v1.24.0 // indirect
 	github.com/go-viper/mapstructure/v2 v2.2.1 // indirect
 	github.com/goccy/go-json v0.10.5 // indirect
 	github.com/godzie44/go-uring v0.0.0-20220926161041-69611e8b13d5 // indirect

go.sum

@@ -6,6 +6,30 @@ github.com/apache/arrow-go/v18 v18.1.0 h1:agLwJUiVuwXZdwPYVrlITfx7bndULJ/dggbnLF
 github.com/apache/arrow-go/v18 v18.1.0/go.mod h1:tigU/sIgKNXaesf5d7Y95jBBKS5KsxTqYBKXFsvKzo0=
 github.com/apache/thrift v0.21.0 h1:tdPmh/ptjE1IJnhbhrcl2++TauVjy242rkV/UzJChnE=
 github.com/apache/thrift v0.21.0/go.mod h1:W1H8aR/QRtYNvrPeFXBtobyRkd0/YVhTc6i07XIAgDw=
+github.com/aws/aws-sdk-go-v2 v1.41.1 h1:ABlyEARCDLN034NhxlRUSZr4l71mh+T5KAeGh6cerhU=
+github.com/aws/aws-sdk-go-v2 v1.41.1/go.mod h1:MayyLB8y+buD9hZqkCW3kX1AKq07Y5pXxtgB+rRFhz0=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.4 h1:489krEF9xIGkOaaX3CE/Be2uWjiXrkCH6gUX+bZA/BU=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.4/go.mod h1:IOAPF6oT9KCsceNTvvYMNHy0+kMF8akOjeDvPENWxp4=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.7 h1:tHK47VqqtJxOymRrNtUXN5SP/zUTvZKeLx4tH6PGQc8=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.7/go.mod h1:qOZk8sPDrxhf+4Wf4oT2urYJrYt3RejHSzgAquYeppw=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.17 h1:xOLELNKGp2vsiteLsvLPwxC+mYmO6OZ8PYgiuPJzF8U=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.17/go.mod h1:5M5CI3D12dNOtH3/mk6minaRwI2/37ifCURZISxA/IQ=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.17 h1:WWLqlh79iO48yLkj1v3ISRNiv+3KdQoZ6JWyfcsyQik=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.17/go.mod h1:EhG22vHRrvF8oXSTYStZhJc1aUgKtnJe+aOiFEV90cM=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.17 h1:JqcdRG//czea7Ppjb+g/n4o8i/R50aTBHkA7vu0lK+k=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.17/go.mod h1:CO+WeGmIdj/MlPel2KwID9Gt7CNq4M65HUfBW97liM0=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.4 h1:0ryTNEdJbzUCEWkVXEXoqlXV72J5keC1GvILMOuD00E=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.4/go.mod h1:HQ4qwNZh32C3CBeO6iJLQlgtMzqeG17ziAA/3KDJFow=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.8 h1:Z5EiPIzXKewUQK0QTMkutjiaPVeVYXX7KIqhXu/0fXs=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.8/go.mod h1:FsTpJtvC4U1fyDXk7c71XoDv3HlRm8V3NiYLeYLh5YE=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.17 h1:RuNSMoozM8oXlgLG/n6WLaFGoea7/CddrCfIiSA+xdY=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.17/go.mod h1:F2xxQ9TZz5gDWsclCtPQscGpP0VUOc8RqgFM3vDENmU=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.17 h1:bGeHBsGZx0Dvu/eJC0Lh9adJa3M1xREcndxLNZlve2U=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.17/go.mod h1:dcW24lbU0CzHusTE8LLHhRLI42ejmINN8Lcr22bwh/g=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.96.0 h1:oeu8VPlOre74lBA/PMhxa5vewaMIMmILM+RraSyB8KA=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.96.0/go.mod h1:5jggDlZ2CLQhwJBiZJb4vfk4f0GxWdEDruWKEJ1xOdo=
+github.com/aws/smithy-go v1.24.0 h1:LpilSUItNPFr1eY85RYgTIg5eIEPtvFbskaFcmmIUnk=
+github.com/aws/smithy-go v1.24.0/go.mod h1:LEj2LM3rBRQJxPZTB4KuzZkaZYnZPnvgIhb4pu07mx0=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/emersion/go-message v0.18.2 h1:rl55SQdjd9oJcIoQNhubD2Acs1E6IzlZISRTK7x/Lpg=