ZeroRetain

Ephemeral Cryptographic Intelligence Store

A secure, zero-knowledge data storage system built with FastAPI that implements envelope encryption to ensure data privacy and security. ZeroRetain allows you to store sensitive data with strong cryptographic guarantees, where even the database administrator cannot access the raw data.

🔐 Overview

ZeroRetain is a production-ready API gateway that implements a sophisticated encryption architecture using the envelope encryption pattern. It combines MongoDB for persistent storage, Redis for caching, and industry-standard cryptographic libraries to provide a secure data storage solution.

Key Features

• 🔒 Envelope Encryption: Two-tier encryption system using Data Encryption Keys (DEK) and Master Keys
• 🛡️ Zero-Knowledge Architecture: Data is encrypted before storage; the system never stores plaintext
• ⚡ High Performance: Async/await patterns with FastAPI and Motor (async MongoDB driver)
• 🔄 RESTful API: Simple, intuitive endpoints for data operations
• 📦 Scalable Design: Ready for Redis caching integration and horizontal scaling
• 🔑 Key Versioning: Built-in support for key rotation and version management

🏗️ Architecture

Encryption Flow

┌─────────────────────────────────────────────────────────────────┐
│                        CREATE RECORD FLOW                        │
└─────────────────────────────────────────────────────────────────┘

1. Client sends payload → FastAPI Router
2. Generate DEK (Data Encryption Key)
3. Encrypt payload with DEK → Ciphertext
4. Encrypt DEK with Master Key → Encrypted DEK
5. Store {ciphertext, encrypted_dek, key_version} in MongoDB
6. Return record_id to client

┌─────────────────────────────────────────────────────────────────┐
│                        RETRIEVE RECORD FLOW                      │
└─────────────────────────────────────────────────────────────────┘

1. Client requests record_id → FastAPI Router
2. Fetch {ciphertext, encrypted_dek} from MongoDB
3. Decrypt DEK using Master Key → DEK
4. Decrypt ciphertext using DEK → Original payload
5. Return payload to client

Project Structure

ZeroRetain/
├── app/
│   ├── main.py                 # FastAPI application entry point
│   ├── core/
│   │   ├── config.py          # Configuration and environment variables
│   │   └── database.py        # MongoDB and Redis client initialization
│   ├── crypto/
│   │   └── encryption.py      # Envelope encryption implementation
│   ├── models/
│   │   └── data.py            # Pydantic models for request/response
│   ├── routers/
│   │   └── data.py            # API endpoint definitions
│   ├── services/
│   │   └── data.py            # Business logic for secure record operations
│   └── workers/               # (Reserved for background tasks)
├── requirements.txt           # Python dependencies
├── LICENSE                    # MIT License
└── README.md                  # This file

🚀 Getting Started

Prerequisites

• Python 3.8+
• MongoDB (local or remote instance)
• Redis (optional, for caching)
• Git

Installation

1. Clone the repository

   git clone https://github.com/yourusername/ZeroRetain.git
   cd ZeroRetain

2. Create a virtual environment

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies

   pip install -r requirements.txt

4. Generate a Master Key

   python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

5. Create a .env file

   touch .env

   Add the following configuration:

   MONGO_URI=mongodb://localhost:27017/
   REDIS_HOST=localhost
   REDIS_PORT=6379
   MASTER_KEY=your_generated_key_here
   GATEWAY_PORT=8000

6. Run the application

   python app/main.py

   Or using uvicorn directly:

   uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

📡 API Endpoints

Base URL

http://localhost:8000

Health Check

GET /

Check if the API gateway is running.

Response:

{
  "message": "ZeroRetain Gateway is live"
}

---

Create Secure Record

POST /data/

Store encrypted data securely.

Request Body:

{
  "payload": {
    "username": "john_doe",
    "email": "john@example.com",
    "sensitive_info": "This is confidential"
  }
}

Response:

{
  "id": "507f1f77bcf86cd799439011"
}

cURL Example:

curl -X POST "http://localhost:8000/data/" \
  -H "Content-Type: application/json" \
  -d '{"payload": {"username": "john_doe", "email": "john@example.com"}}'

---

Retrieve Secure Record

GET /data/{record_id}

Retrieve and decrypt stored data.

Response:

{
  "id": "507f1f77bcf86cd799439011",
  "payload": {
    "username": "john_doe",
    "email": "john@example.com",
    "sensitive_info": "This is confidential"
  }
}

cURL Example:

curl -X GET "http://localhost:8000/data/507f1f77bcf86cd799439011"

🔧 Technical Details

Encryption Implementation

ZeroRetain uses Fernet (symmetric encryption) from the cryptography library, which provides:

• AES-128 encryption in CBC mode
• HMAC using SHA256 for authentication
• Timestamp-based token expiration support

Envelope Encryption Pattern

1. DEK (Data Encryption Key)

   • Unique key generated for each record
   • Used to encrypt the actual payload data
   • Never stored in plaintext

2. Master Key

   • Single key used to encrypt all DEKs
   • Stored securely in environment variables
   • Should be rotated periodically for security

3. Key Versioning

   • Each record stores a key_version field
   • Enables seamless key rotation
   • Maintains backward compatibility

Database Schema

MongoDB Collection: data

{
  "_id": ObjectId("507f1f77bcf86cd799439011"),
  "ciphertext": b"gAAAAABh...",          // Encrypted payload
  "encrypted_dek": b"gAAAAABh...",       // Encrypted DEK
  "key_version": 1                        // Master key version
}

Dependencies

Core Framework

• FastAPI (0.129.0): Modern, high-performance web framework
• Uvicorn (0.40.0): ASGI server for running FastAPI

Database & Caching

• Motor (3.7.1): Async MongoDB driver
• PyMongo (4.16.0): MongoDB driver (Motor dependency)
• Redis (7.1.1): In-memory data structure store

Security & Encryption

• Cryptography (46.0.5): Cryptographic recipes and primitives
• BCrypt (5.0.0): Password hashing (for future authentication)
• Zxcvbn (4.5.0): Password strength estimation

Data Validation

• Pydantic (2.12.5): Data validation using Python type annotations

Utilities

• Loguru (0.7.3): Advanced logging
• Python-dotenv (1.2.1): Environment variable management

🔒 Security Best Practices

Master Key Management

⚠️ NEVER commit your master key to version control!

• Store the master key in environment variables
• Use a secrets management service (AWS Secrets Manager, HashiCorp Vault, etc.) in production
• Rotate the master key periodically
• Implement a key migration strategy for rotation

Recommendations

1. Use HTTPS: Always use TLS/SSL in production
2. Rate Limiting: Implement rate limiting to prevent abuse
3. Authentication: Add JWT or OAuth2 authentication before deployment
4. Input Validation: The application uses Pydantic for validation, but add additional business logic validation
5. Audit Logging: Track all data access operations
6. Backup Strategy: Encrypt backups and store them securely
7. Network Security: Use VPC/firewall rules to restrict database access

🚦 Configuration

All configuration is managed through environment variables in the .env file:

Variable	Description	Default	Required
MONGO_URI	MongoDB connection string	mongodb://localhost:27017/	No
REDIS_HOST	Redis server hostname	localhost	No
REDIS_PORT	Redis server port	6379	No
MASTER_KEY	Fernet encryption key	-	Yes
GATEWAY_PORT	API server port	8000	No

🧪 Testing

Manual Testing with cURL

1. Create a record:

   RECORD_ID=$(curl -s -X POST "http://localhost:8000/data/" \
     -H "Content-Type: application/json" \
     -d '{"payload": {"test": "data", "secret": "value"}}' \
     | jq -r '.id')
   echo "Created record: $RECORD_ID"

2. Retrieve the record:

   curl -X GET "http://localhost:8000/data/$RECORD_ID" | jq

Testing with Python

import requests

# Create a record
response = requests.post(
    "http://localhost:8000/data/",
    json={"payload": {"username": "test", "password": "secret123"}}
)
record_id = response.json()["id"]
print(f"Created record: {record_id}")

# Retrieve the record
response = requests.get(f"http://localhost:8000/data/{record_id}")
print(f"Retrieved data: {response.json()}")

📊 Performance Considerations

• Async Operations: All database operations use async/await for non-blocking I/O
• Connection Pooling: Motor automatically manages connection pools
• Redis Integration: Redis client is initialized but not yet utilized (ready for caching layer)
• Scalability: Stateless design allows horizontal scaling behind a load balancer

🛣️ Roadmap

Planned Features

• TTL (Time-To-Live): Auto-expiration of records after a specified time
• Redis Caching: Cache frequently accessed records
• Authentication: JWT-based authentication system
• Authorization: Role-based access control (RBAC)
• Key Rotation: Automated master key rotation
• Audit Logging: Comprehensive access logs
• Background Workers: Implement cleanup workers for expired data
• API Documentation: Interactive API docs at /docs (already available via FastAPI)
• Metrics & Monitoring: Prometheus/Grafana integration
• Docker Support: Dockerfile and docker-compose setup
• CI/CD Pipeline: Automated testing and deployment

🐛 Troubleshooting

Common Issues

1. MASTER_KEY must be set error

   • Ensure you've set the MASTER_KEY in your .env file
   • Verify the key is a valid Fernet key (44 characters, base64-encoded)

2. MongoDB connection failed

   • Check if MongoDB is running: mongod --version
   • Verify the MONGO_URI in your .env file
   • Check firewall settings if using a remote MongoDB instance

3. Redis connection failed

   • Check if Redis is running: redis-cli ping
   • Verify REDIS_HOST and REDIS_PORT settings
   • Note: Redis is initialized but not critical for current functionality

4. Import errors

   • Ensure all dependencies are installed: pip install -r requirements.txt
   • Verify you're using Python 3.8 or higher

🤝 Contributing

Contributions are welcome! Please follow these guidelines:

1. Fork the repository
2. Create a feature branch: git checkout -b feature/AmazingFeature
3. Commit your changes: git commit -m 'Add some AmazingFeature'
4. Push to the branch: git push origin feature/AmazingFeature
5. Open a Pull Request

Code Style

• Follow PEP 8 guidelines
• Use type hints where possible
• Add docstrings to functions and classes
• Write meaningful commit messages

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

👤 Author

Nitish Kumar

• GitHub: @yourusername

🙏 Acknowledgments

• FastAPI for the excellent async web framework
• Cryptography.io for robust encryption primitives
• MongoDB and Redis teams for powerful data storage solutions

📚 Additional Resources

• FastAPI Documentation
• Cryptography Library
• Envelope Encryption Best Practices
• MongoDB Motor Documentation

---

⚠️ Security Notice: This is a demonstration project. Before using in production, implement proper authentication, authorization, key management, and security auditing. Consider consulting with a security professional for production deployments.

---

Built with ❤️ and 🔐 by Nitish Kumar