redis-snapper

🚀 Overview

redis-snapper is a production-ready, Python-based tool designed to simplify the migration, backup, and restoration of Redis databases. It allows users to export Redis key-value pairs to a JSON-based snapshot file and import them to another Redis instance, with support for TTL preservation, TLS authentication, dry run mode, and detailed audit logging.

Key Features

• Complete Data Migration: Export/import all Redis data types (strings, lists, sets, sorted sets, hashes) using Redis DUMP/RESTORE commands
• TTL Preservation: Maintains time-to-live (TTL) values during migration to preserve key expiration behavior
• Dry Run Mode: Simulate export/import operations without modifying the filesystem or Redis database
• Progress Tracking: Real-time progress bars using tqdm for export and import operations
• Audit Logging: Comprehensive logging with separate file logs for debugging and minimal console output
• TLS Support: Secure connections to Redis instances with TLS certificates
• Database Selection: Support for multiple Redis databases (0-15)
• Environment Variables: Configuration via environment variables for enhanced security
• Performance Tuning: Configurable batch sizes and scan counts for optimal performance
• Robust Error Handling: Graceful handling of connection issues, authentication errors, invalid data, and file I/O errors
• Verification: Automatic verification of export/import completeness

---

📦 Installation

Prerequisites

• Python: 3.8 or higher
• Redis: An accessible Redis server (version 6.x or higher)
• Dependencies: Listed in requirements.txt

Steps

1. Clone the Repository

   git clone https://github.com/siyamsarker/redis-snapper.git
   cd redis-snapper

2. Set Up Virtual Environment (Recommended) It's best practice to use a virtual environment to manage dependencies:

   # Create virtual environment
   python3 -m venv .venv
   
   # Activate it (Mac/Linux)
   source .venv/bin/activate
   
   # Activate it (Windows)
   # .venv\Scripts\activate

3. Install Dependencies

   pip install -r requirements.txt

   The requirements.txt includes:

   redis>=4.6.0
   tqdm>=4.66.1
   

4. Verify Redis Server Ensure your Redis server is running and accessible:

   redis-cli ping
   # Should return: PONG

---

🛠 Usage

Command-Line Interface

python3 migrator.py [options]

Arguments

Argument	Description	Default	Environment Variable
--export	Export snapshot from the source Redis instance	-	-
--import	Import snapshot to the target Redis instance	-	-
--snapshot-file	Path to a specific snapshot file for import	Latest in snap/	-
--host	Redis host address	localhost	REDIS_HOST
--port	Redis port number	6379	REDIS_PORT
--password	Redis password (if required)	None	REDIS_PASSWORD
--db	Redis database number (0-15)	0	REDIS_DB
--ssl	Enable SSL/TLS for Redis connection	False	REDIS_SSL
--ssl-ca-certs	Path to CA certificate for TLS	None	REDIS_SSL_CA_CERTS
--ssl-keyfile	Path to SSL key file for TLS	None	REDIS_SSL_KEYFILE
--ssl-certfile	Path to SSL certificate file for TLS	None	REDIS_SSL_CERTFILE
--snap-dir	Directory to store snapshot files	snap	-
--dry-run	Simulate operations without changes	False	-
--scan-count	Keys to fetch per SCAN iteration	1000	-
--batch-size	Keys to process per pipeline batch	1000	-

Important Notes:

• Either --export or --import must be specified (mutually exclusive)
• --snapshot-file is optional for import; latest snapshot will be used if omitted
• Use environment variables for sensitive data like passwords
• TLS options are required only for secure Redis instances
• Adjust --scan-count and --batch-size for performance tuning with large datasets

---

Examples

1. Export Snapshot

python3 migrator.py --export --host localhost --port 6379

Output:

Connecting to Redis at localhost:6379 (db=0)
Successfully connected to Redis server
Starting snapshot export to snap/snapshot_20260113_143800.db
Exporting snapshot: 100%|██████████| 50000/50000 [00:50<00:00, 1000.00keys/s]
Snapshot successfully exported to snap/snapshot_20260113_143800.db

2. Dry Run Export

python3 migrator.py --export --host localhost --port 6379 --dry-run

Simulates exporting without writing to the filesystem.

3. Import Snapshot

python3 migrator.py --import --host localhost --port 6380

Output:

Connecting to Redis at localhost:6380 (db=0)
Successfully connected to Redis server
Starting snapshot import from snap/snapshot_20260113_143800.db
⚠️  Clearing existing Redis data before import (FLUSHDB)
Importing snapshot: 100%|██████████| 50000/50000 [00:50<00:00, 1000.00keys/s]
Snapshot successfully imported from snap/snapshot_20260113_143800.db

4. Import with Environment Variables

export REDIS_HOST=production-redis.example.com
export REDIS_PORT=6379
export REDIS_PASSWORD=your-secure-password
export REDIS_DB=1

python3 migrator.py --import --snapshot-file snap/snapshot_20260113_143800.db

5. Secure Import with TLS

python3 migrator.py --import --snapshot-file snap/snapshot_20260113_143800.db \
  --host secure-redis.example.com --port 6379 \
  --ssl --ssl-ca-certs ca.pem --password mypassword

6. Performance Tuning for Large Datasets

python3 migrator.py --export --host localhost --port 6379 \
  --scan-count 5000 --batch-size 5000

---

⚠️ Important Warnings

Data Loss Risk

The --import operation executes FLUSHDB which DELETES ALL EXISTING DATA in the target Redis database before importing. Always:

• Backup your target database before importing
• Verify you're connecting to the correct Redis instance
• Test with --dry-run first
• Use different --db numbers to avoid overwriting production data

TTL Behavior

• Keys with TTL (time-to-live) are preserved during migration
• Expired keys are not exported (Redis automatically removes them)
• TTL countdown continues from the snapshot time, not reset

Snapshot File Format

• Snapshot files use newline-delimited JSON format
• Each line contains: {"key": "...", "type": "...", "ttl": ..., "value": "..."}
• Values are hex-encoded DUMP output for all data types
• Manual editing may corrupt the snapshot

---

📋 Logging

Console Output

• Connection status and major operations
• Progress bars for export/import
• Warnings and errors
• Final operation results

Log Files

Located in logs/redis-snapper_<timestamp>.log with detailed information:

• Every key-value operation (batched logging for performance)
• Redis command executions
• Error stack traces
• Performance metrics

Example log entry:

2026-01-13 14:38:00,123 - INFO - Connecting to Redis at localhost:6379 (db=0)
2026-01-13 14:38:00,124 - INFO - Successfully connected to Redis server
2026-01-13 14:38:00,125 - DEBUG - Exporting batch of 1000 keys

---

🧪 Testing

Setting Up a Test Environment

1. Run Redis in Docker

# Source Redis (port 6379)
docker run -d --name redis-source -p 6379:6379 redis:latest

# Target Redis (port 6380)
docker run -d --name redis-target -p 6380:6379 redis:latest

2. Load Sample Data

Use the provided script to populate Redis with test data:

chmod +x load_sample_data.sh
./load_sample_data.sh

This creates 50,000 key-value pairs (key1:value1 to key50000:value50000).

3. Export Test

python3 migrator.py --export --host localhost --port 6379

Verify the snapshot file exists:

ls -lh snap/
# Should show: snapshot_YYYYMMDD_HHMMSS.db

4. Import Test

python3 migrator.py --import --host localhost --port 6380

5. Verify Migration

# Check key count in target
docker exec redis-target redis-cli DBSIZE

# Sample random keys
docker exec redis-target redis-cli KEYS "key*" | head -10

# Verify specific key
docker exec redis-target redis-cli GET key42
# Should return: value42

Cleanup

docker stop redis-source redis-target
docker rm redis-source redis-target

---

🐛 Troubleshooting

Connection Errors

Problem: Failed to connect to Redis server: Connection refused

Solutions:

• Verify Redis is running: redis-cli ping
• Check host and port: netstat -an | grep 6379
• Check firewall rules
• Verify credentials with: redis-cli -h HOST -p PORT -a PASSWORD ping

Authentication Errors

Problem: Authentication failed: Invalid password

Solutions:

• Verify password: redis-cli -a YOUR_PASSWORD ping
• Use environment variable: export REDIS_PASSWORD=your-password
• Check Redis configuration: requirepass in redis.conf

Permission Errors

Problem: Permission denied: Cannot write to snapshot file

Solutions:

# Fix directory permissions
chmod -R u+rw snap/ logs/

# Or change ownership
sudo chown -R $USER:$USER snap/ logs/

Import Issues

Problem: Import may be incomplete: imported X of Y keys

Causes:

• Invalid JSON lines in snapshot file
• Corrupted data
• Memory limits on target Redis

Solutions:

• Check log file for specific errors
• Validate snapshot file: jq empty snap/snapshot_*.db
• Increase target Redis maxmemory

Performance Issues

Problem: Export/import is slow

Solutions:

# Increase batch sizes for larger datasets
python3 migrator.py --export --scan-count 5000 --batch-size 5000

# Use faster hardware or SSD storage
# Reduce logging verbosity (edit setup_logging in migrator.py)

---

🔒 Security Best Practices

1. Use Environment Variables for sensitive data:

   export REDIS_PASSWORD=your-secure-password
   export REDIS_HOST=production.redis.example.com

2. Enable TLS for production environments:

   python3 migrator.py --export --ssl --ssl-ca-certs /path/to/ca.pem

3. Protect Snapshot Files - they contain your data:

   chmod 600 snap/*.db

4. Use Separate Databases for testing:

   # Production uses db 0, testing uses db 1
   python3 migrator.py --import --db 1 --dry-run

5. Never commit credentials to version control

---

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for detailed guidelines.

Quick Start

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Make your changes
4. Test thoroughly
5. Commit: git commit -m "Add amazing feature"
6. Push: git push origin feature/amazing-feature
7. Open a Pull Request

Development Setup

# Clone your fork
git clone https://github.com/siyamsarker/redis-snapper.git
cd redis-snapper

# Install dependencies
pip install -r requirements.txt

# Run tests
./load_sample_data.sh
python3 migrator.py --export --dry-run

---

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🌟 Roadmap

• Add compression support for snapshot files (gzip, zstd)
• Implement incremental snapshots
• Add unit tests with pytest
• Support for Redis Cluster
• Web UI for management
• Snapshot encryption
• Multi-threaded export/import

---

🙏 Acknowledgments

• Dependencies:

  • redis-py - Redis client for Python
  • tqdm - Progress bar library

• Inspiration: Built to solve real-world Redis migration challenges in production environments

---

📞 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: This README and inline code comments

---

Made with ❤️ for the Redis community
Report Bug • Request Feature • Contribute