A robust and extensible file-based storage system built on GitHub repositories. Supports large file handling with optional compression, encryption, chunking, and version control using commit history. Includes a powerful CLI for ease of use.
- ✅ Upload, update, read, and delete files in a GitHub repository
- 📦 Automatic chunking for large files (>50MB)
- 🗜️ Optional zlib compression
- 🔐 Optional AES encryption (CTR mode)
- 🕒 List version history via Git commit logs
- 🔄 Revert files to previous versions by commit SHA
- 🖥️ CLI for basic operations
pip install github-storagefrom github_storage import GitHubStorage
storage = GitHubStorage(
token="your_github_token",
repo_name="username/repo",
compression=True,
encryption_key=b"16_or_24_or_32_byte_key"
)
# Upload file
storage.store_file("data/file.txt", "local/path/file.txt")
# Read file
data = storage.read_file("data/file.txt")
# Delete file
storage.delete_file("data/file.txt")
# List versions
versions = storage.list_versions("data/file.txt")
# Revert to specific version
storage.revert_to_version("data/file.txt", sha="commit_sha")github_storage. --token <GH_TOKEN> --repo <username/repo> [options] <command>| Flag | Description |
|---|---|
--token |
Your GitHub Personal Access Token |
--repo |
Target GitHub repo (e.g. user/repo) |
--branch |
Target branch (default: main) |
--compression |
Enable compression (zlib) |
--encrypt-key |
AES encryption key (hex-encoded string) |
Upload a local file to GitHub.
github_storage. --token TOKEN --repo user/repo store-file <repo_path> <local_file_path> [--msg "commit message"]Store a string as bytes in GitHub.
github_storage --token TOKEN --repo user/repo store-bytes <repo_path> <content_string>Read a file and print or save locally.
github_storage --token TOKEN --repo user/repo read-file <repo_path> [--save-to local_path]Delete a file or chunked file.
github_storage. --token TOKEN --repo user/repo delete-file <repo_path>List recent commit history for a path.
github_storage --token TOKEN --repo user/repo list-versions <repo_path> [--limit 5]Revert a file to a previous version using commit SHA.
github_storage --token TOKEN --repo user/repo revert <repo_path> <commit_sha>- Encryption: AES in CTR mode; use 16, 24, or 32 byte key (128, 192, 256 bits).
- Compression: zlib compression; efficient for text or repetitive binary data.
- Chunking: Enabled automatically for files larger than
chunk_size(default: 50MB).
- Files larger than
chunk_sizeare split into parts and stored under a.chunksfolder in the repo. - A
manifest.txtfile is added to describe the chunked structure. - Old single files or chunk folders are automatically cleaned up.
- Max GitHub file size: 100MB per file (we use 95MB to be safe).
- Requires a repo with appropriate write permissions via GitHub token.
- File size and repo quota limits still apply.
- Python 3.6+
PyGithubpycryptodome
Install them with:
pip install PyGithub pycryptodomeMIT License
Pull requests, issues, and suggestions are welcome! Open an issue or fork the project to contribute.