Cut through text at the speed of light
The fastest text chunking library for RAG applications. Available for both Rust and Python.
kiru is a high-performance text chunking library designed for modern RAG (Retrieval-Augmented Generation) systems. When you need to split millions of documents for vector databases or process streaming data in real-time, kiru delivers unmatched speed without sacrificing correctness.
- ⚡ Blazing Fast (Python): 1000+ MB/s throughput for bytes, 300+ MB/s for characters
- 🎯 UTF-8 Safe: Never breaks multi-byte characters or emoji
- 💾 Memory Efficient: Stream gigabyte files with constant memory usage
- 🚀 Parallel Processing: Utilize all CPU cores automatically
- 🔌 Multiple Sources: Files, URLs, strings, and glob patterns
- 🛠️ Flexible Strategies: Chunk by bytes or characters
- 🦀 Rust Core: Rust performance and memory safety
- 🐍 Python Bindings: Pythonic API for ease of use
Benchmarked on 1MB text file, 1MB chunks, 1KB overlap:
| Implementation | Strategy | Source | Time (ms) | Memory (MB) | Throughput (MB/s) |
|---|---|---|---|---|---|
| kiru (Rust) | bytes | string | 0.23 | - | 4,370 |
| kiru (Python) | bytes | string | 0.71 | 2.9 | 1,408 |
| kiru (Python) | chars | string | 3.13 | 2.9 | 319 |
| LangChain | chars | string | 2,982 | 18.6 | 0.34 |
kiru is 4,000x faster than LangChain for byte chunking and 940x faster for character chunking!
Key insights:
- Rust native performance: Up to 4,370 MB/s for byte chunking
- Python bindings overhead: Still 1,400+ MB/s, beating all pure Python alternatives
- Character-aware chunking: 300+ MB/s while respecting grapheme boundaries
- Memory efficient: Uses 6x less memory than LangChain
pip install kirufrom kiru import Chunker
# Create a chunker
chunker = Chunker.by_bytes(
chunk_size=1024, # 1KB chunks
overlap=128 # 128 bytes overlap
)
# Chunk text
chunks = chunker.on_string("Your text here...").all()
# Chunk files in parallel
sources = ["file://doc1.txt", "https://example.com/page", "glob://*.md"]
for chunk in chunker.on_sources_par(sources):
process(chunk)Add to your Cargo.toml:
[dependencies]
kiru = "0.1"use kiru::{BytesChunker, Chunker};
// Create a chunker
let chunker = BytesChunker::new(1024, 128)?;
// Chunk text
let chunks: Vec<String> = chunker
.chunk_string("Your text here...".to_string())
.collect();
// Stream large files
use kiru::{Source, StreamType};
let stream = StreamType::from_source(&Source::File("huge.txt".to_string()))?;
for chunk in chunker.chunk_stream(stream) {
process(chunk);
}# Perfect for vector database ingestion
chunker = Chunker.by_bytes(512, 50) # Tuned for embedding models
documents = ["glob://knowledge_base/**/*.md"]
chunks = chunker.on_sources_par(documents, channel_size=10000)
for chunk in chunks:
embedding = model.encode(chunk)
vector_db.insert(chunk, embedding)# Stream processing without memory overhead
for chunk in chunker.on_file("10GB_file.txt"):
# Each chunk generated on-demand
send_to_queue(chunk)// Process hundreds of documents concurrently
use kiru::{ChunkerBuilder, ChunkerEnum};
let chunker = ChunkerBuilder::by_bytes(ChunkerEnum::Bytes {
chunk_size: 4096,
overlap: 512,
});
let sources = vec!["glob://docs/**/*.txt"];
let chunks = chunker.on_sources_par_stream(sources, 1000)?;- Splits on byte boundaries while respecting UTF-8
- Fastest performance (1000+ MB/s in Rust, 1400+ MB/s in Python)
- Ideal for token-limited models and consistent memory usage
- Splits on character (grapheme) boundaries
- Ensures exact character counts regardless of byte representation
- Perfect for character-limited APIs (300+ MB/s in Python)
from kiru import Chunker
# Byte-based chunking
chunker = Chunker.by_bytes(chunk_size=1024, overlap=128)
# Character-based chunking
chunker = Chunker.by_characters(chunk_size=1000, overlap=100)# Single string
chunks = chunker.on_string("text...").all()
# Single file
chunks = chunker.on_file("/path/to/file.txt").all()
# HTTP/HTTPS URL
chunks = chunker.on_http("https://example.com/page").all()
# Multiple sources (serial)
sources = ["file://doc1.txt", "https://example.com/page", "glob://*.md"]
chunks = chunker.on_sources(sources).all()
# Multiple sources (parallel)
chunks = chunker.on_sources_par(sources, channel_size=1000).all()
# Or iterate lazily
for chunk in chunker.on_sources_par(sources):
process(chunk)file://path/to/file.txt- Local fileshttp://example.comorhttps://example.com- URLstext://Inline text content- Raw text stringsglob://*.md- Glob patterns- No prefix - Treated as raw text
use kiru::{BytesChunker, CharactersChunker, Chunker};
// Byte-based chunking
let chunker = BytesChunker::new(1024, 128)?;
// Character-based chunking
let chunker = CharactersChunker::new(1000, 100)?;use kiru::Chunker;
// Chunk a string
let chunks: Vec<String> = chunker
.chunk_string("Your text here".to_string())
.collect();
// Stream a file
use kiru::{Source, StreamType};
let stream = StreamType::from_source(&Source::File("file.txt".to_string()))?;
for chunk in chunker.chunk_stream(stream) {
// Process chunk
}use kiru::{ChunkerBuilder, ChunkerEnum, Source, HigherOrderSource, SourceGenerator};
// Create chunker with builder pattern
let chunker = ChunkerBuilder::by_bytes(ChunkerEnum::Bytes {
chunk_size: 4096,
overlap: 512,
});
// Single source
let chunks = chunker.on_source(Source::File("doc.txt".to_string()))?;
// Multiple sources (serial)
let sources = vec![
Source::File("doc1.txt".to_string()),
Source::Http("https://example.com".to_string()),
];
let chunks = chunker.on_sources(sources)?;
// Multiple sources (parallel) - returns Vec
let chunks: Vec<String> = chunker.on_sources_par(sources)?;
// Multiple sources (parallel streaming) - returns iterator
let chunks = chunker.on_sources_par_stream(sources, 1000)?;
for chunk in chunks {
// Process as they arrive
}
// Using glob patterns
let sources = vec![HigherOrderSource::SourceGenerator(
SourceGenerator::Glob("**/*.md".to_string())
)];
let flattened = HigherOrderSource::into_flattened_sources(sources)?;┌─────────────────────────────────────────┐
│ Application Layer │
│ (Python or Rust Application) │
├─────────────────────────────────────────┤
│ kiru-py (PyO3 Bindings) │
│ [Python only] │
├─────────────────────────────────────────┤
│ kiru-core (Rust Library) │
│ │
│ ┌──────────┬───────────┐ │
│ │ Chunkers │ Streaming │ │
│ │ Engine │ Engine │ │
│ └──────────┴───────────┘ │
└─────────────────────────────────────────┘
kiru/
├── README.md # This file (shared documentation)
├── kiru-core/ # Rust implementation
│ ├── src/ # Core chunking algorithms
│ │ ├── bytes_chunker.rs
│ │ ├── characters_chunker.rs
│ │ ├── chunker.rs # Builder pattern & parallel processing
│ │ └── stream.rs # File/HTTP streaming
│ ├── benches/ # Criterion benchmarks
│ └── tests/ # Property-based tests
├── kiru-py/ # Python bindings (PyO3)
│ ├── src/lib.rs # Python wrapper
│ └── python/ # Python tests & benchmarks
└── utils/ # Version management scripts
kiru's killer feature: true streaming with constant memory usage.
Unlike traditional chunkers that load entire files into memory, kiru processes data as it arrives using an intelligent buffering system. This means you can chunk gigabyte-sized files with minimal RAM usage.
File/HTTP Source → Read Blocks (8KB) → UTF-8 Buffer → Chunk Iterator → Your Code
↓ ↓
As needed Constant size
Key advantages:
- Constant Memory: Process 10GB files with ~10MB RAM
- Immediate Results: First chunks available instantly, no waiting for full file load
- Works Everywhere: Local files, HTTP/HTTPS streams, any data source
- UTF-8 Safe: Buffer maintains character boundaries automatically
from kiru import Chunker
chunker = Chunker.by_bytes(chunk_size=4096, overlap=512)
# ⚡ Stream a 10GB file - uses only ~10MB RAM
for chunk in chunker.on_file("huge_dataset.txt"):
# Process chunk immediately as it arrives
vector_db.insert(chunk)
# No waiting, no memory explosion!
# ⚡ Stream from HTTP - process as data downloads
for chunk in chunker.on_http("https://example.com/large_document.txt"):
process(chunk)
# Chunks ready while download continues
# ⚡ Stream multiple sources in parallel
sources = [
"file://10gb_file1.txt",
"https://example.com/doc.txt",
"file://10gb_file2.txt"
]
for chunk in chunker.on_sources_par(sources, channel_size=1000):
# All sources stream in parallel
# Memory stays constant regardless of file sizes
send_to_queue(chunk)use kiru::{BytesChunker, Chunker, Source, StreamType};
let chunker = BytesChunker::new(4096, 512)?;
// ⚡ Stream a massive file with constant memory
let stream = StreamType::from_source(&Source::File("10gb_file.txt".to_string()))?;
for chunk in chunker.chunk_stream(stream) {
// Process immediately, no memory buildup
vector_db.insert(chunk);
}
// ⚡ Stream from HTTP as data arrives
let stream = StreamType::from_source(&Source::Http("https://example.com/doc.txt".to_string()))?;
for chunk in chunker.chunk_stream(stream) {
process(chunk);
}Processing a 1GB file with 4KB chunks:
| Library | Memory Usage | Loads Full File? | Streaming? |
|---|---|---|---|
| kiru | ~10 MB | ❌ No | ✅ Yes |
| LangChain | 1000+ MB | ✅ Yes | ❌ No |
| tiktoken | 1000+ MB | ✅ Yes | ❌ No |
Result: kiru uses 100x less memory while being 4,000x faster!
# Clone repository
git clone https://github.com/yourusername/kiru.git
cd kiru
# Run all tests
cargo test --workspace
# Run Rust benchmarks
cd kiru-core
cargo bench
# Build Python package
cd ../kiru-py
pip install maturin
maturin develop --release
# Run Python tests
pip install pytest hypothesis
pytest python/test.py
# Run Python benchmarks
python python/bench.py# Rust benchmarks
cd kiru-core
cargo bench
# Python benchmarks
cd kiru-py
python python/bench.py- Use byte chunking for maximum throughput (1000+ MB/s)
- Use character chunking when exact character counts matter (300+ MB/s)
- Enable parallel processing with
on_sources_par()for multiple files - Tune chunk size based on your embedding model's context window
- Adjust overlap to balance context preservation and storage
- Stream large files to maintain constant memory usage
"Kiru" (切る) is Japanese for "to cut" - reflecting the library's purpose of cutting text into chunks at lightning speed ⚡🗡️
We welcome contributions! Please check out our Contributing Guide for guidelines.
MIT License - see LICENSE for details.
Built with:
- PyO3 - Rust bindings for Python
- Rayon - Data parallelism for Rust
- maturin - Build and publish Rust Python extensions
Ready to cut through text at the speed of light?
- 🐍 Python:
pip install kiru - 🦀 Rust: Add
kiru = "0.1"to Cargo.toml
Get started with PyPI | Crates.io | Documentation