A high performance sparse file webserver written in Rust
We needed something to serve large, random files fast for a speedtest service. In the past we had prepared files on the filesystem. That was slow and took up a lot of space. When deploying an anycasted speedtest service we didn't want to waste the space on 8 machines - so this project was born.
Some proxies compress or cache the transmitted data. We want to test the end to end speed. That's why we're using a random number generator.
The data is generated using the extremely fast SmallRng of the rand crate.
In our setup running in VMs on AMD EPYC 7302P processors @ 3.3 GHz we achieve ~20 Gbit/s per stream. We're sure further optimization is possible, but this is okay for us, as our machines only have 25 Gbit/s interfaces anyways. Feel free to contribute optimizations.
If you are using the Nix package manager, there is a flake file included. ❄️
Building manually is straightforward using Cargo:
cargo build --release
Tune your kernel network stack:
# Increase network buffer size
net.core.rmem_max=134217728
net.core.wmem_max=134217728
# Increase TCP buffer size
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
# Enable BBR TCP scheduler
net.core.default_qdisc=fq
net.ipv4.tcp_congestion_control=bbr
Consider increasing your NICs buffer sizes and the amount of queues. If you are running this in a VM, consider enabling Multi-Queue in your virtio-net
If you are using nginx as a reverse proxy you want to consider a few settings:
- Set caching and last-modified headers
- Turn off gzip compression
- Increase
proxy_read_timeout - Increase
proxy_bufferssize
That's what our config looks like:
add_header Cache-Control 'no-store, no-cache, max-age=0, no-transform';
add_header Last-Modified $date_gmt;
if_modified_since off;
expires off;
etag off;
gzip off;
proxy_read_timeout 999;
proxy_buffers 16 128k;