Use chunked streaming for ISO file serving to reduce memory #102
+175
−34
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
serve_iso_file() was using ReaderStream which could buffer large amounts of data in memory. For a 2.6GB ISO file, memory usage peaked at 1.9GB. Now uses the same chunked streaming approach as stream_from_iso(): - 32MB chunks via spawn_blocking - Bounded mpsc channel (capacity 2) for backpressure - Max ~96MB in memory (3 chunks: 2 in channel + 1 being read) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Extract common file streaming logic into stream_file_to_channel() - Reuse helper in stream_iso_file() and stream_initrd_with_firmware() - Add test_stream_file_to_channel() for the helper function - Add test_stream_iso_file() to verify chunked streaming works - Add test_stream_iso_file_not_found() for error handling Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Simplify stream_file_to_channel return type to Result<(), Error> - Replace offset tracking with bytes_remaining (cleaner since read_exact advances position) - Add test_stream_file_to_channel_multiple_chunks with 70MB file (3 chunks) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
/iso/ubuntu-22.04/ubuntu-22.04.5-live-server-amd64.iso)serve_iso_file()was usingReaderStreamwhich buffered ~1.9GB for a 2.6GB ISOstream_from_iso())Test plan
watch -n1 'ps -o rss= -p $(pgrep serabutd)'cargo test- all 62 tests pass🤖 Generated with Claude Code