Experimental: Proof of concept of parallel overlay root calculation #170
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Cherry-picks: Mmap page manager only flushes dirty pages
Collaborator
✅ Heimdall Review Status
|
kmchicoine
approved these changes
Nov 14, 2025
Comment on lines
+121
to
+127
| // SAFETY: OverlayState is thread-safe because: | ||
| // - Arc<[...]> provides thread-safe reference counting | ||
| // - All fields are immutable after construction | ||
| // - No interior mutability is used | ||
| // - All operations are read-only or create new instances | ||
| unsafe impl Send for OverlayState {} | ||
| unsafe impl Sync for OverlayState {} |
Contributor
There was a problem hiding this comment.
Is there any reason we can't derive Send+Sync instead of manual impl?
Comment on lines
+159
to
+160
| // Ignore send errors - the channel may be closed during shutdown | ||
| sender.send(work_item).unwrap(); |
Contributor
There was a problem hiding this comment.
Is it ok if this panics here (from calling unwrap on a send error, even if we want to ignore the send error)?
Comment on lines
+102
to
+104
| let end_idx = if i == thread_count { | ||
| // Last thread handles any remaining addresses | ||
| addresses.len() |
Contributor
There was a problem hiding this comment.
Should this be i == thread_count - 1? Since i is zero-indexed, wouldn't it always miss this case?
Contributor
|
General question: what happens if one of the hash builder threads panics? I think |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This updates the overlay root calculation code to compute partial roots for a given trie using an internal thread pool, sharded by first nibble, and then combines these together into a single root.
Uses a fork of alloy-trie with the ability to compute partial roots from a
HashBuilderand compute the merged root from an array ofHashBuilders.