safekeeper: decode and interpret for multiple shards in one go #10201
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
7293 tests run: 6928 passed, 0 failed, 365 skipped (full report)Code coverage* (full report)
* collected from Rust tests only The comment gets automatically updated with the latest test results
4512aaa at 2025-01-13T13:59:11.390Z :recycle: |
erikgrinaker
approved these changes
Dec 19, 2024
Yeah, I felt quite naughty writing this stuff. I'll dust up my benchmark for this stuff to see how much we need to optimise here. |
VladLazar
force-pushed
the
vlad/fan-out-wal-decoder
branch
from
8a04783 to
5fbec70
Compare
Currently, we call `InterpretedWalRecord::from_bytes_filtered` from each shard. To serve multiple shards at the same time, the API needs to allow for enquiring about multiple shards. This commit tweaks it a pretty brute force way. Naively, we could just generate the shard for a key, but pre and post split shards may be subscribed at the same time, so doing it efficiently is more complex.
It cannot be changed dynamically. For the stripe size to change, the shard count needs to change too.
decode-interpret-wal/unsharded
time: [439.01 ms 439.23 ms 439.48 ms]
thrpt: [291.25 MiB/s 291.42 MiB/s 291.56 MiB/s]
decode-interpret-wal/8/8-shards
time: [934.73 ms 935.45 ms 936.41 ms]
thrpt: [136.69 MiB/s 136.83 MiB/s 136.94 MiB/s]
decode-interpret-wal/4/8-shards
time: [646.55 ms 646.72 ms 646.89 ms]
thrpt: [197.87 MiB/s 197.92 MiB/s 197.97 MiB/s]
decode-interpret-wal/2/8-shards
time: [487.38 ms 487.56 ms 487.76 ms]
thrpt: [262.43 MiB/s 262.53 MiB/s 262.63 MiB/s]
VladLazar
force-pushed
the
vlad/fan-out-wal-decoder
branch
from
22d6e91 to
8e96693
Compare
erikgrinaker
approved these changes
Jan 10, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Currently, we call
InterpretedWalRecord::from_bytes_filteredfrom each shard. To serve multiple shards at the same time,
the API needs to allow for enquiring about multiple shards.
Summary of changes
This commit tweaks it a pretty brute force way. Naively, we could
just generate the shard for a key, but pre and post split shards
may be subscribed at the same time, so doing it efficiently is more
complex.