safekeeper: fan out from single wal reader to multiple shards #10190
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
VladLazar
force-pushed
the
vlad/sk-fanout
branch
3 times, most recently
from
0ddecad to
fb0c47c
Compare
VladLazar
changed the title
7304 tests run: 6933 passed, 0 failed, 371 skipped (full report)Flaky tests (4)Postgres 17
Postgres 16
Postgres 15
Code coverage* (full report)
* collected from Rust tests only The comment gets automatically updated with the latest test results
98b57a5 at 2025-01-15T12:19:37.235Z :recycle: |
VladLazar
force-pushed
the
vlad/sk-fanout
branch
from
248050c to
7123d89
Compare
VladLazar
force-pushed
the
vlad/sk-fanout
branch
3 times, most recently
from
34377a1 to
f9956dc
Compare
VladLazar
force-pushed
the
vlad/sk-fanout
branch
2 times, most recently
from
ad551e6 to
34f6a71
Compare
VladLazar
force-pushed
the
vlad/sk-fanout
branch
from
34f6a71 to
d0c47d8
Compare
erikgrinaker
approved these changes
Jan 15, 2025
Each protocol gets its version of wal sender state. Future commits will add internal state here, so define an internal version of the type to be used within the safekeeper crate.
We want to suppport resetting the wal stream at arbitrary wal positions. This commit reworks the existing streaming wal reader to suppport this functionality.
The basic idea for wal reader fanout is that wal decoding and interpretation happens in a separate tokio task and senders attach to it. Senders only receive batches concerning their shard and only past the Lsn they've last seen. When the wal reading task is spawned, it returns a reference through which it can be controlled. This reference is shared by multiple wal senders. When all wal senders go out of scope, the reference goes out of scope and the task is aborted.
The same shard may be attached to two or more different pageservers, so the fan out reader needs to deal with this scenario too. Add a `SenderId` to differentiate between senders belonging to the same shard. Doesn't change the code flow much, but adds some extra complexity.
VladLazar
force-pushed
the
vlad/sk-fanout
branch
from
1f40ce8 to
98b57a5
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Safekeepers currently decode and interpret WAL for each shard separately.
This is wasteful in terms of CPU memory usage - we've seen this in profiles.
Summary of changes
Fan-out interpreted WAL to multiple shards.
The basic is that wal decoding and interpretation happens in a separate tokio task and senders
attach to it. Senders only receive batches concerning their shard and only past the Lsn they've last seen.
Fan-out is gated behind the
wal_reader_fanoutsafekeeper flag (disabled by default for now).When fan-out is enabled, it might be desirable to control the absolute delta between the
current position and a new shard's desired position (i.e. how far behind or ahead a shard may be).
max_delta_for_fanoutis a new optional safekeeper flag which dictates whether to create a newWAL reader or attach to the existing one. By default, this behaviour is disabled. Let's consider enabling
it if we spot the need for it in the field.
Testing
Tests passed here with wal reader fanout enabled
as of 34f6a71.
Related: #9337
Epic: #9329