tests: enable wal reader fanout in tests #10301
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
VladLazar
added
the
run-benchmarks
7531 tests run: 7136 passed, 0 failed, 395 skipped (full report)Flaky tests (5)Postgres 17
Postgres 16
Postgres 14
Code coverage* (full report)
* collected from Rust tests only The comment gets automatically updated with the latest test results
bacb773 at 2025-01-24T11:04:02.837Z :recycle: |
|
Note to self: the only "real" failure here is |
VladLazar
force-pushed
the
vlad/sk-fanout
branch
from
ad551e6 to
34f6a71
Compare
VladLazar
force-pushed
the
vlad/sk-fanout-test-enable
branch
from
3fdf4f6 to
0b1eb70
Compare
VladLazar
force-pushed
the
vlad/sk-fanout
branch
from
34f6a71 to
d0c47d8
Compare
VladLazar
force-pushed
the
vlad/sk-fanout-test-enable
branch
from
0b1eb70 to
abc511d
Compare
VladLazar
force-pushed
the
vlad/sk-fanout-test-enable
branch
from
abc511d to
aa5e83e
Compare
VladLazar
force-pushed
the
vlad/sk-fanout
branch
from
1f40ce8 to
98b57a5
Compare
VladLazar
force-pushed
the
vlad/sk-fanout-test-enable
branch
from
aa5e83e to
9dc10b8
Compare
github-merge-queue bot
pushed a commit
that referenced
this pull request
Jan 15, 2025
## Problem Safekeepers currently decode and interpret WAL for each shard separately. This is wasteful in terms of CPU memory usage - we've seen this in profiles. ## Summary of changes Fan-out interpreted WAL to multiple shards. The basic is that wal decoding and interpretation happens in a separate tokio task and senders attach to it. Senders only receive batches concerning their shard and only past the Lsn they've last seen. Fan-out is gated behind the `wal_reader_fanout` safekeeper flag (disabled by default for now). When fan-out is enabled, it might be desirable to control the absolute delta between the current position and a new shard's desired position (i.e. how far behind or ahead a shard may be). `max_delta_for_fanout` is a new optional safekeeper flag which dictates whether to create a new WAL reader or attach to the existing one. By default, this behaviour is disabled. Let's consider enabling it if we spot the need for it in the field. ## Testing Tests passed [here](#10301) with wal reader fanout enabled as of 34f6a71. Related: #9337 Epic: #9329
VladLazar
force-pushed
the
vlad/sk-fanout-test-enable
branch
from
9dc10b8 to
2fd1f51
Compare
VladLazar
force-pushed
the
vlad/sk-fanout-test-enable
branch
from
2fd1f51 to
bacb773
Compare
erikgrinaker
approved these changes
Jan 23, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note: this has to merge after the release is cut on
2025-01-17for compat tests to start passing.Problem
SK wal reader fan-out is not enabled in tests by default.
Summary of changes
Enable it.