Jp/perf 2.9 2025013 #689
Merged
Jp/perf 2.9 2025013 #689
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JamesPiechota
changed the base branch from
master
to
release/N.2.9.0-early-adopter
ldmberman
force-pushed
the
release/N.2.9.0-early-adopter
branch
from
cd97224 to
324923a
Compare
JamesPiechota
force-pushed
the
jp/perf-2.9-2025013
branch
4 times, most recently
from
9906730 to
89d5b1f
Compare
JamesPiechota
force-pushed
the
jp/perf-2.9-2025013
branch
from
43982fe to
e51e384
Compare
ar_device_lock ensures that only one storage module on a device is preparing entropy at a time, and ensures that no other storage modules on that device are syncing data.
- entropy is now written in a separate thread from entropy generation this allows *some* parallelization since entropy generation is usually faster than storing. We now start generatng the next batch of entropy while the previous batch is being written - slight optimization for ar_storage_module:get_all_packed. We now cache each module and its covered range once, rather than building it each time. Since this is part of a loop that gets called for each chunk stored (potentially thousands of times a seconds), it makes a measurable difference in performance. - ensure only one thread writes all chunks in a given storage module. This avoid the disk thrash that used to occur when writing many different chunks concurrently. - Only emit the add_range event for the ar_data_sync sync_record. This was the only event that was handled by any clients, however many more events were previously being emitted. Even the null-handling of the events was sometimes enough to impact performace.
JamesPiechota
force-pushed
the
jp/perf-2.9-2025013
branch
from
e51e384 to
ecb4b7e
Compare
JamesPiechota
force-pushed
the
jp/perf-2.9-2025013
branch
from
b67b30a to
5611fef
Compare
JamesPiechota
requested review from
ldmberman
and removed request for
ldmberman
1. When writing entropy to the overlap zone between storage modules sometimes entropy could be written over chunk data. Fix is not to write to neighboring storage modules and instead waste a bit of entropy. Wasted entropy is about 0.07% of all entropy generated per partition. 2. When repacking in place make sure we wait for entropy to be written whenever moving to a new slice index. This is primarily an issue during tests where there are only 3 chunks per sector, but could hypothetically be an issue in production. 3. Remove all code related to sub-chunk iteration since we don't need it and it may impact performance.
This reverts commit e48607f.
JamesPiechota
force-pushed
the
jp/perf-2.9-2025013
branch
from
2905679 to
4066145
Compare
JamesPiechota
force-pushed
the
jp/perf-2.9-2025013
branch
from
42a3dc8 to
cec9db1
Compare
this reduces the disk thrash that can occur when doing a cross-module repack
JamesPiechota
force-pushed
the
jp/perf-2.9-2025013
branch
from
cec9db1 to
42890af
Compare
JamesPiechota
merged commit 174c73d
into
release/N.2.9.0-early-adopter
72 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.