rescan: use batch filter fetching #275
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ellemouton
force-pushed
the
rescanBatchFilterFetch
branch
from
9a49d03
to
a2e91f4
Compare
Roasbeef
requested changes
May 23, 2023
ellemouton
force-pushed
the
rescanBatchFilterFetch
branch
from
a2e91f4
to
9f4a9d9
Compare
ellemouton
force-pushed
the
rescanBatchFilterFetch
branch
from
9f4a9d9
to
a899564
Compare
ellemouton
force-pushed
the
rescanBatchFilterFetch
branch
from
a899564
to
df74c66
Compare
ellemouton
force-pushed
the
rescanBatchFilterFetch
branch
2 times, most recently
from
bca0cb2
to
833a492
Compare
|
@Roasbeef: review reminder |
ellemouton
force-pushed
the
rescanBatchFilterFetch
branch
from
833a492
to
0c22f0e
Compare
Introduce a new `rescanState` struct which holds the variables required for a rescan. We then also conver the existing `rescan` function into a method of `rescanState`. This refactor will assist in making future clean up of the `rescan` method easier.
Add a new `waitForBlocks` helper method to rescanState that will wait on block notifications until a given predicate returns true. Currently this method is only called with one predicate but an upcoming commit will make use of it using a differnet predicate.
This is required for the follow up commit where access to the IsCurrent method is required in the rescanState.rescan method which only has access to the ChainSource interface.
Let the rescan function wait until the filter headers have either caught up to the back end chain or until they have caught up to the specified rescan end block. This lets the rescan operation take advantage of doing batch filter fetching during rescan making the operation a lot faster since filters can be fetched in batches of 1000 instead of one at a time.
ellemouton
force-pushed
the
rescanBatchFilterFetch
branch
from
0c22f0e
to
b38fa7f
Compare
Roasbeef
approved these changes
Jul 25, 2023
| @@ -446,6 +446,36 @@ func (rs *rescanState) rescan() error { | |||
| return err | |||
| } | |||
|
|
|||
| // To ensure that we batch as many filter queries as possible, we also | |||
| // wait for the header chain to either be current or for it to at least | |||
| // have caught up with the specified end block. | |||
|
cc @ProofOfKeags for review request |
positiveblue
approved these changes
Jul 27, 2023
|
post merge ACK b38fa7f It took me a sec to realize most of the diff was just nesting some values under the state structure. Seems like the only major change was blocking on waiting for the header chain to get far enough ahead to maximize the benefits of batching. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With this PR, we ensure that
rescancan make use of batch filter fetching bywaiting until the header chain is either current or until it is ahead of the specified
end height before starting the scan. This greatly improves the speed of a rescan
because before this commit, the filters would be fetched one-by-one.
The first few commits of this PR are just refactor commits that clean-up the
rescanfunction and then the actual behaviour change is added in the final commit.
Fixes #68
Replaces #236
Performance Improvement
On current master branch, syncing 3200 filters on my local regtest network takes 4m25s
With this PR, syncing the same number of filters takes 653ms
LND CI
lightningnetwork/lnd#7788