First-pass ledger hashes in replayer checkpoint files #14390
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
!ci-build-me |
|
!ci-build-me |
psteckler
pushed a commit
that referenced
this pull request
Oct 19, 2023
|
!ci-build-me |
dkijania
approved these changes
Oct 30, 2023
georgeee
approved these changes
Nov 1, 2023
| incr count | ||
|
|
||
| let find ledger_hash = | ||
| Base.Hash_set.find hash_set ~f:(fun (hash, _n) -> |
There was a problem hiding this comment.
This is an O(n) operation, right?
If yes, I wonder why the hash set is used instead of simple list?
There was a problem hiding this comment.
Yes. Probably when I started on it, I thought we could do lookups on the hash alone.
The hash set is a mutable data structure, with the list we'd need to use a mutable reference. Not sure which is better.
|
!ci-build-me |
|
!approved-for-mainnet |
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The replayer checks that every new snarked ledger hash it sees is some first-pass ledger hash. If we run the replayer from a checkpoint file, a new snarked ledger hash might be equal to a first-pass ledger hash that was produced before the checkpoint.
Solution: save the pending first-pass ledger hashes to the checkpoint file, and populate the set of hashes from the saved hashes when the replayer starts. Also save the last snarked ledger hash, which is used to know whether to search the first-pass hashes.
The set of ledger hashes are indexed by order of occurrence. That indexing allows flushing older hashes when we find the first-pass hash that matches a snarked ledger hash. The ledger hashes are saved as a list whose order is given by the indexes. When starting the replayer, the set of hashes uses new indexes created when iterating over that list.
The
first_pass_ledger_hashesandlast_snarked_ledger_hashfields are optional in the input file, so compatibility with old checkpoint files is maintained. Confirmed that fact by running the replayer on an old checkpoint file.Closes #13732.