Make bridge nodes recoverable #193
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kcalvinalvin
force-pushed
the
2024-07-03-recoverable-bridge
branch
2 times, most recently
from
ccf37bf to
993ed1a
Compare
For NodesBackend and CachedLeavesBackend, flush is exported to let the utreexo indexes to be able to call it directly.
For utreexo bridges, the accumulator state was kept in separate leveldb instances for the nodes and the leaves. This creates issues during recovery because it's hard to ensure that the writes will be atomic. Using the same leveldb backend for both the nodes and the leaves solves this problem.
On delete, cached leaves backend would delete directly from the database. This behavior is changed and now the deletes are cached until a flush happens. This is a step towards achieving a recoverable accumulator state as now only flushes are able to modify the database.
db close was handling both the flushes and the database closes. These are now separated out into 2 different functions.
backends These newly added functions ask for leveldb txs and now many writes and deletes can be atomically written into the database.
The flush functions weren't atomic which meant that in an unexpected crash the node would be left in an unrecoverable state. The atomic write is a step closer to achieving recoverable accumulator state for bridges.
The writes to the database now only happen through flushes. Since flushes are atomic, now all the writes to the database are atomic.
Memory splits between the nodes and cached leaves backend is updated to 70/30 after monitoring how the memory is used during ibd.
The overflow map allows for entries to be added that exceed the amount that the map slice was originally allocated for. This trades off memory usage guarantees with the ability to not flush in the middle of modifying the accumulator.
The overflow map allows for entries to be added that exceed the amount that the map slice was originally allocated for. This trades off memory usage guarantees with the ability to not flush in the middle of modifying the accumulator.
UtreexoState flush used to flush and close the database. Now the closing of the database is separated from the flushing so that the flush function can be used elsewhere besides when the node is shutting down.
The overflowed method allows for callers to check if the node maps and the cached leaves slice is overallocated.
Getting rid of flushes on put, get and delete. This is another step closer to making the accumulator state recoverable on crashes.
Getting rid of flushes on length and foreach. This is another step closer to making the accumulator state recoverable on crashes.
For cached leaves backend and nodes backend, the flush functions created their own leveldb transaction and wrote to the database. This meant that one couldn't guarantee atomicity when flushing the utreexo accumulator. Requiring the leveldb transaction allows for the caller to generate a single leveldb transaction and guarantee atomicity.
The caches used for utreexo indexes can now overflow. The newly added method allows for the caller to check if the caches have overflowed and thus require a flush.
FlushIfNeeded only flushes the utreexo indexes if the cache is full. It's called at the end of every block connect.
The added usage stats helps in monitoring how much each cache was utilized on flushes.
The best hash will be written along with the numleaves of the utreexo accumulator so that it'll be used to mark that the accumulator state on disk is at least consistent to that block.
Utreexo state db is exposed by including it in the UtreexoState during initialization. This allows for functions on UtreexoState to create leveldb transactions which allow for it to write to the database.
The functions that are passed into ForEach now are required to have errors returned and ForEach will return early if there's any errors. This is desirable as on flushes ForEach is called and thus can return early with an error on flushes.
It'll never really be called but not having it risk runtime panics. Just adding it in case it'll be called with later code changes.
Flushes now return errors and ask for leveldb.Transaction from the caller. This is so that the caller can write the best block hash and the utreexo state numleaves on flushes to mark that the state is consistent up until that point.
When the utreexo state is flushed to disk, the best block hash and the numleaves are also written. This is all done in one leveldb transaction and so it's guaranteed to be atomic. The best block hash that's written allows for callers to check if the utreexo state is consistent or not with the blockindex.
Pruned and max memory is moved to the utreexo config and init functions now take in just the utreexo config as all the needed information is stored in the config.
In order to support checking if the utreexo state is consistent, we need the best block hash, which can only be attained when the chain is initialized. The current initialization for the utreexo state was being done when the index was created. Moving the initialization to when the chain is being initialized allows us to pass in the best block hash.
This change requires the indexers to take in the tip hash and height as an argument to Init(). With this, utreexo indexes are able to see what tip they're synced to. Since the accumulator isn't written to disk every block, this is useful information to check which block each utreexo index should sync to.
Always flush the utreexo state on block disconnects. This allows the utreexo state to always stay at a recoverable state. Even if the ffldb fails to write, we can always re-connect blocks and reorganize. However, if the utreexo state isn't flushed, we can't disconnect on restart as the data could have been deleted from ffldb.
Flush allows a caller to flush the cached internals of the database. This is useful for keeping the two separate databases (ffldb and utreexo state db) in sync.
Utreexo proof indexes used to use the same amount of memory as the utxocache but since the utreexo proof index's entire size is basically 2 times the utxo cache, it makes sense to increase it by that much to minimize the db fetches.
The utreexo state may be behind the index tip if the node had an unexpected shutdown. Calling initConsistentState syncs up the utreexo state so that it stays consistent with the indexer tip.
The main database has a cache where the data is written to and this must be flushed as well in order for the utreexo state to be recoverable. Allowing access to the main db flush let's us keep the utreexo state consistent.
Exported so that the utreexo state can use the same variable for flushing the utreexo state.
The new flushes are able to support different modes which flush the utreexo state when it meets various conditions. The flushes that were also called on ConnectBlocks are removed so that they can be controlled by outside callers.
CloseUtreexoState asked for the best hash because the best hash fetched when catching up the index was different from the one you get from the best snapshot. This is solved by not calling CloseUtreexoState but calling the internal methods directly. This simplifies calling CloseUtreexoState for the callers.
Flush is added to the indexer interface so that we can call the flush from when connecting the block in the blockchain package. This allows the utreexo indexes to also flush the main database. They couldn't when being called from the ConnectBlock on the indexers as that acquires the database tx lock.
There are two quit on interrupts when indexers are catching up. The first flush was problematic in that the flush would save the blockhash of a block that wasn't processed. Getting rid of this first flush solves this problem.
The older utreexo states used to save the numLeaves to the flatfiles. We read the numLeaves from the flatfile and save it to the database.
FlushIndexes method is added to blockchain so that external callers can trigger flushes on the indexes. It's useful for periodic flushes were the node is already caught up to the tip but should be flushed to keep the node from being too far behind if there were to be an unexpected shutdown.
The flatfile states were not recoverable if the node was suddenly crashed if data were being written to it. These recovery methods recover the flatfile state to the latest readable data on restarts.
tip The index tip may be behind what's saved in the flatfile state as the main database has a cache but the flatfile states do not. After an unexpected crash, the on recovery the flatfile states are now disconnected to match the index tip height to keep the indexer consistent.
For the utreexo state to be recoverable on unexpected crashes, there must be blocks available for it to reindex on crashes. If there aren't, the utreexo state is irrecoverable. To prevent this from happening, we check what the last stored block on the disk is after a prune and flush the utreexo state if the last flush happened before the last kept block.
kcalvinalvin
force-pushed
the
2024-07-03-recoverable-bridge
branch
from
993ed1a to
a3b68d4
Compare
When mapping to a uint64, it wasn't possible to mark if the positions were fresh or not. If a position could be marked fresh, then we can delete it completely from the memory without it having to touch the disk. This change comes from observing that there's a lot of slowdowns coming from flushing.
For NodesBackend, the cached entry is not removed from the cache even if it's being deleted as subsiquent calls would be made to fetch the key. Caching it as removed saves on disk reads and on flushes, the leaf is not attempted to be flushed if it's marked as fresh.
kcalvinalvin
force-pushed
the
2024-07-03-recoverable-bridge
branch
from
f097be2 to
9f74eaf
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When running either the
utreexoproofindexor theflatutreexoproofindexthey will not be able to recover in cases of unexpected shutdown. This forces the user to reindex which is very costly.The changes here make it so that the utreexo proof indexes are able to recover even in cases of unexpected shutdowns.