multi: Rework sync model to use hdr annoucements. #2555

This modifies the progress logger to make it a bit more generic so that it can support logging header sync progress in the future too and modernizes the code while here.

This removes the flag from the SubmitBlock method of the sync manager since those flags are not really intended for general block submission which is why it is only ever called with blockchain.BFNone. Also, the intention is to modify the way the sync manager deals with block submission in the future which this change help simplify.

This removes the orphan return flag from the SubmitBlock and ProcessBlock netsync methods in favor of allowing the caller to detect it from the error. This is a step towards the goal of removing handling of orphan blocks altogether in favor of using block headers for sync.

This removes all orphan block handling from the syncing process and is part of the overall effort to move to full headers-first syncing as opposed to the current partial header-first checkpoint-based process. It should be noted that this change is made in such a way that the syncing process continues to work properly, however, it is only intended to be an intermediate step, and, as such, it does result in suboptimal behavior once it passes the final checkpoint. That will no longer be the case one the entire sync process conversion is complete.

This overhauls the primary code that deals with synchronizing the blockchain with other peers on the network to use a model based on header announcements instead of inventory vectors. Currently, all blocks are discovered via a combination of inventory vector announcements and the getblocks protocol message, both of which only include the hash of the blocks in question. This is not ideal because it means blocks are blindly downloaded based on those announcements without knowing if they're actually likely to be something that is useful since there is no further information such as what blocks they connect to. It also means that extra logic is needed to deal with orphan blocks (those whose parents are not yet known) such as caching them, determining known orphan roots, and expiration. In short, the current method generally ends up wasting bandwidth and also makes it much more difficult to detect certain classes of malicious behavior. The recently added capability of blockchain to process headers independently from blocks while the block data is added out of order later opened the door for a much better model that addresses all of the aforementioned issues as well as paves the way for many other related enhancements. The new model discovers and downloads all of the headers prior to any block data via the getheaders protocol message and then uses those headers to determine exactly which blocks need to be downloaded to reach the tip of the best chain. Notably, this means orphan blocks are now a thing of the past as blocks that do not connect are no longer downloaded under any circumstance. Further, the new model also makes use of sendheaders so that all block announcements are made via the associated block header. This in turn is used to better determine if an announced block is likely to be useful prior to downloading it. It should be noted that the changes herein are intentionally limited to those necessary to use the new sync model based on headers in an incremental fashion to help simplify review and make it easier to assert correctness. There are many more improvements that this model paves the way to support planned for future commits. For example, syncing from multiple peers in parallel and improved DoS protection. The following is a high-level overview of the key features: - All headers are downloaded and examined prior to downloading any blocks - The initial header sync process: - Detects and recovers from stalled/malicious peers - The concept of orphan blocks no longer exists - This means blocks which are not already known to connect are not downloaded - All block announcements are handled via header announcements - Detects and prevents malicious behavior related to orphan headers - The chain sync process: - Starts once the headers are downloaded and entails both downloading blocks as well as verifying them - Uses the headers to determine the best blocks to download - Pipelines the requests to increase throughput - Actively avoids downloading duplicate blocks - The sync height is dynamically updated as new headers are discovered

This modifies the progress logger to expose a new method for logging the progress of header syncing and adds tests to ensure proper functionality.

This modifies netsync to provide progress logging for the initial headers sync including a progress percentage. A new method is added that determines the progress of the headers sync based on an algorithm that calculates the expected number of headers there should be in the time interval since the current best known header and the current time given the target block time for the network. This is the preferred approach because, unlike the sync height reported by remote peers, it is difficult to game since it is based on the target proof of work, but it assumes consistent mining, which is not the case on all networks, so it is limited to the main and test networks where that applies. For the other networks, it falls back to the sync height reported by the remote peers. The best known header is now logged at initial load time in blockchain and the netsync code is updated to make use of the new headers sync progress logger to tally the number of processed and periodically log the progress.

This modifies blockchain, progresslog, and netsync to provide a progress percentage when logging the main chain verification process. A new method is added to blockchain that determines the verification progress of the current best chain based on how far along it towards the best known header and blockchain is also modified to log the verification progress in the initial chain state message when the blockchain is first loaded. In addition, the progess logger is modified to log the verification progress instead of the timestamp of the last block and the netsync code is updated to make use of the new verification progress func per above. Finally, the netsync code now determines when the chain is considered current and logs new blocks as they are connected along with their hash and additional details. This provides a cleaner distinction between the initial verification and catchup phase and steady state operation.

This removes the response dealine for getheaders since there is no guaranteed response if the remote peer does not have any headers for the locator. Also, while here, modify the debug logging to only log a deadline was added when it actually was.

This modifies the RPC server getblockchaininfo to base the verification progress calculation on the best header now that all headers are determined prior to downloading and verifying blocks.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

multi: Rework sync model to use hdr annoucements. #2555

multi: Rework sync model to use hdr annoucements. #2555

Commits on Jan 22, 2021