Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: introduce init tip setting for indexer to skip previous block synchronization #4244

Closed

Conversation

EthanYuan
Copy link
Collaborator

@EthanYuan EthanYuan commented Nov 22, 2023

What problem does this PR solve?

Problem Summary:

CKB-indexer supports building indexes only for interested blocks. This can be done by setting block_filter before this PR. But the filter can only traverse the block from 0 and for the uninterested block also need to save its Header(BlockNumber, &'a Byte32, bool).

What is changed and how it works?

This pr introduces the init tip setting, which can be set directly when the indexer initializes the database for the first time, which brings two benefits:

  1. synchronization can be started instantly from any height
  2. the previous block does not need to save any data

What's Changed:

  • Added commented out configuration item to ckb.toml template (as an example)
  • Implement apply_init_tip function, which is called in IndexerService::new
  • Necessary unit tests

Related changes

This feature will also be used for index-r:

Check List

Tests

  • Unit test
  • Manual test
    • general: the indexer can skip the synchronization of the block before the init tip, and the init tip block only saves the block Header(BlockNumber, &'a Byte32, bool)
    • tip hash doesn't match, it rolls back to empty
    • when the init tip is set greater than the current node tip, there are no side effects, it waits for the node to synchronize

Side effects

  • Performance regression
  • Breaking backward compatibility

Release note

Title Only: Include only the PR title in the release note.

@EthanYuan EthanYuan requested a review from a team as a code owner November 22, 2023 16:14
@EthanYuan EthanYuan requested review from zhangsoledad and quake and removed request for a team November 22, 2023 16:14
@EthanYuan EthanYuan self-assigned this Nov 22, 2023
@EthanYuan EthanYuan added the t:enhancement Type: Feature, refactoring. label Nov 22, 2023
# # The initial tip number and hash can be set as the starting height for building indexes in indexer-r. Effective only during the initial index creation.
# init_tip_number = 10000
# # The initial tip hash must match the tip number; otherwise, it will result in a rollback to empty.
# init_tip_hash = "0x8fbd0ec887159d2814cee475911600e3589849670f5ee1ed9798b38fdeef4e44"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggest to use one block hash parameter only, indexer may use get_header rpc to fetch the block number, and start the indexing service only when corresponding header was found.

Copy link
Collaborator Author

@EthanYuan EthanYuan Nov 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Entering a number manually considering a situation where the block height of the node itself may not have been reached and get_header may fail.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please refer to my previous comment:

start the indexing service only when corresponding header was found.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another issue is that if the hash is wrong, then there is no chance of rollback.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indexer service won't starting if the hash is wrong.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, got it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
t:enhancement Type: Feature, refactoring.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants