Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix memory usage of bbolt block header index #213

Merged
merged 5 commits into from
Mar 15, 2021

Commits on Mar 1, 2021

  1. headerfs: add baseline benchmarks

    As a preparation for memory behavior optimizations, we first add a set
    of benchmark tests to establish a baseline against.
    The benchmarks determine the speed and memory usage of writing different
    sized batches to the bbolt index as well as the random access latency
    for retrieving entries from the index.
    guggero committed Mar 1, 2021
    Configuration menu
    Copy the full SHA
    42923a1 View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2021

  1. headerfs: extract heightFromHashWithTx

    We want to isolate the code that reads from/writes to the index bucket
    within the headerIndex type. We prepare to do so by extracting re-usable
    code into methods.
    guggero committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    844a600 View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2021

  1. headerfs: extract chainTipWithTx

    We want to isolate the code that reads from/writes to the index bucket
    within the headerIndex type. We prepare to do so by extracting re-usable
    code into methods.
    guggero committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    dfe4a78 View commit details
    Browse the repository at this point in the history
  2. headerfs: access index only through methods

    Now that we have methods for accessing the index buckets, we use those
    in the blockHeaderStore instead of manipulating the DB directly.
    guggero committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    430444e View commit details
    Browse the repository at this point in the history
  3. headerfs: add sub buckets to index

    With this commit we store the index keys (hash->height) in sub-
    buckets with the first two bytes of the hash as the bucket name.
    Storing a large number of keys in the same bucket has a large
    impact on memory usage in bbolt if small-ish batch sizes are
    used (the b+ tree needs to be copied with every resize operation).
    Using sub buckets is a compromise between memory usage and
    access time. 2 bytes (=max 65535 sub buckets) seems to be the
    sweet spot (-50% memory usage, +30% access time). We take the
    bytes from the beginning of the byte-serialized hash since all
    Bitcoin hashes are reverse-serialized when displayed as
    strings. That means the leading zeroes of a block hash
    are actually at the end of the byte slice.
    guggero committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    9ea609b View commit details
    Browse the repository at this point in the history