feat(storage): support pruning Merkle tries

[kkovaacs]

This PR adds support for a mode of operation where only the last Merkle tries are kept in storage.

Storage can now be configured to prune old nodes when doing trie updates. Upon creating the storage this setting is validated to be safe. Pruning can be enabled only for databases with empty tries and the fact that pruning was enabled is stored in a new storage_flags table.

The storage.prune-state-tries command line flag is added to explicitly set the pruning mode. When not specified the default is the setting set in the database. For new databases pruning is disabled unless explicitly requested via --storage.prune-state-tries true.

Closes #1488

[Mirko-von-Leipzig]

LGTM; some (very) minor comments which you can ignore.

Am I correct in saying this isn't actually enabled within the sync process yet?

I am in fact just blind. Its internalised/abstracted within the storage code.

[Mirko-von-Leipzig]

This works; but I'm thinking about the ergonomics long term here.

Ideally I think we would:

1. Set the flag the very first time the db is created
2. Only check the flag matches the provided one

Currently we can't do this because the rusqlite Connection::open by default enables the SQLITE_OPEN_CREATE flag.

What do you think about changing the first migration step to:

let mut connection = {
    // Try open existing db first
    let mut open_flags = rusqlite::OpenFlags::default();
    open_flags.remove(SQLITE_OPEN_CREATE);

    // TODO: actually ensure that the error is file not found..
    if let Ok(connection) = Connection::open_with_flags(open_flags) {
        break connection;
    }
    
    // Database file should be created, and flags should be set.
    let mut connection = Connection::open()?;
    // TODO: set the flag(s).
};

[kkovaacs]

I've pushed something similar as an update: 9ba6f1d

Does that make more sense to you?

[Mirko-von-Leipzig]

Its better; I don't like that the verification does both verification but also sometimes writes the flags.

// This also sets the one-time flags when creating the database.
let connection = open_or_create(flags)?;[
...
migratate(connection)?;

// Only verification/ensure db makes sense with options.
verify_config(connection)?;

But now that I consider the issue more closely - its that you need the migrations to be completed before you can write the flags? So one would have to bundle the open/create and migration into one step.. bleg.

Okay 👍 I give up 😬

[kkovaacs]

Looks like this breaks enabling pruning for the in-memory DB created by StorageBuilder::in_memory(). That's because we do have to pre-create the in-memory DB and keep a connection open to that while migrate() is running. That's something we can take care of after merging this PR though?

[sistemd]

LGTM. Maybe flags should be bool columns instead of one text column? Not sure if that makes sense, but might be easier to understand.

[CHr15F0x]

LGTM

[kkovaacs]

I'll make this a draft until PR #1898 is merged. I've already resolved conflicts and would rather keep it that way.