feat: add support for penumbra v0.81.x

[conorsch]

Adds support for the v0.80.x -> v0.81.x chain upgrade for the penumbra-1 chain.

Testing and review

1. Use a preexisting reindexer_archive.bin, or obtain from https://artifacts.plinfra.net/: the one marked reindexer_archive-height-2609670.sqlite would be great because it's right before the target upgrade height that we're adding support for.
2. Use an up-to-date node state dir to update the archive: systemctl stop penumbra cometbft && penumbra-reindexer archive
3. Ideally, target a fresh database and run penumbra-reindexer archive [...] to write to it. That'll take a while, ~20h or so.

[cronokirby]

Looks like this implements the no-op migration logic we need ; curious to see how a run of this goes

[conorsch]

Just made it over the first upgrade boundary, at 501974, so a while to go yet until the new logic is hit. But that's running from scratch, to make a full database. In order to validate just the new logic, can I just run from a target block?

I'd thought this would have worked: penumbra-reindexer regen --archive-file reindexer_archive.bin --start-height 2000000 --working-dir reindexer-working-dir --database-url 'postgresql://penumbra:penumbra@127.0.0.1:6666/penumbra_raw_new' but nay:

full logs and error

❯ penumbra-reindexer regen --archive-file reindexer_archive.bin --start-height 2000000 --working-dir reindexer-working-dir --database-url 'postgresql://penumbra:penumbra@127.0.0.1:6666/penumbra_raw_new'
2025-01-23T02:06:59.679260Z  INFO init{database_url="postgresql://penumbra:penumbra@127.0.0.1:6666/penumbra_raw_new"}: penumbra_reindexer::indexer: initializing database
2025-01-23T02:06:59.681964Z  INFO init{database_url="postgresql://penumbra:penumbra@127.0.0.1:6666/penumbra_raw_new"}: sqlx::postgres::notice: relation "blocks" already exists, skipping
2025-01-23T02:06:59.682467Z  INFO init{database_url="postgresql://penumbra:penumbra@127.0.0.1:6666/penumbra_raw_new"}: sqlx::postgres::notice: relation "idx_blocks_height_chain" already exists, skipping
2025-01-23T02:06:59.682606Z  INFO init{database_url="postgresql://penumbra:penumbra@127.0.0.1:6666/penumbra_raw_new"}: sqlx::postgres::notice: relation "tx_results" already exists, skipping
2025-01-23T02:06:59.682735Z  INFO init{database_url="postgresql://penumbra:penumbra@127.0.0.1:6666/penumbra_raw_new"}: sqlx::postgres::notice: relation "events" already exists, skipping
2025-01-23T02:06:59.682856Z  INFO init{database_url="postgresql://penumbra:penumbra@127.0.0.1:6666/penumbra_raw_new"}: sqlx::postgres::notice: relation "attributes" already exists, skipping
2025-01-23T02:06:59.683004Z  INFO init{database_url="postgresql://penumbra:penumbra@127.0.0.1:6666/penumbra_raw_new"}: sqlx::postgres::notice: relation "idx_attributes_event_id" already exists, skipping
2025-01-23T02:06:59.699911Z  INFO cnidarium::storage: opening rocksdb config column path="reindexer-working-dir"
2025-01-23T02:06:59.743640Z  INFO cnidarium::storage: reading prefixes from config column family
2025-01-23T02:06:59.744432Z  INFO cnidarium::storage: initializing global store config
2025-01-23T02:06:59.744440Z  INFO cnidarium::storage: creating substore config for prefix prefix="cometbft-data"
2025-01-23T02:06:59.744446Z  INFO cnidarium::storage: creating substore config for prefix prefix="ibc-data"
2025-01-23T02:06:59.744458Z  INFO cnidarium::storage: opening rocksdb path="reindexer-working-dir"
2025-01-23T02:06:59.786255Z  INFO cnidarium::storage: snapshot dispatcher task has started
2025-01-23T02:06:59.786968Z  INFO cnidarium::storage: opening rocksdb config column path="reindexer-working-dir"
2025-01-23T02:06:59.829233Z  INFO cnidarium::storage: reading prefixes from config column family
2025-01-23T02:06:59.829539Z  INFO cnidarium::storage: initializing global store config
2025-01-23T02:06:59.829547Z  INFO cnidarium::storage: creating substore config for prefix prefix="cometbft-data"
2025-01-23T02:06:59.829552Z  INFO cnidarium::storage: creating substore config for prefix prefix="ibc-data"
2025-01-23T02:06:59.829570Z  INFO cnidarium::storage: opening rocksdb path="reindexer-working-dir"
2025-01-23T02:06:59.871087Z  INFO cnidarium::storage: snapshot dispatcher task has started
2025-01-23T02:06:59.871705Z  INFO run_to{version=V0o80 first_block=2000001 last_block=Some(2611800)}: penumbra_reindexer::penumbra: regeneration step
2025-01-23T02:06:59.871748Z  INFO run_to{version=V0o80 first_block=2000001 last_block=Some(2611800)}: cnidarium::storage: opening rocksdb config column path="reindexer-working-dir"
2025-01-23T02:06:59.914178Z  INFO run_to{version=V0o80 first_block=2000001 last_block=Some(2611800)}: cnidarium::storage: reading prefixes from config column family
2025-01-23T02:06:59.914592Z  INFO run_to{version=V0o80 first_block=2000001 last_block=Some(2611800)}: cnidarium::storage: initializing global store config
2025-01-23T02:06:59.914601Z  INFO run_to{version=V0o80 first_block=2000001 last_block=Some(2611800)}: cnidarium::storage: creating substore config for prefix prefix="cometbft-data"
2025-01-23T02:06:59.914607Z  INFO run_to{version=V0o80 first_block=2000001 last_block=Some(2611800)}: cnidarium::storage: creating substore config for prefix prefix="ibc-data"
2025-01-23T02:06:59.914619Z  INFO run_to{version=V0o80 first_block=2000001 last_block=Some(2611800)}: cnidarium::storage: opening rocksdb path="reindexer-working-dir"
2025-01-23T02:06:59.958625Z  INFO cnidarium::storage: snapshot dispatcher task has started
2025-01-23T02:06:59.958820Z  INFO run_to{version=V0o80 first_block=2000001 last_block=Some(2611800)}: penumbra_reindexer::penumbra: running chain from heights 2000001 to 2611800
thread 'main' panicked at /home/conor/.cargo/git/checkouts/penumbra-54633d8978d5e293/0d0b319/crates/core/app/src/app/mod.rs:653:14:
chain id is always set
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2025-01-23T02:06:59.962495Z  INFO cnidarium::storage: dispatcher task has terminated
^C

Perhaps I'm misunderstanding the purpose of the --start-height option. It'd sure be nice to have test reindexing events in discrete chunks of blocks, rather than en masse.

[conorsch]

Results:

2025-01-23T11:51:09.047656Z  INFO init_then_run_to{genesis_height=501975 version=V0o80 first_block=501975 last_block=Some(2611800)}: penumbra_reindexer::penumbra: reached height 2609600
2025-01-23T11:51:09.724047Z  WARN init_then_run_to{genesis_height=501975 version=V0o80 first_block=501975 last_block=Some(2611800)}:end_block{height=2609640}:shielded_pool: penumbra_shielded_pool::fmd: decreasing clue count at height 2609640: 31 then 17
2025-01-23T11:51:10.243356Z  INFO cnidarium::storage: dispatcher task has terminated
Error: missing block at height 2609671

The failure occurs shortly before the new upgrade boundary at 2611800, ostensibly due to an incomplete sqlite3 db. I'll try to patch together the missing ranges from backups.

[conorsch]

Patched up the sqlite3 db, see #22 for details. After progressing beyond 2609671, to the newly added v0.81.x logic that constitutes the core of this PR, I encountered an off-by-one error in the planned halt blocks. That's resolved now, in a recent commit, but I'm struggling to resume the regen process without starting completely over.

[conorsch]

The latest code successfully regenerated a cometbft event db for the penumbra-1 chain, up to block height 3186459. That's good enough for the protocol compat we were aiming for in this PR, so I'm merging.