Record transactions into pindexer

[JasonMHasperhoven]

Describe your changes

Issue ticket number and link

Checklist before requesting a review

• I have added guiding text to explain how a reviewer should test these changes.

• If this code contains consensus-breaking changes, I have added the "consensus-breaking" label. Otherwise, I declare my belief that there are not consensus-breaking changes, for the following reason:

  REPLACE THIS TEXT WITH RATIONALE (CAN BE BRIEF)

[conorsch]

Needs rebasing, due to merge of #5063.

[conorsch]

While working locally on this branch, I encountered a compilation error, despite CI being green. Investigating CI caching...

[conorsch]

While working locally on this branch, I encountered a compilation error, despite CI being green. Investigating CI caching...

This was caused by rebase-vs-merge differences in how CI was being executed. After rebasing on top of latest main, I can confirm that everything's building correctly.

[conorsch]

In the absence of testing guidance in the PR submission text, I opted to run pindexer as built from this branch against a dbdump of raw ABCI events database for the testnet. The process failed on or around height 2424373 (current at time of writing is 2564234). Given that the upgrade for LQT compat (#5010) for the testnet occurred at height 2358330, I suspect this is a protocol compatibility issue, given that this PR targets main. I tried rebasing this branch on protocol/lqt_branch, but encountered conflicts in pindexer, and shelved that effort.

So, the question is: should we be making these changes against main? Given that they unblock ongoing DEX-explorer work, that makes sense to me, but we need to be careful about honoring compatibility while we work towards LQT support.

Below are some additional logging outputs from my local testing, for reference.

pindexer debug logs

2025-02-19T22:15:46.718515Z DEBUG sqlx::query: summary="INSERT INTO insights_supply(height, total, …" db.statement="\n\nINSERT INTO\n  insights_supply(\n    height,\n    total,\n    staked,\n    price,\n    price_numeraire_asset_id\n  )\nVALUES\n  ($1, $2, $3, $5, $4) ON CONFLICT (height) DO\nUPDATE\nSET\n  total = excluded.total,\n  staked = excluded.staked,\n  price = excluded.price,\n  price_numeraire_asset_id = excluded.price_numeraire_asset_id\n" rows_affected=1 rows_returned=0 elapsed=135.811µs elapsed_secs=0.000135811
2025-02-19T22:15:46.718524Z DEBUG pindexer::insights: unrecognized event event=ContextualizedEvent { event: Event { kind: "penumbra.core.component.fee.v1.EventBlockFees", attributes: [V037(EventAttribute { key: "swappedBaseFeeTotal", value: "{\"amount\":{}}", index: false }), V037(EventAttribute { key: "swappedFeeTotal", value: "{\"amount\":{}}", index: false }), V037(EventAttribute { key: "swappedTipTotal", value: "{\"amount\":{}}", index: false })] }, block_height: 2424372, tx: None, local_rowid: 10062008 }
2025-02-19T22:15:46.718533Z DEBUG pindexer::insights: unrecognized event event=ContextualizedEvent { event: Event { kind: "penumbra.core.component.sct.v1.EventAnchor", attributes: [V037(EventAttribute { key: "anchor", value: "{\"inner\":\"mrHv5/y/csZc2waknV0db/rG+I43Abc693vIoCLZ6Q0=\"}", index: false }), V037(EventAttribute { key: "height", value: "\"2424372\"", index: false }), V037(EventAttribute { key: "timestamp", value: "\"2025-02-11T17:17:05+00:00\"", index: false })] }, block_height: 2424372, tx: None, local_rowid: 10062009 }
2025-02-19T22:15:46.718540Z DEBUG pindexer::insights: unrecognized event event=ContextualizedEvent { event: Event { kind: "penumbra.core.component.sct.v1.EventBlockRoot", attributes: [V037(EventAttribute { key: "height", value: "\"2424372\"", index: false }), V037(EventAttribute { key: "root", value: "{\"inner\":\"AQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=\"}", index: false }), V037(EventAttribute { key: "timestamp", value: "\"2025-02-11T17:17:05+00:00\"", index: false })] }, block_height: 2424372, tx: None, local_rowid: 10062010 }
2025-02-19T22:15:46.718547Z DEBUG pindexer::insights: unrecognized event event=ContextualizedEvent { event: Event { kind: "block", attributes: [V037(EventAttribute { key: "height", value:"2424373", index: false })] }, block_height: 2424373, tx: None, local_rowid: 10062011 }
2025-02-19T22:15:46.718553Z DEBUG pindexer::insights: unrecognized event event=ContextualizedEvent { event: Event { kind: "penumbra.core.component.distributions.v1.EventLqtPoolSizeIncrease", attributes: [V037(EventAttribute { key: "epoch", value: "\"24244\"", index: false }), V037(EventAttribute { key: "increase", value: "{\"lo\":\"63411\"}", index: false }), V037(EventAttribute { key: "newTotal", value: "{}", index: false })] }, block_height: 2424373, tx: None, local_rowid: 10062012 }
2025-02-19T22:15:46.718657Z DEBUG sqlx::query: summary="INSERT INTO dex_ex_block_summary ( …" db.statement="\n\nINSERT INTO\n  dex_ex_block_summary (\n    height,\n    time,\n    batch_swaps,\n    num_open_lps,\n    num_closed_lps,\n    num_withdrawn_lps,\n    num_swaps,\n    num_swap_claims,\n    num_txs\n  )\nVALUES\n  ($1, $2, $3, $4, $5, $6, $7, $8, $9)\n" rows_affected=1 rows_returned=0 elapsed=178.581µs elapsed_secs=0.000178581
2025-02-19T22:15:46.718907Z DEBUG sqlx::query: summary="SELECT total, staked, price …" db.statement="\n\nSELECT\n  total,\n  staked,\n  price\nFROM\n  insights_supply\nORDER BY\n  HEIGHT DESC\nLIMIT\n  1\n" rows_affected=0 rows_returned=0 elapsed=314.482µs elapsed_secs=0.000314482
2025-02-19T22:15:46.719482Z DEBUG sqlx::query: summary="SELECT events.rowid, events.type, events.height, …" db.statement="\n\nSELECT\n  events.rowid,\n  events.type,\n  events.height,\n  tx_results.tx_hash,\n  events.attrs\nFROM\n  (\n    SELECT\n      (\n        SELECT\n          height\n        FROM\n          blocks\n        WHERE\n          blocks.rowid = block_id\n  ) as height,\n      rowid,\n      type,\n      block_id,\n      tx_id,\n      jsonb_object_agg(attributes.key, attributes.value) AS attrs\n    FROM\n      events\n      LEFT JOIN attributes ON rowid = attributes.event_id\n    WHERE\n      block_id >= (\n        SELECT\n          rowid\n        FROM\n          blocks\n        where\n          height = $1\n      )\n      AND block_id <= (\n        SELECT\n          rowid\n        FROM\n          blocks\n        where\n          height = $2\n      )\n    GROUP BY\n      rowid,\n      type,\n      block_id,\n     tx_id\n    ORDER BY\n      rowid ASC\n  ) events\n  LEFT JOIN LATERAL (\n    SELECT\n      *\n    FROM\n      tx_results\n    WHERE\n      tx_results.rowid = events.tx_id\n    LIMIT\n      1\n  ) tx_results ON TRUE\nORDER BY\n  events.rowid ASC\n" rows_affected=0 rows_returned=0 elapsed=166.249614ms elapsed_secs=0.166249614
Error: transaction body malformed

Caused by:
    0: action malformed while parsing transaction body
    1: missing action content

pindexer db height

❯ psql postgresql://127.0.0.1:6666/pindexer -c 'SELECT height FROM block_details ORDER BY height DESC LIMIT 5;'
 height
---------
 2425000
 2424999
 2424998
 2424997
 2424996
(5 rows)

pindexer db height

❯ psql postgresql://127.0.0.1:6666/penumbra_raw -c 'SELECT height FROM blocks ORDER BY height DESC LIMIT 5;'
 height
---------
 2562814
 2562813
 2562812
 2562811
 2562810
(5 rows)

It might be worthwhile to run this version of pindexer against mainnet events; I can do that, too, as a cross-check on the protocol compatibility hypothesis above, but that'll take significantly longer.

[conorsch]

Contains breaking changes that cause pindexer to fail when run against testnet data. See detailed comment above. Requesting changes to block merge, to avoid breaking indexing pipelines.

[conorsch]

It might be worthwhile to run this version of pindexer against mainnet events

Summarizing discussion with @cronokirby out of band: yes, we agree we should do this, and we both expect that testing against mainnet data will succeed. If not, we'll revise the PR accordingly. But assuming we get green on the mainnet reindex, I'll approve this PR, merge it, then we'll need to rebase protocol/lqt_branch on top and manually resolve conflicts, to ensure continued testnet support.