Release UTxO-HD v0.1

[dnadales]

Previous work

• ✔️ Consensus implementation ready (both the in-memory and LMDB backend).
• ✔️ Feature branch integrated into a cardano-node branch.
• ✔️ System-level benchmarks for the in-memory backend done.
  • ✔️ The value-only workload benchmark showed no regressions for the metrics of interest (forging, peer-propagation, end-to-end propagation). ⚠️ However, this feature demands more resources. Ad-hoc measurements on a running node show we consume about 2.5 GB extra with the in-memory solution.
  • ✔️ Like in the value-only benchmark, the Plutus workload benchmark did not show any regression for the metrics of interest; however, it also showed an increase in resource usage.
• 🚫 System-level benchmarks for the LMDB backend blocked (see Depends On below).
• ✔️ We managed to run a UTxO-HD capable node in legacy mode, which maintains the baseline memory usage while keeping all the ledger states in memory (as the current node does). This legacy mode is not production-ready because it requires further integration and testing.
• ✔️ We finished the redesign of the Ledger DB API, which the LSM-tree backend will require, but, in addition, it might enable us to design an in-memory backend that will keep the baseline performance and resource requirements without requiring the code duplication and additional maintenance effort the legacy mode does.
• ✔️ We integrated the existing implementations (in-memory and LMDB into the new Ledger DB API).
• ✔️ We implemented the new in-memory backend.

Current work

• 🛠️ P&T is working on the LMDB benchmarking setup. the current benchmark setup uses virtualized I/O, which renders the results unreliable. The benchmarking setup needs to be adapted accordingly.
• 🛠️ Perform ad-hoc and system-level benchmarks on the new in-memory backend.
• Re-enable Consensus microbenchmarks (mempool) or establish a new baseline if system-level benchmarks show good performance.
• Refine the structure of the Git history for future reference.
• Merge the behaviour-preserving parts of the feature branch into main.
• Document the feature and inform the community (eg through blogposts).

Current implementation status

• UTXO-HD code is rebased on top of cardano-node 8.9.1
• UTXO-HD node is runnable in three modes:
  • V1InMemory (not recommended)
  • V2InMemory
  • V1LMDB
• It successfully syncs, replays and answers queries
• Branches containing the current implementation:
  • WIP: UTXO-HD for node 8.9.0 ouroboros-consensus#977
  • WIP: UTXO-HD for node 8.9.1 cardano-api#494
  • WIP: UTXO-HD for node 8.9.1 #5726

Roadmap

• Finish V2 flavor of the in-memory implementation ouroboros-consensus#905
• Fix Consensus testsuites in the UTxO-HD branch ouroboros-consensus#906
• Rebase UTxO-HD prototype on top of node 8.9.1 ouroboros-consensus#907
• Onboard (Renate|Fraser)+ on UTxO-HD ouroboros-consensus#908
• UTXO-HD: Prepare LMDB implementation for benchmarking #5697
• Test synchronization and replay for the new in-memory implementation ouroboros-consensus#923
• Add new Consensus tests for the V2 implementation ouroboros-consensus#909
• Merge UTxO-HD-independent code to main ouroboros-consensus#910
• Experiment with db-analyzer on the UTxO-HD branch ouroboros-consensus#911
• Run system-level benchmarks and confirm integration tests results ouroboros-consensus#912
• Document feature UTxO-HD v0.1 ouroboros-consensus#126
• Get UTxO-HD ready for handover ouroboros-consensus#28
• Use Simple diffs in V2 InMemory ouroboros-consensus#1114

Requirements

• We should provide an in-memory alternative with equal or better performance and resource requirements than the current node.
• Default backend should be in-memory.
• Release should be signed off by P&T.
• Release should be signed off by SDET.
• The UTxO-HD feature should be explained and advertised to the community through different channels and appropriately documented.
• The feature should not impact the Conway release.

Impact

• No migration mechanism will be implemented: users will need to re-apply the chain (which will take several hours but is less time-consuming than syncing from scratch).
• [SPIKE] - Assess UTxO HD impact on DBSync & Make Compatibility Changes cardano-db-sync#1296

[dnadales]

Memory usage comparison using baseline 8.1.2 and 8.2.1-pre against UTxO-HD with both backends after 11 days running (2 corresponds to in-memory, 3 to LMDB). NOTE: we have not run a node whose memory is restricted to 8GB. This might be a very useful data point.

Edge #    Version                Extra Config                Post Replay RSS    +~3 day RSS    +~11 day RSS
1         8.1.2                                              13.47 GiB          13.88 GiB      13.88 GiB
2         8.2.1 HD (52e9e448)                                15.82 GiB          16.15 GiB      16.39 GiB
3         8.2.1 HD (52e9e448)    --lmdb-ledger-db-backend    9.50 GiB           9.74 GiB       10.43 GiB
4         8.2.1-pre                                          13.28 GiB          13.98 GiB      14.01 GiB

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.

[dnadales]

• The latest preemtive sync showed the following results:
  
  It is important to note that the log for the baseline node was 7.7MBs whereas the UTXO-HD runs resulted in 4.1GBs. This was done with the legacy tracing system in which eliding tracers might be at fault for so much output and at least part of the regression.

[jasagredo]

Closing in favor of #5918.