Skip to content

Conversation

@m-kus
Copy link
Member

@m-kus m-kus commented Sep 29, 2025

This PR replaces the SQLite store backend from rust-accumulators with a new one, more configurable. The implementation:

  • Adds a new MMRStore that implements the BlockMMRStore trait with SQLite backend
  • Uses WAL mode for concurrent read access during writes
  • Implements transaction support for atomic operations
  • Separates reader and writer connection pools for better concurrency
  • Stores block headers directly in the database instead of relying on RPC calls
  • Refactors the BlockMMR to use the new storage interface
  • Updates the indexer to work directly with the MMR instead of going through the app client

The PR also adds the async-trait dependency and integrates the accumulators crate from the workspace dependencies.

Copy link
Member Author

m-kus commented Sep 29, 2025

@m-kus m-kus mentioned this pull request Sep 29, 2025
@m-kus m-kus marked this pull request as ready for review September 29, 2025 18:22
@m-kus m-kus requested a review from maciejka as a code owner September 29, 2025 18:22
@m-kus m-kus force-pushed the 09-29-single_writer_multiple_reader_mmr_store branch from d3336ab to a621be0 Compare September 29, 2025 18:34
@maciejka
Copy link
Collaborator

Does it make sense to use sqllite if it require maintaining extra logic to manage connections? Wouldn't a good old postgres be a better choice?

@m-kus
Copy link
Member Author

m-kus commented Sep 30, 2025

External PG seems like an overkill, embedded db is a simpler option. We do need acid transaction though (to avoid weird indexing issues), so sqlite is a good choice.
I was actually thinking of using mdbx but it's really clumsy to use with async. So either rewrite rust-accumulators (which I'd really want to avoid) or just keep as it is, just expose a bit more control to the pool management (which is implemented).

@m-kus m-kus mentioned this pull request Sep 30, 2025
@m-kus m-kus force-pushed the 09-29-single_writer_multiple_reader_mmr_store branch 2 times, most recently from fdd9498 to 5ffb65f Compare October 3, 2025 16:16
@m-kus m-kus force-pushed the m-kus/index-chain-state branch from 4365521 to 69221cc Compare October 3, 2025 16:19
@m-kus m-kus force-pushed the 09-29-single_writer_multiple_reader_mmr_store branch from 5ffb65f to e967826 Compare October 3, 2025 16:19
@m-kus m-kus mentioned this pull request Oct 3, 2025
@m-kus m-kus force-pushed the 09-29-single_writer_multiple_reader_mmr_store branch from e967826 to f8254bd Compare October 6, 2025 13:27
@m-kus m-kus force-pushed the m-kus/index-chain-state branch from 69221cc to 295b2ac Compare October 6, 2025 13:27
Base automatically changed from m-kus/index-chain-state to main October 6, 2025 19:16
@m-kus m-kus force-pushed the 09-29-single_writer_multiple_reader_mmr_store branch from f8254bd to f87e158 Compare October 6, 2025 20:37
@m-kus m-kus merged commit e9452c0 into main Oct 6, 2025
9 checks passed
@m-kus m-kus deleted the 09-29-single_writer_multiple_reader_mmr_store branch October 6, 2025 20:47
m-kus added a commit that referenced this pull request Oct 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants