clickhouse and poller settings

[iuwqyir]

TL;DR

Added support for configuring parallel pollers and asynchronous inserts in ClickHouse storage.

What changed?

• Added a new configuration option poller.parallelPollers to control the number of parallel pollers.
• Introduced asyncInsert configuration for ClickHouse storage in main, staging, and orchestrator settings.
• Updated the Poller to use the configured number of parallel pollers.
• Modified ClickHouse connection setup to enable async inserts when configured.
• Fixed error messages in InsertBlockData function to correctly reflect insert operations.

How to test?

1. Update your configuration to include the new poller.parallelPollers setting.
2. Set asyncInsert: true in the ClickHouse configuration for main, staging, or orchestrator storage.
3. Run the application with these new settings and monitor performance.
4. Check logs for debug messages showing executed queries.
5. Verify that data is being inserted correctly with the new parallel polling and async insert features.

Why make this change?

This change aims to improve performance and scalability of the application by:

1. Allowing multiple pollers to work in parallel, potentially increasing the rate of data collection.
2. Enabling asynchronous inserts in ClickHouse, which can improve write performance for large datasets.

These optimizations should lead to faster data processing and reduced latency, especially when dealing with high volumes of blockchain data.

[iuwqyir]

• clickhouse and poller settings #110 👈
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @iuwqyir and the rest of your teammates on Graphite

[catalyst17]

what's the reason to have asyncInserts?

[iuwqyir]

To not overload Clickhouse with too many inserts