How to speed up stagesync: IntermediateHashes?

[shchitlyak47]

Hello, I think I need a little help!

I have a fully synced node and I found that the block execution step takes 1-2 seconds. See the logs where the difference between processing a new payload and notifying the rpc daemon is 1.5 seconds.

[INFO] [08-25|09:40:36.702] [2/15 Headers] Handling new payload      height=17990875 hash=0xc855cb6e363838a56abe4a91f286c84c196ef1a7ecc8d990ed1d2f5331a52bbd
[INFO] [08-25|09:40:36.841] [5/7 Execution] Completed on             block=17990875
[INFO] [08-25|09:40:37.458] Timings (slower than 50ms)               Headers=11.003s
[INFO] [08-25|09:40:37.458] Tables                                   PlainState=97.8GB AccountChangeSet=244.5GB StorageChangeSet=293.0GB BlockTransaction=14.2GB TransactionLog=595.8GB FreeList=83.2MB ReclaimableSpace=166.5GB
[INFO] [08-25|09:40:37.458] [2/15 Headers] Waiting for Consensus Layer...
[INFO] [08-25|09:40:37.497] [2/15 Headers] Fork choice update: flushing in-memory state (built by previous newPayload)
[INFO] [08-25|09:40:38.282] RPC Daemon notified of new headers       from=17990874 to=17990875 hash=0xc855cb6e363838a56abe4a91f286c84c196ef1a7ecc8d990ed1d2f5331a52bbd header sending=23.689µs log sending=271ns
[INFO] [08-25|09:40:38.282] Timings (slower than 50ms)               Headers=137ms CallTraces=139ms AccountHistoryIndex=114ms StorageHistoryIndex=143ms LogIndex=94ms TxLookup=93ms
[INFO] [08-25|09:40:38.282] Tables                                   PlainState=97.8GB AccountChangeSet=244.5GB StorageChangeSet=293.0GB BlockTransaction=14.2GB TransactionLog=595.8GB FreeList=83.2MB ReclaimableSpace=166.5GB
[INFO] [08-25|09:40:39.104] [2/15 Headers] Waiting for Consensus Layer...

I turned on debug mode and found that the intermediate hashing step takes the most time. Especially the Merkle tree calculation which is in trie_root.go. Also, I changed the logProgress in the file to be logged every 30ms and found that this merkle tree calculation takes between 600ms and 1.1s.

My instance is AWS Cloud EC2: 8 CPUs, 64 RAM, and a 3.5TB gp3 SSD 10500 IOPS.

Can anyone explain a bit about this step? Is this stage connected to a database to make calculation or is it just an in-memory?

Note: I also see the trie_async branch, but I'm not sure if it helps with speed or works correctly.

[AskAlexSharov]

gp3 is high-latency disk - disk-latency (not throughput) is your bottleneck. more ram may help a bit.
most of things in erigon are related to database.
trie_async will not help in your case.

[shchitlyak47]

Just for understanding.

Are 0.41ms r_await and 1.19ms w_await high latency?

[shchitlyak47]

Sorry, I had a mistake. I use gp2 type of disk instead of gp3. That is why the latency can be lower.

[shchitlyak47]

Solved.

I switched to another AWS instance that is more storage optimized and latency reduced from 0.4ms to 0.1ms.

[INFO] [08-30|13:13:35.983] [2/15 Headers] Handling new payload      height=18027638 hash=0x5007d8f2fb39282aeb24e6d226e5eb8262726c552b3b44b91c015cc4bf6a05fa
[INFO] [08-30|13:13:36.092] [5/7 Execution] Completed on             block=18027638
[INFO] [08-30|13:13:36.265] Timings (slower than 50ms)               Headers=10.814s
[INFO] [08-30|13:13:36.265] Tables                                   PlainState=98.1GB AccountChangeSet=245.3GB StorageChangeSet=293.9GB BlockTransaction=14.2GB TransactionLog=598.5GB FreeList=80.8MB ReclaimableSpace=161.5GB
[INFO] [08-30|13:13:36.265] [2/15 Headers] Waiting for Consensus Layer...
[INFO] [08-30|13:13:36.304] [2/15 Headers] Fork choice update: flushing in-memory state (built by previous newPayload)
[INFO] [08-30|13:13:36.600] RPC Daemon notified of new headers       from=18027637 to=18027638 hash=0x5007d8f2fb39282aeb24e6d226e5eb8262726c552b3b44b91c015cc4bf6a05fa header sending=21.998µs log sending=287ns

Now the block execution takes ~500 - 900ms.

[AskAlexSharov]

well done

[mariel-sendblocks]

@shchitlyak47 Can you elaborate what ec2 instance you used, if you moved to gp3 or to a storage-optimized ec2 ?
@AskAlexSharov Why you think gp3 isnt good enough ?

[shchitlyak47]

I tried switching to GP3 and GP2 storage types, and also increasing IOPS and throughput, but it didn't help much.

Finally, I changed the EC2 instance to a storage-optimized instance type, especially im4gn.2xlarge, and got a good result.

As I understand it, storage-optimized instances are cleaned up every time the instance crashes, or you want to reboot it. For me, this was the main disadvantage.

[AskAlexSharov]

gp3 is network-based parallel-throughput-oriented high-latency disk. Erigon’s current bottleneck is disk latency.

[mariel-sendblocks]

@shchitlyak47 The storage-optimized's drawback is a huge drawback since I dont want to sync the node every time the ec2 crashes.
Do you remember the numbers you saw from the gp3 ebs ?
I have a gp3 EBS with 16000 iops and 1000 throughput.
In my polygon erigon I have bad numbers for all stages from execution - Each step takes 2h+, and IntermediateHashes takes up to 6h ! .
On the EBS side, I see a write latency of 0.7 ms/op which isnt that bad.
Wondering if you guys have some recommendations how to handle it.

[AskAlexSharov]

On gp3 - you can try:

• MDBX_READAHEAD=true
• no pruning (it actually require more io)
• WRITE_MAP=true
• add more RAM to machine (it will utilized by OS's PageCache and reduce disk-IO)
• ./build/bin/integration warmup --datadir=/mnt/erigon --bucket=PlainState

[mariel-sendblocks]

I'll open a dedicated issue for my case, appreciate if you can have a look - #9347
. Thank you !!