Speed up processing of block 2469002 on Mainnet

[masonforest]

This transaction[https://etherscan.io/vmtrace?txhash=0xA7B767C051661415164BEC1901874F7E7BFFEC33D01146D8A72A23919A0E6107] currently takes about 15 minutes to process.

It calls BALANCE on address 0x001, 0x002, 0x003 and so on. Each BALANCE call takes about a second to complete. The reason it’s slow is because every time we read a value we have to read a hash from disk at each level of the Merkle Patricia Tree.

We should cache nodes in the Merkle Patricia Tree to make reads faster.

[willmeister]

In looking into this, we've discovered the following:

• Single Trie RocksDB disk reads can take anywhere from 1 microsecond to 1 second and they vary a lot
  • This was detected on the mana-ethereum-sync-2 box
  • This leads us to believe contention for IO on the disk might be an issue
  • For this setup, parity was also writing to the same disk, so we are guessing that parity flushes were causing these extremely slow reads
• All Tries (State DB, Storage, Transactions, Receipts) are stored in the same DB as a single Trie
  • We have a branch mf-wm-optimize-mpt has some code changes that split the State DB off from the rest of the Tries to see if that improves performance when under the BALANCE attack listed above
• We don't do any caching of intermediate Trie nodes that are read from the DB when we look up account info
  • Caching intermediate nodes with a sensible purge strategy could significantly improve performance of BALANCE attack transaction processing, as well as transaction processing in general
• We haven't [yet] looked into optimizing RocksDB configuration for our specific setup, but that could be useful as well.

[willmeister]

Closing this issue in favor of #622, #623, and #624. @masonforest please re-open if you think we should.