Need long-term benchmarking

[zramsay]

@zramsay commented on Wed Aug 16 2017

@jaekwon commented on Fri Jan 06 2017

The problem w/ benchmarking using Golang's Benchmark system is that it's not a long-term persistent benchmark. The issue with this is that, for example, when we're using the CLevelDB binding, we're not sure whether we're really testing persistence, or just some memory cache.

We need to write a benchmarking program that can run forever, simulating what might happen during usage in a TMSP app. We need to preload the tree w/ many key/value pairs, then make random modifications and occasionally save, forever.

---

@ethanfrey commented on Fri Jan 06 2017

PR #15 includes some heavy usage of leveldb (lots of data nodes) in make fullbench. I think that is a start.

The daemon program is a good idea, and I would be happy to write something, but I wonder how we would measure stats from it? Sample timing for a few blocks every 5 minutes? It is nice to have some reproducible metrics from it, rather than just taking a look at top. Can you please give a more detailed description of the output of such the process, then I would be happy to work on it.

---

@jaekwon commented on Fri Jan 06 2017

The tests don't have to parameterize everything, but it should be easy to set some parameters at least by tweaking the code. Some parameters that I suspect we'll want to experiment w/:

• Number of bytes for keys
• Number of bytes for values
• Number of pre-fill items (e.g. pre-fill the DB w/ 1M objects)
• Number of txs per "block"

The test should be able to run at memory-equilibrium by default, so maintain the number of elements in memory while still doing a nice mix of removing/updating/inserting leaf items. We'll also want to run experiments where the number of items continue to grow to fill the disk. But if we parameterize all of it it'll just make the code more brittle, so it can just be a simple sample_bench.go file to be modified as needed.

Some output parameters:

• Memory usage upon initialization of initial X items
• Time it took to populate those X items

And for every "block", we can measure

• Memory usage
• Time to run all updates and commit changes

Printing out every block measurement will be too noisy, so we can print avg'd block measurements every X seconds or blocks.

---

@ethanfrey commented on Mon Jan 09 2017

Sounds clear. Shall I add that to PR #15 or do a separate PR for this info?