redpanda: low latency mode

[emaxerrno]

hi @adamw - i saw the post (8873fa5?short_path=e28ac69#diff-e28ac696dc377acf5d4494af8b82c2fa6467c86cfea831bdd149c1d124eb5edf) when i was looking up redpanda projects.

We just introduced a patch for even lower latency (at expense of lower throughput): redpanda-data/redpanda@1b0f027

but also, the post talks about gp2 as the ebs volumes. In this setup (while I think is good to keep setup the same) you will be bottlenecke'd by IOPs, not by throughput. In fact is a bit disadvantageous to redpanda in that mode if you care about performance in that we issue a bit higher IOPs. For example, in order to gain performance improvements on local NVMe ssd devices we leverage sparse file system support ( dispatch of iocb in order 1, 2, 3, 4 but reap events from the kernel in order of 3, 2, 1, 4 - as one example)

Just wanted to add the 2 cents there.

Let me know if you want to brainstorm help to tune your setup on vectorized.io/slack always happy to help folks benchmarking.

[emaxerrno]

One last thing!, next wednesday we'll release v21.8.1 has a bunch of perf improvements too.

[adamw]

Thanks for the feedback! I think I did monitor IOPS for this test as well, but I don't have a hard record of it, so I can't be 100% sure. We should probably add this metric to the data we save for each test :)

We periodically re-run the tests with newer versions, I'll let you know here when this happens.