Test bookkeeper performance

[csjx]

We expect to have large numbers of calls to listQuotas(), getQuota(), createUsage(), and updateUsage(). stress test these calls using a tool like JMeter with increasing concurrency, such as 10, 100, 1000, and 10000 concurrent calls.

[csjx]

As an initial test of Apache JMeter, I did the following:

1. Installed JMeter on my laptop and configured a test for the Bookkeeper listQuotas() REST endpoint
   a. Added a Thread Group of 8 threads, ramp up of 1 sec, 1000 iterations
   b. Added an HTTP Request Sampler, configured to call https://api.test.dataone.org:30443/bookkeeper/v1/quotas
   c. Added an HTTP Header Manager and configured it to inject an Authorization header with my token

2. Ran this locally, and saw the 8000 calls take about 106 sec, or about 75 requests/sec. (or 13 ms/request)

3. Saved the test plan to a .jmx file and transferred it to aurora

4. Ran the test plan on aurora, configured with 64 threads instead of 8

./jmeter -n -t ~/bookkeeper-test-plan.jmx -l ~/bookkeeper-test-plan-results.txt -j ~/bookkeeper-test-plan.log

5. Results took 9 sec for 64000 requests - 6941 requests/sec with no errors from Bookkeeper

Here's the summary line in the log output:

2020-08-15 15:34:27,123 INFO o.a.j.e.StandardJMeterEngine: Notifying test listeners of end of test
2020-08-15 15:34:27,129 INFO o.a.j.r.Summariser: summary =  64000 in 00:00:09 = 6941.4/s Avg:     8 Min:     1 Max:   419 Err:     0 (0.00%)

So I think JMeter will be a good tool to set up a more diverse set of tests with different users and authorization levels, and testing different endpoints. We'll need to probably load up the Bookkeeper database with a bunch of quotas and usages content so we have more realistic content to test against. This will come into play when we start supporting storage quotas. Note that the log file shows that it took about 1 second to spin up all 64 threads on aurora.