[Feature Request] Benchmark HTTP/1.1 vs HTTP/2 (over clear text) for _bulk APIs

[reta]

Is your feature request related to a problem? Please describe

In scope of implementing an experimental gRPC transport [1], a round of benchmarks have been performed [2] to compare HTTP/1.1 and gRPC in context of _bulk APIs. However, it turned out we haven't actually benchmarked HTTP/1.1 vs HTTP/2.

[1] #16534
[2] #16711 (comment)

Describe the solution you'd like

Benchmark HTTP/1.1 vs HTTP/2 (over clear text) for _bulk APIs

Setup

• 1 data node (48 cores, 32GB mem)
• 100 index + 100 update + 100 delete operations per bulk request (source=true) see please bulk.json
• The document ingested was 4 key/value pairs of string type.
• RPS = 100
• Ran test for 10 minutes
• Restarting the OpenSearch service before each test
• Tool used: vegeta
• OpenSearch 3.0.0 SNAPSHOT on JDK-23 (as of d920bff)
• No secure transport (to match [1])

[1] #16534

Related component

Indexing

Describe alternatives you've considered

N/A

Additional context

N/A

[reta]

Benchmarks results

Every benchmark round consisted of 3 runs.
There is no CPU/Heap/Memory graphs attached but there is no significant variations noticed across all runs.
No errors have been recorded, server logs were clean.
Use persistent connections by default (keep-alive)

Default transport: transport-netty4

HTTP/1.1:

Command:

$ ./vegeta attack -body bulk.json -duration 10m -targets bulk.target -output bulk-h1-1.out -rate 100/1s`

Runs:

$ cat bulk-h1-1.out |   ./vegeta report -type=text                                                                                        
Requests      [total, rate, throughput]         60000, 100.00, 100.00
Duration      [total, attack, wait]             10m0s, 10m0s, 11.537ms
Latencies     [min, mean, 50, 90, 95, 99, max]  9.478ms, 70.225ms, 11.885ms, 13.443ms, 17.363ms, 2.612s, 5.149s
Bytes In      [total, mean]                     4313485091, 71891.42
Bytes Out     [total, mean]                     2294160000, 38236.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:60000 

$ cat bulk-h1-1.out |   ./vegeta report -type=text
Requests      [total, rate, throughput]         60000, 100.00, 100.00
Duration      [total, attack, wait]             10m0s, 10m0s, 10.852ms
Latencies     [min, mean, 50, 90, 95, 99, max]  9.224ms, 64.53ms, 11.909ms, 13.384ms, 17.319ms, 2.398s, 5.099s
Bytes In      [total, mean]                     4336811404, 72280.19
Bytes Out     [total, mean]                     2294160000, 38236.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:60000  
Error Set:

$ cat bulk-h1-1.out |   ./vegeta report -type=text
Requests      [total, rate, throughput]         60000, 100.00, 100.00
Duration      [total, attack, wait]             10m0s, 10m0s, 11.966ms
Latencies     [min, mean, 50, 90, 95, 99, max]  9.174ms, 57.278ms, 12.043ms, 13.488ms, 16.61ms, 1.956s, 4.975s
Bytes In      [total, mean]                     4337309503, 72288.49
Bytes Out     [total, mean]                     2294160000, 38236.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:60000  
Error Set:

HTTP/2 (over clear text):

Command:

$ ./vegeta attack -body bulk.json -duration 10m -targets bulk.target -output bulk-h2c-1.out -rate 100/1s -h2c

Runs:

$ cat bulk-h2c-1.out |   ./vegeta report -type=text
Requests      [total, rate, throughput]         60000, 100.00, 100.00
Duration      [total, attack, wait]             10m0s, 10m0s, 12.319ms
Latencies     [min, mean, 50, 90, 95, 99, max]  9.637ms, 50.003ms, 12.539ms, 23.265ms, 31.214ms, 1.692s, 4.199s
Bytes In      [total, mean]                     4337049404, 72284.16
Bytes Out     [total, mean]                     2294160000, 38236.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:60000  
Error Set:

$ cat bulk-h2c-1.out |   ./vegeta report -type=text
Requests      [total, rate, throughput]         60000, 100.00, 100.00
Duration      [total, attack, wait]             10m0s, 10m0s, 12.229ms
Latencies     [min, mean, 50, 90, 95, 99, max]  9.228ms, 68.099ms, 12.325ms, 16.919ms, 35.241ms, 2.451s, 4.669s
Bytes In      [total, mean]                     4336808509, 72280.14
Bytes Out     [total, mean]                     2294160000, 38236.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:60000  
Error Set:

$ cat bulk-h2c-1.out |   ./vegeta report -type=text
Requests      [total, rate, throughput]         60000, 100.00, 100.00
Duration      [total, attack, wait]             10m0s, 10m0s, 11.42ms
Latencies     [min, mean, 50, 90, 95, 99, max]  9.215ms, 60.492ms, 12.188ms, 18.018ms, 29.076ms, 2.209s, 4.746s
Bytes In      [total, mean]                     4361174773, 72686.25
Bytes Out     [total, mean]                     2294160000, 38236.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:60000  
Error Set:

Reactive transport: transport-reactor-netty4

Command:

$ ./vegeta attack -body bulk.json -duration 10m -targets bulk.target -output bulk-h2c-1.out -rate 100/1s -h2c

Runs:

$ cat bulk-h1-1.out |   ./vegeta report -type=text
Requests      [total, rate, throughput]         60000, 100.00, 100.00
Duration      [total, attack, wait]             10m0s, 10m0s, 12.492ms
Latencies     [min, mean, 50, 90, 95, 99, max]  9.67ms, 52.997ms, 12.525ms, 14.094ms, 17.205ms, 1.861s, 4.531s
Bytes In      [total, mean]                     4361606954, 72693.45
Bytes Out     [total, mean]                     2294160000, 38236.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:60000  
Error Set:

$ cat bulk-h1-1.out |   ./vegeta report -type=text
Requests      [total, rate, throughput]         60000, 100.00, 100.00
Duration      [total, attack, wait]             10m0s, 10m0s, 12.559ms
Latencies     [min, mean, 50, 90, 95, 99, max]  9.779ms, 81.012ms, 12.546ms, 14.245ms, 22.254ms, 2.93s, 5.826s
Bytes In      [total, mean]                     4360873923, 72681.23
Bytes Out     [total, mean]                     2294160000, 38236.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:60000  
Error Set:

$ cat bulk-h1-1.out |   ./vegeta report -type=text
Requests      [total, rate, throughput]         60000, 100.00, 100.00
Duration      [total, attack, wait]             10m0s, 10m0s, 13.552ms
Latencies     [min, mean, 50, 90, 95, 99, max]  9.743ms, 88.863ms, 12.641ms, 14.158ms, 26.195ms, 3.255s, 6.329s
Bytes In      [total, mean]                     4360454640, 72674.24
Bytes Out     [total, mean]                     2294160000, 38236.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:60000  
Error Set:

HTTP/2 (over clear text):

Command:

$ ./vegeta attack -body bulk.json -duration 10m -targets bulk.target -output bulk-h2c-1.out -rate 100/1s -h2c

Runs:

$ cat bulk-h2c-1.out |   ./vegeta report -type=text
Requests      [total, rate, throughput]         60000, 100.00, 100.00
Duration      [total, attack, wait]             10m0s, 10m0s, 11.492ms
Latencies     [min, mean, 50, 90, 95, 99, max]  9.66ms, 45.976ms, 12.741ms, 19.183ms, 29.889ms, 1.47s, 4.326s
Bytes In      [total, mean]                     4361725824, 72695.43
Bytes Out     [total, mean]                     2294160000, 38236.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:60000  
Error Set:

$ cat bulk-h2c-1.out |   ./vegeta report -type=text
Requests      [total, rate, throughput]         60000, 100.00, 100.00
Duration      [total, attack, wait]             10m0s, 10m0s, 11.339ms
Latencies     [min, mean, 50, 90, 95, 99, max]  9.483ms, 63.149ms, 12.72ms, 18.511ms, 30.777ms, 2.359s, 4.573s
Bytes In      [total, mean]                     4360296491, 72671.61
Bytes Out     [total, mean]                     2294160000, 38236.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:60000  
Error Set:

$ cat bulk-h2c-1.out |   ./vegeta report -type=text
Requests      [total, rate, throughput]         60000, 100.00, 100.00
Duration      [total, attack, wait]             10m0s, 10m0s, 16.787ms
Latencies     [min, mean, 50, 90, 95, 99, max]  9.812ms, 53.23ms, 12.835ms, 17.654ms, 27.709ms, 1.876s, 4.502s
Bytes In      [total, mean]                     4361137188, 72685.62
Bytes Out     [total, mean]                     2294160000, 38236.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:60000  
Error Set:

[reta]

Benchmarks comparison

Default transport: transport-netty4

Latencies	HTTP/1.1	HTTP/1.1	HTTP/1.1	HTTP/2	HTTP/2	HTTP/2
min (ms)	9.478	9.224	9.174	🔼 9.637 🔼	🔼 9.228 🔼	🔼 9.215 🔼
mean (ms)	70.225	64.53	57.28	🔽 50.003 🔽	🔼 68.099 🔼	🔼 60.492 🔼
50th (ms)	11.885	11.909	12.043	🔼 12.539 🔼	🔼 12.325 🔼	🔼 12.188 🔼
90th (ms)	13.443	13.384	13.488	🔼 23.265 🔼	🔼 16.919 🔼	🔼 18.018 🔼
95th (ms)	17.363	17.319	16.61	🔼 31.214 🔼	🔼 35.241 🔼	🔼 29.076 🔼
99th (ms)	2612	2398	1956	🔽 1692 🔽	🔼 2451 🔼	🔼 2209 🔼
max (ms)	5149	5099	4975	🔽 4199 🔽	🔽 4669 🔽	🔽 4746 🔽

Reactive transport: transport-reactor-netty4

Latencies	HTTP/1.1	HTTP/1.1	HTTP/1.1	HTTP/2	HTTP/2	HTTP/2
min (ms)	9.67	9.779	9.743	🔽 9660 🔽	🔽 9483 🔽	🔼 9812 🔼
mean (ms)	52.997	81.012	88.863	🔽 45.976 🔽	🔽 63.149 🔽	🔽 53.230 🔽
50th (ms)	12.525	12.546	12.641	🔼 12.741 🔼	🔼 12.720 🔼	🔼 12.835 🔼
90th (ms)	14.094	14.245	14.158	🔼 18.183 🔼	🔼 18.511 🔼	🔼 17.654 🔼
95th (ms)	17.205	22.254	26.195	🔼 29.889 🔼	🔼 30.777 🔼	🔼 27.709 🔼
99th (ms)	1861	2930	3255	🔽 1470 🔽	🔽 2359 🔽	🔽 1876 🔽
max (ms)	4531	5826	6329	🔽 4326 🔽	🔽 4573 🔽	🔽 4502 🔽

Conclusions

• The HTTP/2 has consistently lower tail latencies
• The HTTP/1.1 is overall showing better latency distribution across majority of measurements
• The benchmarking is limited to _bulk API only (non-streaming)

[finnegancarroll]

Thanks for the detailed comparison @reta!

It's interesting that p50 latencies stay around ~12ms across the board but the largest discrepancies are seen in p90/99s.

I'm curious to hear your thoughts on which HTTP2 features we expect to deliver the most impact. My initial impression is multiplexing is the biggest deal, allowing the server to re-use a single connection to handle multiple requests from a client. I think we would see this impact in the worker thread pool/queue of our client/server transport? It's not clear to me how much this should trickle down to total latency, particularly if our thread pool is not under significant strain and no requests are being pushed back into a queue.

[reta]

Thanks for the detailed comparison @reta!

Thanks @finnegancarroll !

I'm curious to hear your thoughts on which HTTP2 features we expect to deliver the most impact. My initial impression is multiplexing is the biggest deal, allowing the server to re-use a single connection to handle multiple requests from a client.

I would agree with you, that is something we should get out of the box now with respect to JVM based implementations (at least) since clients do support HTTP/2 out of the box. More importantly, I would love to see streaming to have first class support in transports: we do have it now (with transport-reactor-netty4) but it is mostly behind the scene. With gRPC and Apache Arrow on the horizon, that would be tremendously useful I think.

One of the questions I would like to have an answer for rather soon-ish is if we could swap the transport-netty4 with transport-reactor-netty4 by default.