Performance testing with various traffic profiles for 4.X

[andimullaraj]

I would like to see an assessment of helidon 4.x server performance in various traffic profiles. For this I am suggesting an http client-server test where clients can arrive at preset frequencies, and send a preset number of requests, with preset payload sizes, consume the responses, then disconnect; whereas the server responds with preset payload sizes, within a preset simulated delay. The payload is also checked by the server and client to be the correct one. Suggesting that the clients run in one machine and the server in a separate machine to better segregate CPU metrics. Would like to capture the metrics on errors, and latencies for each permutation scenario. Metrics on CPU usage on both client and server.

This ask is to mainly get an idea on how the server performs in scenarios with high connection rate (50/sec) and active-connection count (30k) for moderate request/response http traffic sizes.

Environment Details

Server:

• Helidon Version: 4.X
• Helidon SE
• JDK version: 21+
• OS: Linux
• HW: Intel-based 8 CPU, memory 16GB

Client (to profile helidon client, otherwise any client):

• Helidon Version: 4.X
• JDK version: 21+
• OS: Linux
• HW: Intel-based 8 CPU, memory 16GB

Traffic Profiles (marking with * all the optional ones)

Any permutation of the following:

• Client Connection Rate: {1, 10, 50} connections/second
• Connection Type: {Plain *, SSL, ProxyProtocolV2+SSL *}
• Http-Protocol-Version: {1.1, 2 *}
• Requests Per Connection: {1, 10, 100}
• Request-Headers-Size: 16kb
• Request-Body-Size: {1b, 16kb, 1mb, 10mb *, 100mb *}
• Response-Headers-Size: 16Kb
• Response-Body-Size: {16kb, 256kb, 1mb, 10mb, 100mb *}
• Server-Simulated-Delay: {0ms, 10ms, 50ms *}

Individual test lifecycle

Generate clients for the first 10 minutes of each profile, then finish the test when all the responses for the corresponding client requests have been received, and connection has failed or is closed.

Filtering out of some traffic profiles

• A traffic profile that generates more than 100gb of traffic should not be exercised (as in: 50 connections/sec sending 100 requests of 100mb each for 10 minutes).
• If this is still too many profiles, we can reduce that number by trimming the outer values (ex: Response-Body-Size: {256kb, 1mb, 10mb} or even {1mb})

Metrics captured

for each Profile
capture from client:

• Total tried, successful and failed connections versus expected (successful + failed = tried; tried <= expected)
• Time in milliseconds for the connections to be established
• Send rate in mbps for all successful requests
• Send rate in mbps for all (successful + failed) requests
• Receive rate in mbps for all successful responses
• (Less important) Total tried requests versus expected (there are no failed requests, just connections; tried <= expected)
• (Less important) Total sent bytes versus expected
• (Less important) Total received bytes versus expected
  capture from client and server:
• CPU Use
• Possibly CPU-Core Use

[trentjeff]

Related to #7672 above when mentioning "ProxyProtocolV2+SSL *" (nice-to-have for this test)

[spericas]

Initial work to test these scenarios has been completed here:

https://github.com/helidon-io/benchmarks/tree/main/splat

Unfortunately, not everyone has access to the workspace. If necessary, we can add more users or attach a zip file to this issue.

[dmutreja]

Curious, do we have any performance data for similar profiles (could be a subset) for 3.x?

[spericas]

Closing after latest meeting where we discussed new approach to testing performance.