Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replacing Java serialization #366

Open
jabolina opened this issue Jun 19, 2024 · 1 comment
Open

Replacing Java serialization #366

jabolina opened this issue Jun 19, 2024 · 1 comment

Comments

@jabolina
Copy link
Contributor

Hey, everyone. Opening this one to continue the work on #350 and to discuss alternatives. I can tackle it when we reach a decision.

Currently, Hyperfoil utilizes the Serializable interface on objects that need serialization. This approach causes noise with unnecessary bytes[] allocation when transmitting on clustered mode. Below is some info from JFR:

Agent 1:

Class	Max Live Count	Max Live Size	Live Size Increase   	Alloc Total	Total Allocation (%)
byte[]								4.63815696E8 B	52.363017157273994 %

Agent 2:

Class	Max Live Count	Max Live Size	Live Size Increase	Alloc Total	Total Allocation (%)
byte[]								4.4464836E8 B	51.822946493282295 %

Controller:

Class	Max Live Count	Max Live Size	Live Size Increase	Alloc Total	Total Allocation (%)
byte[]								6.48228368E8 B	70.56213568204004 %

I see some alternatives here.

  1. Tackle directly the instances causing most of the allocation;
  2. Utilize something external for serialization.

Approach 1 would be more direct and likely have fewer changes (famous last words). Looking at the agent recordings, most allocations come from RequestStatsMessage and Vert.x. On the controller, the internal stack from Vert.x occupies the most part, but we have the lambdas on the ControllerVerticle#start method. I can't tell about the Vert.x stack, but I assume that's OK.

Approach 2 would likely touch several places in the code base, going from the Benchmark to the cluster messages. At the moment, I see Hyperfoil uses Jackson to expose some of the classes as JSON. We could continue with Jackson to keep things uniform or adopt an alternative. I suggest Protostream as an alternative for internal usage. We keep the public API with Jackson+JSON and internally with Protostream+Protobuf.

With Protostream, in addition to being a binary protocol, we could guarantee backward compatibility with updates. In addition to the cluster messages, we could serialize the API objects and guarantee compatibility as well. For example, updating the benchmark schema definition would be safe between versions. However, all this serialization would be internal to Hyperfoil and not exposed to the user.

Let me know what you think. I could work on it once we decide how to proceed.

@franz1981
Copy link
Contributor

I am considering to add a jmh module to Hyperfoil, for another enhancement, and it could be a nice playground to evaluate different serialization mechanisms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants