You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hey, everyone. Opening this one to continue the work on #350 and to discuss alternatives. I can tackle it when we reach a decision.
Currently, Hyperfoil utilizes the Serializable interface on objects that need serialization. This approach causes noise with unnecessary bytes[] allocation when transmitting on clustered mode. Below is some info from JFR:
Agent 1:
Class Max Live Count Max Live Size Live Size Increase Alloc Total Total Allocation (%)
byte[] 4.63815696E8 B 52.363017157273994 %
Agent 2:
Class Max Live Count Max Live Size Live Size Increase Alloc Total Total Allocation (%)
byte[] 4.4464836E8 B 51.822946493282295 %
Controller:
Class Max Live Count Max Live Size Live Size Increase Alloc Total Total Allocation (%)
byte[] 6.48228368E8 B 70.56213568204004 %
I see some alternatives here.
Tackle directly the instances causing most of the allocation;
Utilize something external for serialization.
Approach 1 would be more direct and likely have fewer changes (famous last words). Looking at the agent recordings, most allocations come from RequestStatsMessage and Vert.x. On the controller, the internal stack from Vert.x occupies the most part, but we have the lambdas on the ControllerVerticle#start method. I can't tell about the Vert.x stack, but I assume that's OK.
Approach 2 would likely touch several places in the code base, going from the Benchmark to the cluster messages. At the moment, I see Hyperfoil uses Jackson to expose some of the classes as JSON. We could continue with Jackson to keep things uniform or adopt an alternative. I suggest Protostream as an alternative for internal usage. We keep the public API with Jackson+JSON and internally with Protostream+Protobuf.
With Protostream, in addition to being a binary protocol, we could guarantee backward compatibility with updates. In addition to the cluster messages, we could serialize the API objects and guarantee compatibility as well. For example, updating the benchmark schema definition would be safe between versions. However, all this serialization would be internal to Hyperfoil and not exposed to the user.
Let me know what you think. I could work on it once we decide how to proceed.
The text was updated successfully, but these errors were encountered:
I am considering to add a jmh module to Hyperfoil, for another enhancement, and it could be a nice playground to evaluate different serialization mechanisms
Hey, everyone. Opening this one to continue the work on #350 and to discuss alternatives. I can tackle it when we reach a decision.
Currently, Hyperfoil utilizes the
Serializable
interface on objects that need serialization. This approach causes noise with unnecessary bytes[] allocation when transmitting on clustered mode. Below is some info from JFR:Agent 1:
Agent 2:
Controller:
I see some alternatives here.
Approach 1 would be more direct and likely have fewer changes (famous last words). Looking at the agent recordings, most allocations come from
RequestStatsMessage
and Vert.x. On the controller, the internal stack from Vert.x occupies the most part, but we have the lambdas on theControllerVerticle#start
method. I can't tell about the Vert.x stack, but I assume that's OK.Approach 2 would likely touch several places in the code base, going from the
Benchmark
to the cluster messages. At the moment, I see Hyperfoil uses Jackson to expose some of the classes as JSON. We could continue with Jackson to keep things uniform or adopt an alternative. I suggest Protostream as an alternative for internal usage. We keep the public API with Jackson+JSON and internally with Protostream+Protobuf.With Protostream, in addition to being a binary protocol, we could guarantee backward compatibility with updates. In addition to the cluster messages, we could serialize the API objects and guarantee compatibility as well. For example, updating the benchmark schema definition would be safe between versions. However, all this serialization would be internal to Hyperfoil and not exposed to the user.
Let me know what you think. I could work on it once we decide how to proceed.
The text was updated successfully, but these errors were encountered: