Optimize proto translation for merger

[lahsivjar]

Optimizes merging of metrics by optimizing translation to proto models.

Benchmark with main:

name                         old time/op    new time/op    delta
AggregateCombinedMetrics-10    3.52µs ± 2%    1.69µs ± 2%  -51.96%  (p=0.000 n=8+10)
AggregateBatchSerial-10        8.71µs ± 2%    5.40µs ± 3%  -37.98%  (p=0.000 n=10+10)
AggregateBatchParallel-10      8.89µs ± 2%    5.74µs ± 1%  -35.38%  (p=0.000 n=10+9)
CombinedMetricsEncoding-10     3.63µs ± 1%    0.98µs ± 0%  -72.93%  (p=0.000 n=10+8)
CombinedMetricsDecoding-10     4.43µs ± 1%    0.38µs ± 0%  -91.40%  (p=0.000 n=10+10)
CombinedMetricsToBatch-10       354µs ± 0%     206µs ± 1%  -41.76%  (p=0.000 n=10+10)
EventToCombinedMetrics-10       599ns ± 1%     598ns ± 0%     ~     (p=0.151 n=8+10)

name                         old alloc/op   new alloc/op   delta
AggregateCombinedMetrics-10    4.99kB ± 0%    1.58kB ± 1%  -68.34%  (p=0.000 n=6+10)
AggregateBatchSerial-10        11.5kB ± 1%     2.7kB ± 0%  -76.24%  (p=0.000 n=10+7)
AggregateBatchParallel-10      11.6kB ± 0%     2.7kB ± 2%  -76.37%  (p=0.000 n=9+10)
CombinedMetricsEncoding-10      0.00B        660.30B ± 1%    +Inf%  (p=0.000 n=10+10)
CombinedMetricsDecoding-10     10.3kB ± 0%     1.9kB ± 0%  -81.74%  (p=0.000 n=7+10)
CombinedMetricsToBatch-10      38.6kB ± 0%    38.6kB ± 0%     ~     (all equal)
EventToCombinedMetrics-10       0.00B          0.00B          ~     (all equal)

name                         old allocs/op  new allocs/op  delta
AggregateCombinedMetrics-10      39.0 ± 0%      25.0 ± 0%  -35.90%  (p=0.019 n=6+8)
AggregateBatchSerial-10          61.0 ± 0%      37.0 ± 0%  -39.34%  (p=0.000 n=10+10)
AggregateBatchParallel-10        61.0 ± 0%      36.7 ± 2%  -39.84%  (p=0.000 n=10+10)
CombinedMetricsEncoding-10       0.00           0.00          ~     (all equal)
CombinedMetricsDecoding-10       40.0 ± 0%       7.0 ± 0%  -82.50%  (p=0.000 n=10+10)
CombinedMetricsToBatch-10         308 ± 0%       308 ± 0%     ~     (all equal)
EventToCombinedMetrics-10        0.00           0.00          ~     (all equal)

[carsonip]

draft lgtm

[axw]

Even better would be to get rid of the FromProto methods. I think we should be able to do that if we:

• create an empty CombinedMetrics when constructing a new merger, and merge the initial metrics into that (rather than using FromProto for the first metrics)
• change the Processor function type to accept an *aggregationpb.CombinedMetrics

I think ideally we would make CombinedMetrics and friends unexported types, and only use them for merging. Perhaps we should move all merging logic to an internal package. WDYT?

The second point might be a heavy lift. If it is, then I think as an interim step we could still remove FromProto by merging onto an empty CombinedMetrics with no limits.

[axw]

BTW that doesn't need to happen in this PR. I can create a followup if you think it sounds reasonable.

[lahsivjar]

That sounds reasonable, I will give it a go for point 1. Point 2 also I think is simple, will create a followup for that.

[lahsivjar]

Also, love the idea of hinding the full model (minus the *Key structs). Will give that a go to.

BTW, *Key need to stay since the proto models are not comparable. This is because the proto models have some private fields (ref).

[lahsivjar]

Discussed with @axw offline, decided to create follow up PRs.

[carsonip]

change the Processor function type to accept an *aggregationpb.CombinedMetrics

I had the same idea. Agg key map access aesbodyhash is taking some time now, but can't be sure if performance is better without maps (sorted array merge) or with a temporary map (most likely worse).

BTW, *Key need to stay since the proto models are not comparable. This is because the proto models have some private fields

Could EqualVT be useful?

[lahsivjar]

Agg key map access aesbodyhash is taking some time now

Didn't quite get this but I am working on updating the Processor to use proto type and avoid the final translation after harvest.

Could EqualVT be useful?

We use *Key as keys for our maps and map keys must be strictly comparable so probably not.

[carsonip]

lgtm!

[axw]

LGTM! We may need to do something about directly merging a binary representation into an existing HLL++ sketch at some point. I think we should be able to do that non-breaking though.