[prometheusremotewriteexporter] reduce allocations in createAttributes

[edma2]

Description:

While profiling the collector, we found that the createAttributes function was responsible for a significant chunk of allocations (30-40%) which was leading to a high CPU usage spent in GC.

createAttributes is responsible for converting attributes of a given data point to Prometheus labels. For simplicity, it allocates a new labels slice for every data point. We found that reducing allocations here significantly reduced GC time in our environment (in some deployments as much as ~50%).

The strategy in this PR is to reuse the slice array as much as possible. The backing array will automatically resize as needed (batching with a batch processor will effectively set an upper bound). Note: we don't need to synchronize access to this (e.g. sync.Pool) since the exporter is configured with 1 consumer.

Link to tracking Issue:

Testing:

Modified unit tests and ran benchmarks locally.
Works in our production environment.

benchstat output

cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
                    │ /tmp/old.txt  │            /tmp/new.txt             │
                    │    sec/op     │   sec/op     vs base                │
CreateAttributes-16   1010.0n ± 25%   804.0n ± 7%  -20.40% (p=0.000 n=10)

                    │ /tmp/old.txt │            /tmp/new.txt            │
                    │     B/op     │    B/op     vs base                │
CreateAttributes-16    371.00 ± 0%   91.00 ± 0%  -75.47% (p=0.000 n=10)

                    │ /tmp/old.txt │            /tmp/new.txt            │
                    │  allocs/op   │ allocs/op   vs base                │
CreateAttributes-16     7.000 ± 0%   5.000 ± 0%  -28.57% (p=0.000 n=10)

Documentation:

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: edma2 / name: Eugene Ma (9c0b9b9, 3333130, 315b11d, c570b75, 863436d, 16f3fbd, 62cfc35, b4b80d4, 2f5c758, 928529a, f184393, cf66976, f13e02a, e7ce10e, 31186a8, 185c721, f276037, dbb42e9, 8108fab, e407444)

[dashpole]

cc @jmichalek132 @ArthurSens

[dashpole]

Should we emit a warning log or something if someone calls FromMetrics without Resetting?

[edma2]

Moved the reset to always be called inside FromMetrics so this is no longer a user concern.

[ArthurSens]

Now that the user doesn't need to call reset, should we remove this part of the comment?
// Each FromMetrics call should be followed by a Reset.....

[ArthurSens]

Note: we don't need to synchronize access to this (e.g. sync.Pool) since the exporter is configured with 1 consumer.

We don't plan to keep this forever, right? Ideally we'll be able to shard this to improve throughput, we're just hardcoding this to 1 because OTel's exporter helper doesn't ensure ordering.

On the other hand, I agree that we shouldn't block optimizations based on something we want to do in the future 😬. @edma2, knowing that we'll eventually shard the output, any suggestions on how to do this without sacrificing your optimization?

[jmichalek132]

Note: we don't need to synchronize access to this (e.g. sync.Pool) since the exporter is configured with 1 consumer.

We don't plan to keep this forever, right? Ideally we'll be able to shard this to improve throughput, we're just hardcoding this to 1 because OTel's exporter helper doesn't ensure ordering.

On the other hand, I agree that we shouldn't block optimizations based on something we want to do in the future 😬. @edma2, knowing that we'll eventually shard the output, any suggestions on how to do this without sacrificing your optimization?

I wonder also since you can have multiple pipelines with multiple remote write exporters (i.e. sending data from dev cluster to 2 destination, dev and prod) if that would break this too.

[edma2]

On the other hand, I agree that we shouldn't block optimizations based on something we want to do in the future 😬. @edma2, knowing that we'll eventually shard the output, any suggestions on how to do this without sacrificing your optimization?

@ArthurSens my initial thought here is maybe wrap things in a sync.Pool but I'm not sure how that might affect performance. I'm kinda leaning to making that a future change but I can think about it more.

I wonder also since you can have multiple pipelines with multiple remote write exporters (i.e. sending data from dev cluster to 2 destination, dev and prod) if that would break this too.

@jmichalek132 Each exporter would have its own instance of PrometheusConverter so I think it would be ok.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[edma2]

@ArthurSens @dashpole @jmichalek132 I addressed comments and also changed the implementation so it's now in a sync.Pool. This now supports concurrent access from the exporter class in case it ever supports more than 1 worker at a time. Please take a look!

[ArthurSens]

Awesome edma! I'm struggling a bit to find time to review this one, just wanted to let you know that this is on my list :)

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[github-actions]

Closed as inactive. Feel free to reopen if this PR is still being worked on.

[edma2]

@atoulme I've updated my branch and merged with main but I'm not seeing my changes reflected here. Can you please re-open this PR?

[ArthurSens]

Hi @edma2 , thanks for continuously coming back to this work! I've been struggling to find time to review this PR again, I just wanted to let you know that I'm aware of its existence and that I'm trying to allocate time for the review!

[atoulme]

I see one more conflict on the code - please resolve and mark ready for review again.

[ArthurSens]

@edma2, to be fully transparent and respectful of your time, let's put the work here on hold for a while. The optimization you're doing is excellent, but it's also not easy to understand without paying a decent amount of attention.

The team's priority right now is adhering to the OTel->Prometheus specification and implementing version 2 of the Remote Write Protocol, which is creating many conflicts in your PR.

I'm a bit concerned that what I'm saying will put you in a bad mood, and I apologize for that, but I think it would be worse if I don't say anything and let you keep resolving Merge conflicts every single week 😕

[edma2]

@ArthurSens no problem, I totally understand if there are higher priorities right now. Thanks for giving me a heads up. Do you know when would be a good time to revisit the PR? Also, if splitting the PR into smaller pieces would make reviewing it easier, I can do that.

[ArthurSens]

I'd say that after we solve #33661, it should be a good time to get back to this. We shouldn't see multiple merge conflicts after that

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[github-actions]

Closed as inactive. Feel free to reopen if this PR is still being worked on.