OTel Metrics with no heap allocation

[cijothomas]

Opening a parent issue to track making it possible to do OTel Metrics with zero heap allocation. Since Metrics are always pre-aggregated and exported once every few seconds in background, the main goal is to achieve zero allocation in the hot path of reporting measurements.

This requires support from

1. OTel Metrics API - the API accepts slice of KV, so users can pass stack allocated KV pair array.
2. OTel Metrics SDK - The Sdk is currently allocating and copying to Heap in hot path. There is a way to avoid this completely for existing timeseries (the common case is updating existing points, not creating new ones). For non-hot path (i.e Exporting), it is also possible to avoid allocation by re-using the memory for each export.
3. Metric Exporter - It maybe possible to reuse the buffer required to hold serialization data

The overall flow in hot path would be:
user created stack allocated array of KV pairs -> OTel Metric API -> OTel Metric SDK -> Use the slice to do lookup and find the metric point to update -> Update

After the initial "hydration" phase, then measurements with same KV pairs require no allocation, as SDK simply needs the slice to perform lookup. If a previously unseen (or not seen for at least one cycle) measurement is provided, it'd need allocation.

Also, need some tooling to verify the above claims. Zero heap hot path is achieved on prototyping separately (the prototyped code will be used as reference to refactor large parts of Metrics SDK implementation shortly).

[cijothomas]

We have now achieved the goal of no heap allocation in the hot path. This will be part of the upcoming release (~Aug 30th).

The export/collect path is still being worked on, so this issue is not closed, but this is in a future release.

[utpilla]

Related issue #1740.