Support exemplars in Metrics

[cijothomas]

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md#exemplar

First stable release (1.9.0)

• SDK default behavior will be AlwaysOff. The spec says the default SHOULD be TraceBased. We decided to deviate from this due to performance concerns when Exemplars are enabled: [sdk-metrics] Turn exemplars on by default in prerelease builds #5545

• Expose APIs covered by the Exemplar diagnostic:

  This will allow users to enable Exemplars either AlwaysOn or TraceBased for all metrics.

• Add an environment variable for turning on Exemplars specifically for histograms:

  We feel Exemplars are most useful when combined with histograms. The current spec design is all-or-nothing. Due to the performance concerns, we have decided to expose an experiment mechanism for enabling Exemplars for histograms only. The plan is for this to be a temporary thing while we work to improve the spec to allow for this scenario via a stable API.

• Tasks:

  • Remove ExemplarFilter API: Exemplar - remove ExemplarFilter API #5333
  • Give ExemplarReservoir + View API its own diagnostic: Exemplar - expose ExemplarReservoir and enable View based config #5334, [sdk-metrics] ExemplarReservoir dedicated diagnostic and custom ExemplarReservoir support #5558
  • Add an experimental envvar for enabling exemplars for histograms: [sdk-metrics] Add experimental envvar for setting ExemplarFilter for histograms #5611
  • Remove experimental api decorations & update OTEL1002 diagnostic: [sdk-metrics] Expose Exemplars stable #5607
  • Add docs which show how to use the view api portion: [docs-metrics] Customer ExemplarReservoir and View API configuration #5624
  • Add example which shows how to implement a custom reservoir: [docs-metrics] Customer ExemplarReservoir and View API configuration #5624

Future release

• Propose spec changes to make it easier to enable Exemplars for specific metrics: [metrics] Allow setting ExemplarFilter via View API opentelemetry-specification#4014

• Propose spec changes to make it easier to enable Exemplars for specific instrument types: TODO

• Expose APIs covered by the ExemplarReservoir diagnostic:

  We felt the use case for building custom ExemplarReservoirs is unclear. We want to gather feedback before exposing something.

[cijothomas]

Have reviewed this in detail, and concluded the changes are non-trivial, and will not be part of 1.4.0 (Nov 2022) release. We expect to add beta support for this, shortly after 1.4.0 has shipped, with the goal of making it stable by 1.5.0 (no ETA for stable yet, but 3-4 months after 1.4.0)

[cijothomas]

Update:
1.5.0-alpha.1 added basic exemplar support. See https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/metrics/customizing-the-sdk#exemplars

E2E Example/Demo - https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/metrics/exemplars

Basic implementation only, more coming in then next few weeks.

[cijothomas]

Removing from 1.7.0 milestone. The spec is still experimental, so this feature remains experimental here as well. Will update the issue here once we get a firmer grasp of ETA for stability.

Meanwhile, please continue sharing feedback on the Exemplar feature!

[CodeBlanch]

I just updated the description for the plan @cijothomas and I have worked out for 1.9.0 and releasing Exemplars stable. We decided to stay close to what the spec has defined.

Users who want to turn on Exemplars can do so using the spec-defined SetExemplarFilter mechanism. If enabled, the SDK will use all the spec-defined default ExemplarReservoirs.

We feel the cost for enabling Exemplars for all metrics is too high and they are most useful for histograms. We want to enable the scenario of turning on Exemplars just for histograms. What we are going to do is introduce an experimental envvar for this. See the details below for why we decided to go this route. The key point is it will work in stable builds and doesn't require an API which would require a prerelease build.

We have decided NOT to expose ExemplarReservoir or the configuration via the View API in the first release. This will only be available in prerelease builds. We want to gather feedback first on the use cases to make sure what is being exposed will meet the needs of users. It also seems likely the default reservoirs may change so we were hesitant to expose those initially.

Details

The main design we considered was this:

• Set SDK ExemplarFilterType default to: TraceBased (spec-compliant)
• Set the default ExemplarReservoir to NoopExemplarReservoir or SimpleFixedSizeExemplarReservoir(0) (deviates from the spec-defined defaults)
• Allow users to use the View API to set ExemplarReservoir to SimpleFixedSizeExemplarReservoir, AlignedHistogramBucketExemplarReservoir, or something custom (spec-compliant).

This is a pretty big deviation from the spec. It forces users to use the View API to enable Exemplars individually for any metric where they are needed. It also requires a lot of API to be exposed publicly. There is also a concern that if users were to enable Exemplars via something like AlignedHistogramBucketExemplarReservoir and then the spec comes up with a better default algorithm, users would have to update their code to take advantage of that in future releases. ExemplarReservoir is a more advanced API. For users who want to selectively enable/disable Exemplars forcing that into the ExemplarReservoir seemed heavy-handed. We felt it better to work through the spec to enable such scenarios in well-defined APIs.

We feel the spec API is good at enabling or disabling Exemplars for everything. The only core use case we felt was not represented is enabling Exemplars for just histograms. There isn't a good way to do this via the spec. Our current thinking is this API would be useful:

meterProviderBuilder.SetExemplarFilter(InstrumentType.Histogram, ExemplarFilterType.TraceBased)

But we would need to expose that experimentally which would only be available in pre-release builds. We're going to go with an experimental envvar for histograms to get close in a way that will work in stable builds.

[prajon84]

@CodeBlanch : Does it mean that exemplars can be turned on by SetExemplarFilter(ExemplarFilterType.TraceBased), but only for histogram metrics with release 1.9.0 ?

[cijothomas]

@CodeBlanch : Does it mean that exemplars can be turned on by SetExemplarFilter(ExemplarFilterType.TraceBased), but only for histogram metrics with release 1.9.0 ?

SetExemplarFilter(ExemplarFilterType.TraceBased) will enable for all instruments, not just histograms.

[prajon84]

Well, Do you have any example with counter metrics? I do not see any relevant example

[cijothomas]

You can run the getting started doc : https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/docs/metrics/getting-started-console/Program.cs
and do SetExemplarFilter(ExemplarFilterType.AlwaysOn). This should print the exemplars for counters on the console.

[prajon84]

@cijothomas thank you but that gives only counter value and timestamp as exemplar. If I want something like traceID as exemplar, how could I do that I was expecting ExemplartFilterType.TraceBased to do that but no luck.

[cijothomas]

@cijothomas thank you but that gives only counter value and timestamp as exemplar. If I want something like traceID as exemplar, how could I do that I was expecting ExemplartFilterType.TraceBased to do that but no luck.

Have you enabled Tracing, which is pre-req for seeing TraceId in exemplars. Please open a new issue, with a minimal reproducible example showing the problem being faced.

[prajon84]

ok. I misunderstood initially, I enabled tracing:

• Add Source in Tracing

• Add Meter in Metrics

• Had to updated metrics within using block of activity.StartActivity....

• ANd, I noticed that I can see exemplars with TraceId and SpanId - if I use COnsoleExporter

• However, if I use OtelExporter or PrometheusExporter, I cannot see those exemplars.
  🤔