[APM] proposal: fine-grained otel supported metrics detection #180885
Labels
enhancement
New value added to drive a business result
OpenTelemetry
Team:obs-ux-infra_services
Observability Infrastructure & Services User Experience Team
In APM Service metrics in Java, we currently provide two variants:
In order to detect which version should be displayed and which metrics to query, the
has_otel_process_metrics
request is made and a boolean value is returned from the presence or not of some known OTel metrics.The problem here is that the OTel metrics is a moving target:
process.runtime.jvm.
namespace, which is what we rely on until Update APM dashboards with stable JVM OTel metrics #174445 is fixed.agent.name
oragent.version
fields.jvm.*
namespace (semconv)We should be able to provide a dedicated "portable dashboard" for any of the following configurations, ans possibly more in the future:
process.runtime.jvm.*
namespace (1)jvm.*
namespace (2)jvm.*
namespace (3)Each variant would only be displayed when there are matching metrics, and the dashboard selection process would be implemented in a single easy to update function that takes the list of available metrics as input. This function would also provide a heuristic to select which variant has the priority when there is a mix (for example in the case of agents with and without stable JVM metrics in java).
This approach would be relevant for both the Java agent and all of the other agents, which will also have this "moving target" problem as the metrics definition evolve.
In addition to the name of the metrics, the way the data is structured might also change, so we should also query and include known labels that provide breakdown for a given metric in order to select the appropriate dashboard variant.
For example, when taking the "used heap memory" metric, we have different ways to represent it:
jvm.memory.heap.used
orjvm.memory.non_heap.used
.process.runtime.jvm.memory.usage
withlabels.type
that containsheap
ornon_heap
jvm.memory.usage
withlabels.jvm_memory_type
that containsheap
ornon_heap
jvm.memory.usage
could havejvm.memory.type
that containsheap
ornon_heap
.The text was updated successfully, but these errors were encountered: