[exporter/prometheus] shouldn't resource_to_telemetry_conversion.enabled be true by default?

[mathnogueira]

Component(s)

exporter/prometheus, exporter/prometheusremotewrite

Describe the issue you're reporting

I was working on a local stack for retrieving metrics from Kubernetes and also receive metrics from our applications and publish them to our local Grafana. Everything was working fine but all metrics were strange because they had no labels (i.e. no attributes were sent to grafana).

After 4 days debugging the connections between our k8s scrapper collectors, our main collector, and our prometheus server, I discovered I was missing the resource_to_telemetry_conversion.enabled: true option in my prometheus exporter config.

#27839 was useful to discover I was missing that configuration. It's not the first time I had this problem, last year I had the same issue with the prometheusremovewrite exporter and I had forgotten about it. I was wondering why this flag is set as false as its default value. Couldn't this be set as true as this is crucial to have metrics working properly with prometheus?

[github-actions]

Pinging code owners:

• exporter/prometheus: @Aneurysm9 @dashpole
• exporter/prometheusremotewrite: @Aneurysm9 @rapphil @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

This is something we are considering, but have chosen not to do right now because resource can contain a huge number of attributes, which are often very long (e.g. process.command_args).

Instead, we provide a metric named target_info, which can be joined with your metrics to achieve similar behavior.

cc @gouthamve, since this is related to recent proposals around entities.

[ArthurSens]

@mathnogueira, please take look at Prometheus' guide about how to use target_info to add resource attributes to your queries: https://prometheus.io/docs/guides/opentelemetry/#including-resource-attributes-at-query-time

There is also effort on OTel's side to improve compatibility with Prometheus. This OTEP is a good step forward, but it will take a while for both communities to come up with implementations

[mathnogueira]

@ArthurSens thanks for the reference. This will definitely help!

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/prometheusremotewrite: @Aneurysm9 @rapphil @dashpole
• exporter/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

I think we are going to stick with the current default. There are too many resource attributes from some (java) SDKs to enable this by default. Feel free to reopen if you have further questions