Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrate prometheus.exporter and discovery components #1443

Open
ptodev opened this issue Aug 9, 2024 · 3 comments · May be fixed by #1826
Open

Integrate prometheus.exporter and discovery components #1443

ptodev opened this issue Aug 9, 2024 · 3 comments · May be fixed by #1826
Labels
enhancement New feature or request needs-attention

Comments

@ptodev
Copy link
Contributor

ptodev commented Aug 9, 2024

Request

prometheus.exporter components tend to not take targets as "input" attributes. Some do (e.g. prometheus.exporter.snmp and prometheus.exporter.blackbox), but others such as prometheus.exporter.kafka simply have an attribute for the URL of the system they are monitoring (kafka_uris). It would be nice to leverage the discovery components when working with prometheus.exporter, so that the metrics can have extra labels.

The simplest way I can think of is to have an additional argument for every prometheus.exporter that takes in targets. For example, prometheus.exporter.kafka can have a kafka_targets argument. kafka_targets must contain a special __address__ label with the Kafka API URL, and it may contain various additional labels which can be passed down to prometheus.scrape.

An alternative approach for prometheus.exporter.kafka would be to leave the existing kafka_uris attribute as the only attribute which can contain Kafka API URLs, and instead we can have some sort of additional_labels attribute. However, it may be harder to associate the right labels with the right kafka instance.

Use case

You may want to run prometheus.exporter to monitor a Kafka or Redis running on k8s, and you may want k8s labels such as pod name.
@ptodev ptodev added the enhancement New feature or request label Aug 9, 2024
@grafana grafana deleted a comment from manoja1 Aug 12, 2024
@ptodev ptodev mentioned this issue Aug 15, 2024
Draft
@ptodev
Copy link
Contributor Author

ptodev commented Aug 19, 2024

So far I can think of two ways to do this.

Option 1: Dynamic pipelines

This option is being discussed in a dedicated proposal PR. I'm not sure how realistic this option is, because it could become quite a bad footgun.

Option 2: A purpose-built component for decorating metrics with labels from SD targets

The new component would have two types of inputs:

  • targets from discovery components
  • metrics from prometheus components

It'd add extra labels to the metrics, and then send the metrics downstream to other prometheus components. It'd work similarly to otelcol.processor.k8sattributes and its pod_association block. otelcol.processor.discovery also does something similar, and it is a very unique Alloy/Agent component which is not in the Collector.

@ptodev
Copy link
Contributor Author

ptodev commented Aug 20, 2024

Ideally we should come up with a solution which can also be reused for Loki components - #810.

@ptodev ptodev mentioned this issue Aug 23, 2024
Closed
4 tasks
@github-actions GitHub Actions
Copy link
Contributor

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it.
If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!

@ptodev ptodev linked a pull request Oct 4, 2024 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request needs-attention
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Prototype targets.merge function grafana/alloy
1 participant
@ptodev