how prometheus operator configs with the ability of k8s service discovery

[yangvipguang]

I want to know wether it is ture。
Become sometime Prometheus scrape nothing with the annotaion configuring in the service。

[metalmatze]

Yes, it's true. Annotations are simply ignored by the Prometheus Operator.

The annotations are a bit limited (a Prometheus can only scrape one port per Pod, for example) and the Prometheus Operator introduces for that exact reason the ServiceMonitor object to allow for more fine-grained control.

[yangvipguang]

@metalmatze thanks a lot. But In this case, is it necessary to manually add serviceMonitor every time I add a type of service monitoring？Prometheus operator how to automatically discover k8s service

[metalmatze]

What exactly are you asking now? Best case scenario you can have a labelSelector in the ServiceMonitor, so that once a new service with that label is added the Service will be scraped within a few seconds.

[songrijie]

Take a look at this example. It finds all services with label wbox-product: tulip in namespace wbox-product.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tulip
  namespace: wbox-product
spec:
  endpoints:
  - interval: 60s
    path: /core/health.api
    relabelings:
    - action: replace
      regex: (.*)
      replacement: $1
      sourceLabels:
      - __meta_kubernetes_endpoint_node_name
      targetLabel: instance
    - action: labeldrop
      regex: (pod|instance)
    targetPort: 80
  jobLabel: tulip-metrics
  selector:
    matchLabels:
      wbox-product: tulip

[sstubbs]

I have the same issue. Istio 1.6 can add annotations for metrics merging https://istio.io/latest/docs/ops/integrations/prometheus/ I see custom config can be added to prometheus operator to achieve this #16 (comment)

Although this is quite an old issue. How would I do this now?

[metalmatze]

What exactly do you want to achieve?