Additional Scrap Configs for Prometheus

[robincher]

Hi ,

I am trying to implement federated Prometheus ,but have some issues discovering the target prometheus server.

On the networking side, i did some peering (yes its in the cloud) and the connectivity is working fine.

I am using the official kube-prometheus helm charts, and here are the snippets for the values.yaml that being applied when i run helm install.

I am exposing the target server with a internal load-balancer with port 80.

# vaules.yaml
additionalScrapeConfigs: 
    - job_name: 'federate'
    scrape_interval: 15s

    honor_labels: true
    metrics_path: '/federate'

    params:
      'match[]':
        - '{__name__=~"^job:.*"}'
    static_configs:
      - targets:
        - '10.96.xx.xx.80'

Going into the prometheus pods, i realised the additional scrap config is not being applied.

 kubectl exec -it prometheus-monitoring-kube-prometheus-prometheus-0 -n monitoring sh
/etc/prometheus $ cat ~/etc/prometheus/prometheus.yml
# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

Any tips on debugging is welcomed :)

Robin

[robincher]

Hi,

I managed to solve it, seems like the private networking for the cloud provider is wonky at that time :(, all good now.

Robin