How to expose ingress-controller metrics?

[mosheavni]

What did you do?
Just installed the operator stack via helm.

Did you expect to see some different?

Environment

• Prometheus Operator version:
  quay.io/prometheus-operator/prometheus-operator:v0.44.0

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-13T18:42:23Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

Digital Ocean managed k8s

Just a newb question, I installed the stack, I have an nginx ingress-controller and I want to add the nginx metrics and also a dashboard to my grafana.
How can I do it? Is there a better place to ask this question? Can you refer me to a user guide on how to get started?
Ideally I'd like to persist this data so adding through the UI might work but I want it to be through a manifest of some CRD, dunno how and where.

Thanks!

[coderanger]

If you installed ingress-nginx via Helm, you can enable the controller.metrics.enabled and .controller.metrics.serviceMonitor.enabled values (https://github.com/kubernetes/ingress-nginx/blob/7d6c958c9d89cf5e5d8ae872624adaa912418100/charts/ingress-nginx/templates/controller-servicemonitor.yaml#L1)

If you didn't you can write a similar ServiceMonitor resource yourself that matches your ingress-nginx deployment. Just make sure the endpoint has port: metrics, defaults should work for the rest of the endpoint settings.

[mosheavni]

If you installed ingress-nginx via Helm, you can enable the controller.metrics.enabled and .controller.metrics.serviceMonitor.enabled values (https://github.com/kubernetes/ingress-nginx/blob/7d6c958c9d89cf5e5d8ae872624adaa912418100/charts/ingress-nginx/templates/controller-servicemonitor.yaml#L1)

If you didn't you can write a similar ServiceMonitor resource yourself that matches your ingress-nginx deployment. Just make sure the endpoint has port: metrics, defaults should work for the rest of the endpoint settings.

Cool, I enabled it and now I can get servicemonitor object in the ingress namespace.
What's the next step? How can I verify the metrics are being received? How to I visualize it with a dashboard?
Is there a dynamic way to add dashboard? The official docs link to here: https://github.com/kubernetes/ingress-nginx/tree/master/deploy/grafana/dashboards, how can I load them ?

Thanks.

[mosheavni]

For those googling, here's how to configure nginx-ingress controller to send metrics to prometheus using the ServiceConfig resource. Note the additionalLabels are required if the prometheus stack was deployed via helm:

controller:
metrics:
  enabled: "true"
  serviceMonitor:
    enabled: "true"
    namespace: monitoring
    additionalLabels:
      release: prometheus-stack

Now my question remains, how do I add dashboard and persist them?

[paulfantom]

how do I add dashboard and persist them?

That depends on how you deployed your dashboarding solution. Prometheus-operator doesn't have any dashboarding solution built-in.

[mosheavni]

how do I add dashboard and persist them?

That depends on how you deployed your dashboarding solution. Prometheus-operator doesn't have any dashboarding solutions built-in.

I deployed with helm this stack:
https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

I noticed the dashboards are mounted to /tmp/dashboards but on the pod spec I can see that it is an emptyDir so I believe the sidecar is somehow loading the dashboards into that dir.
Unfortunately, this MD fails to properly describe how and where to add new dashboards.

[mosheavni]

Got it.
Adding a configmap with the label:

      grafana_dashboard: "1"

Will be caught by the container: grafana-sc-dashboard and loaded to grafana.
Described (poorly) here:
https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml#L621

[guanzo]

Took me a while to figure this out, posting my experience here.

https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#prometheusioscrape

By default, Prometheus discovers PodMonitors and ServiceMonitors within its namespace, that are labeled with the same release tag as the prometheus-operator release.

I installed kube-prometheus-stack in the kube-prometheus namespace, and the default release label is kube-prometheus. So my nginx ingress helm values should be

  metrics:
    enabled: true
    serviceMonitor:
      namespace: kube-prometheus
      enabled: true
      additionalLabels:
        release: kube-prometheus     

Now find a dashboard json you want to use, for example https://github.com/kubernetes/ingress-nginx/blob/master/deploy/grafana/dashboards/nginx.json.

Create a configmap with the label grafana_dashboard: "1", and paste that json as the configmap data.

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    grafana_dashboard: "1"
  name: nginx-ingress-dashboard
  # I dont think the configmap has to be in a certain namespace, I just did it to keep
  # all the dashboards in the same namespace.
  namespace: kube-prometheus 
data:
  nginx.json: |-
    // ... paste json here

Then go to your grafana site and wait ~30 seconds, the dashboard and it's data should appear.