v2: metricLabelsAllowlist for kube-state-metrics not working in k8s-monitoring-helm

[cxk314]

Hi,
I am trying to get all kube-state-metrics but specifying metricLabelsAllowlist: [pods=[*], namespaces=[*]] doesn't seem to be doing anything. I was able to partially fix it by using metricsTuning and listing metrics which I was able to identify missing but that is not robust solution as I don't know what else is missing. For example kube_namespace_labels only starter to show up once I added it in metricsTuning. I would like to get all kube-state-metrics scraped but not sure how to configure it. Here is the config I am trying:

cluster:
  name: sb-cluster

destinations:
  - name: otlp
    type: otlp
    url: https://np-grpc.np-shared.com
    processors:
      batch:
        size: 2000
        maxSize: 2000
      k8sattributes:
        # -- Kubernetes metadata to extract and add to the attributes of the received telemetry data.
        # @section -- Processors: K8s Attributes
        metadata:
          - container.name
          - deployment.environment
          - k8s.cluster.name
          - k8s.container.name
          - k8s.replicaset.name
          - k8s.namespace.name
          - k8s.pod.name
          - k8s.deployment.name
          - service.name
          - service.namespace
          - k8s.statefulset.name
          - service.instance.id
          - k8s.daemonset.name
          - k8s.cronjob.name
          - k8s.job.name
          - k8s.node.name
          - k8s.pod.uid
          - k8s.pod.start_time
      attributes:
        actions:
          - key: TENANT_ID
            action: upsert
            value: sb-cluster
      transform:
        metrics:
          resource:
            - set(attributes["CUSTOM_TENANT_ID"], "Test_Custom_Label") where attributes["k8s.namespace.name"] == "dapr-system"
        logs:
            log:
              - set(resource.attributes["TENANT_ID"], "sb-cluster")
        traces:
          resource:
            - set(attributes["CUSTOM_TENANT_ID"], "Test_Custom_Label") where attributes["k8s.namespace.name"] == "dapr-system"
    metrics:
      # -- Whether to send metrics to the OTLP destination.
      # @section -- Telemetry
      enabled: true
    logs:
      # -- Whether to send logs to the OTLP destination.
      # @section -- Telemetry
      enabled: true
    tls:
      # -- Disables validation of the server certificate.
      # @section -- TLS
      insecureSkipVerify: true

clusterMetrics:
  enabled: true
  kube-state-metrics:
    podAnnotations: {kubernetes.azure.com/set-kube-service-host-fqdn: "true"}
    metricsTuning:
      useDefaultAllowList: false
      includeMetrics:
      - kube_namespace_labels
      - kube_service_created
      - kube_deployment_created
      - kube_configmap_created
      - kube_ingress_created
      - kube_secret_created
      - kube_persistentvolumeclaim_info
      - kube_persistentvolume_status_phase
      - kube_pod_container_status_ready
      - kube_pod_container_status_waiting
      - kube_pod_container_status_terminated
    metricLabelsAllowlist:
      - pods=[*]
      - namespaces=[*]

clusterEvents:
  enabled: true

podLogs:
  enabled: true

nodeLogs:
  enabled: true

windows-exporter:
  deploy: false

selfReporting:
  enabled: false

# -- Application Observability.
# Requires destinations that supports metrics, logs, and traces.
# To see the valid options, please see the [Application Observability feature documentation](https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/feature-application-observability).
# @default -- Disabled
# @section -- Features - Application Observability
applicationObservability:
  # -- Enable gathering Kubernetes Pod logs.
  # @section -- Features - Application Observability
  enabled: true
  receivers:
    otlp:
      grpc:
        enabled: true

alloy-metrics:
  enabled: true
  liveDebugging:
    enabled: true
  alloy:
    enableReporting: false
    stabilityLevel: experimental
  controller:
    podAnnotations: {kubernetes.azure.com/set-kube-service-host-fqdn: "true"}

alloy-singleton:
  enabled: true
  liveDebugging:
    enabled: true
  controller:
    podAnnotations: {kubernetes.azure.com/set-kube-service-host-fqdn: "true"}
  alloy:
    enableReporting: false
    stabilityLevel: experimental

alloy-logs:
  enabled: true
  liveDebugging:
    enabled: true
  alloy:
    enableReporting: false
    stabilityLevel: experimental
  controller:
    podAnnotations: {kubernetes.azure.com/set-kube-service-host-fqdn: "true"}

# An Alloy instance for opening receivers to collect application data.
alloy-receiver:
  # -- Deploy the Alloy instance for opening receivers to collect application data.
  # @section -- Collectors - Alloy Receiver
  enabled: true
  liveDebugging:
    enabled: true
  controller:
    podAnnotations: {kubernetes.azure.com/set-kube-service-host-fqdn: "true"}
  extraConfig: |-
      faro.receiver "integrations_app_agent_receiver" {
        server {
          listen_address           = "0.0.0.0"
          listen_port              = 8027
          cors_allowed_origins     = ["*"]
          max_allowed_payload_size = "10MiB"

          rate_limiting {
            rate = 100
          }
        }

        output {
          logs   = [otelcol.receiver.loki.otlp.receiver]
          traces = [otelcol.processor.transform.otlp.input]
        }
      }
  alloy:
    enableReporting: false
    stabilityLevel: experimental
    extraPorts:
      - name: otlp-grpc
        port: 4317
        targetPort: 4317
        protocol: TCP
      - name: faro
        port: 8027
        targetPort: 8027
        protocol: TCP

[petewall]

If you want all metrics from KSM, you just need to set:

clusterMetrics:
  kube-state-metrics:
    metricsTuning:
      useDefaultAllowList: false

This doc shows the behavior of metrics tuning: https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring/docs/examples/metrics-tuning

So by setting useDefaultAllowList to false and setting includeMetrics, you're limiting the total number of metrics.

[cxk314]

Thank you @petewall. Somehow that doesn't work. For example, kube_namespace_labels only shows up when I do

    metricsTuning:
      useDefaultAllowList: false
      includeMetrics:
      - kube_namespace_labels

Is it possible useDefaultAllowList is ignored?

[petewall]

Ah... I found it. It's because the alloy module sets its own default allow list.
Let me fix that.

[petewall]

OK, actually, rather than overriding the defaults in the modules, try setting this:

clusterMetrics:
  kube-state-metrics:
    metricsTuning:
      includeMetrics: ["(.+)"]

Or more specifically to the labels:

clusterMetrics:
  kube-state-metrics:
    metricsTuning:
      includeMetrics: ["kube_.+_labels"]

[cxk314]

Thank you @petewall. It seems to be working. I used following:

  kube-state-metrics:
    podAnnotations: {kubernetes.azure.com/set-kube-service-host-fqdn: "true"}
    metricsTuning:
      useDefaultAllowList: false
      includeMetrics: ["(.+)"]

[petewall]

I created a new issue to create a document that explains how to get all metrics in v2. I'll fix that in that issue.