How can I edit and add prometheus alerts using the same libsonnet file?

[mmn01-sky]

I'm currently using kube-prometheus to build and modify the prometheus alerts and rules. I now also need to add some additional custom alerts of my own however there seems to be a limitation of doing this in jsonnet in the same file.

Here is my current libsonnet file:

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  {
    _config+:: {
      namespace: 'monitoring',
    },
    
    prometheusAlerts+:: {
      groups+: (import 'prometheus-rules/custom-prometheus-rules.json').groups,
    },

    prometheusAlerts+:: {
      groups: std.map(
        function(group)
            group {
              rules: std.map(
                function(rule)
                  if std.objectHas(rule, 'labels') && std.objectHas(rule.labels, 'severity') && rule.labels.severity == "warning" then 
                      rule {
                        labels: {
                          application: "{{ if $labels.service }} {{ $labels.service }} {{ else }} UKISS GKE {{ end }}",
                          severity: "P4",
                          sparkGroup: std.extVar("SPARK_GROUP"),
                          affectedCi: "{{ $labels.spark_ci }}",                      
                        },
                      }
                  else if std.objectHas(rule, 'labels') && std.objectHas(rule.labels, 'severity') && rule.labels.severity == "critical" then 
                      rule {
                        labels: {
                          application: "{{ if $labels.service }} {{ $labels.service }} {{ else }} UKISS GKE {{ end }}",
                          severity: "P3",
                          sparkGroup: std.extVar("SPARK_GROUP"),
                          affectedCi: "{{ $labels.spark_ci }}",                         
                          },
                      }
                  else
                    rule,
                group.rules
              ),
            },
        super.groups
      ),
    },

    grafanaDashboards+:: {  //  monitoring-mixin compatibility
    'multi-cluster.json': (import 'grafana-dashboards/multi-cluster-dashboard.json'),
    },
    grafana+:: {
      dashboards+:: {  // use this method to import your dashboards to Grafana
        'multi-cluster.json': (import 'grafana-dashboards/multi-cluster-dashboard.json'),
      },

      // Adds the grafana_datasource metadata label to all grafana dashboards so they can be autodiscovered by grafana
      dashboardDefinitions+: {items: std.map(
          function(cm) cm + {metadata+:{labels+: {grafana_datasource: 'true'}}},
        super.items)},
    },

  };

{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

Here is my build.sh (modified a few things):

#!/usr/bin/env bash

# This script uses arg $1 (name of *.jsonnet file to use) to generate the manifests/*.yaml files.

set -e
set -x
# only exit with zero if all commands of the pipeline exit successfully
set -o pipefail

# Make sure to use project tooling
PATH="$(pwd)/tmp/bin:${PATH}"

# Set the Spark Group depending on environment
if [ "${ENVIRONMENT}" == "prod" ]; then
    SPARK_GROUP="UK ISS Tech Ops"
else
    SPARK_GROUP="UK IS DOST Containerisation Support"
fi

# Make sure to start with a clean 'manifests' dir
rm -rf manifests
mkdir -p manifests/setup

# Calling gojsontoyaml is optional, but we would like to generate yaml, not json
jsonnet -J vendor -m manifests "${1-example.jsonnet}" --ext-str SPARK_GROUP="$SPARK_GROUP" | xargs -I{} sh -c 'cat {} | gojsontoyaml > {}.yaml' -- {}

# Make sure to remove json files
find manifests -type f ! -name '*.yaml' -delete
rm -f kustomization

For my custom rules, i created it as a yaml file and then i convert it to json using the following command:
filename: prometheus-rules/custom-prometheus-rules.yaml

cat prometheus-rules/custom-prometheus-rules.yaml | gojsontoyaml -yamltojson > prometheus-rules/custom-prometheus-rules.json

File contents: prometheus-rules/custom-prometheus-rules.yaml

groups:
- name: "http-request.rules"
  rules:
    - alert: StatusCodes3xxHigh
      expr: sum(increase(nginx_ingress_controller_requests{status=~"3.."}[5m])) by (ingress) / sum(increase(nginx_ingress_controller_requests[5m])) by (ingress) * 100  > 50
      for: "10m"
      labels:
        severity: "warning"
      annotations:
        description: "Please check the app"
        summary: "There is a high amount of 3xx status codes being reported on {{ $labels.ingress }}"

When I run the following command, I'm presented with the following error:

./build.sh ukiss-alerts.libsonnet

Error:

+ set -o pipefail
++ pwd
+ PATH=/go/src/github.com/coreos/prometheus-operator/tmp/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ '[' poc == prod ']'
+ SPARK_GROUP='UK IS DOST Containerisation Support'
+ rm -rf manifests
+ mkdir -p manifests/setup
+ jsonnet -J vendor -m manifests ukiss-alerts.libsonnet --ext-str 'SPARK_GROUP=UK IS DOST Containerisation Support'
+ xargs '-I{}' sh -c 'cat {} | gojsontoyaml > {}.yaml' -- '{}'
ukiss-alerts.libsonnet:12:5-21 Duplicate field: prometheusAlerts

    prometheusAlerts+:: {

I'm aware that this is happening because in my ukiss-alerts.libsonnet I am specifying prometheusAlerts twice, however I can't seem to find a way to add my new rules but also modify the default rules at the same time.

Ideally, I'd want to add my new rules in first via:

    prometheusAlerts+:: {
      groups+: (import 'prometheus-rules/custom-prometheus-rules.json').groups,
    },

And once my new rules are added, I want to loop through all of the rules and modify them via:

 prometheusAlerts+:: {
      groups: std.map(
        function(group)
            group {
              rules: std.map(
                function(rule)
                  if std.objectHas(rule, 'labels') && std.objectHas(rule.labels, 'severity') && rule.labels.severity == "warning" then 
                      rule {
                        labels: {
                          application: "{{ if $labels.service }} {{ $labels.service }} {{ else }} UKISS GKE {{ end }}",
                          severity: "P4",
                          sparkGroup: std.extVar("SPARK_GROUP"),
                          affectedCi: "{{ $labels.spark_ci }}",                      
                        },
                      }
                  else if std.objectHas(rule, 'labels') && std.objectHas(rule.labels, 'severity') && rule.labels.severity == "critical" then 
                      rule {
                        labels: {
                          application: "{{ if $labels.service }} {{ $labels.service }} {{ else }} UKISS GKE {{ end }}",
                          severity: "P3",
                          sparkGroup: std.extVar("SPARK_GROUP"),
                          affectedCi: "{{ $labels.spark_ci }}",                         
                          },
                      }
                  else
                    rule,
                group.rules
              ),
            },
        super.groups
      ),
    },

Anyone able to help?

Side note: If someone has a better suggestion of how I'm adding new labels to the default alerts, please let me know. I'm aware my if, else if statements are a bit long but it's the only way i know how.

[brancz]

Yeah this is a jsonnet thing, you need to have an additional merge operation to do both the map and adding. Something along the lines of this should work:

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  {
    _config+:: {
      namespace: 'monitoring',
    },
    
    prometheusAlerts+:: {
      groups+: (import 'prometheus-rules/custom-prometheus-rules.json').groups,
    },

    grafanaDashboards+:: {  //  monitoring-mixin compatibility
    'multi-cluster.json': (import 'grafana-dashboards/multi-cluster-dashboard.json'),
    },
    grafana+:: {
      dashboards+:: {  // use this method to import your dashboards to Grafana
        'multi-cluster.json': (import 'grafana-dashboards/multi-cluster-dashboard.json'),
      },

      // Adds the grafana_datasource metadata label to all grafana dashboards so they can be autodiscovered by grafana
      dashboardDefinitions+: {items: std.map(
          function(cm) cm + {metadata+:{labels+: {grafana_datasource: 'true'}}},
        super.items)},
    },

  } + {
      prometheusAlerts+:: {
      groups: std.map(
        function(group)
            group {
              rules: std.map(
                function(rule)
                  if std.objectHas(rule, 'labels') && std.objectHas(rule.labels, 'severity') && rule.labels.severity == "warning" then 
                      rule {
                        labels: {
                          application: "{{ if $labels.service }} {{ $labels.service }} {{ else }} UKISS GKE {{ end }}",
                          severity: "P4",
                          sparkGroup: std.extVar("SPARK_GROUP"),
                          affectedCi: "{{ $labels.spark_ci }}",                      
                        },
                      }
                  else if std.objectHas(rule, 'labels') && std.objectHas(rule.labels, 'severity') && rule.labels.severity == "critical" then 
                      rule {
                        labels: {
                          application: "{{ if $labels.service }} {{ $labels.service }} {{ else }} UKISS GKE {{ end }}",
                          severity: "P3",
                          sparkGroup: std.extVar("SPARK_GROUP"),
                          affectedCi: "{{ $labels.spark_ci }}",                         
                          },
                      }
                  else
                    rule,
                group.rules
              ),
            },
        super.groups
      ),
    },
  };

{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

We tend to organize patches like the one you have for changing alert labels in separate files, that way it's much easier to keep your main file organized. Let me know if this works for you! :)

[mmn01-sky]

The above worked great! Really appreciate your help with this.

Thank you.