Merge pull request #1724 from grafana/docs/notification-policy-docs

docs: improve alerting documentation

config/manifests/bases/grafana-operator.clusterserviceversion.yaml

@@ -16,6 +16,12 @@ spec:
   apiservicedefinitions: {}
   customresourcedefinitions:
     owned:
+    - description: GrafanaNotificationPolicy is the Schema for the GrafanaNotificationPolicy
+        API
+      displayName: Grafana Notification policy
+      kind: GrafanaNotificationPolicy
+      name: grafananotificationpolicies.grafana.integreatly.org
+      version: v1beta1
     - description: GrafanaAlertRuleGroup is the Schema for the grafanaalertrulegroups
         API
       displayName: Grafana Alert Rule Group

config/samples/kustomization.yaml

@@ -6,4 +6,5 @@ resources:
 - grafana_v1beta1_grafanafolder.yaml
 - grafana_v1beta1_grafanaalertrulegroup.yaml
 - grafana_v1beta1_grafanacontactpoint.yaml
+- grafana_v1beta1_grafananotificationpolicy.yaml
 #+kubebuilder:scaffold:manifestskustomizesamples

docs/_index.html

@@ -5,7 +5,7 @@
 +++
 
 {{< blocks/cover title="Welcome to the Grafana Operator documentation" image_anchor="top" height="full" >}}
-<a class="btn btn-lg btn-primary me-3 mb-4" href="/docs/">
+<a class="btn btn-lg btn-primary me-3 mb-4" href="docs/">
   Learn More <i class="fas fa-arrow-alt-circle-right ms-2"></i>
 </a>
 <a class="btn btn-lg btn-secondary me-3 mb-4" href="https://github.com/grafana/grafana-operator/releases">

docs/docs/alerting/_index.md

@@ -0,0 +1,76 @@
+---
+title: Alerting
+weight: 13
+---
+{{% pageinfo color="primary" %}}
+Alerting resources require Grafana version 9.5 or higher.
+{{% /pageinfo %}}
+
+The Grafana Operator currently only supports _Grafana Managed Alerts_.
+
+For data source managed alerts, refer to the documentation and tooling available for the respective data source.
+{{% alert title="Note" color="primary" %}}
+When using Mimir/Prometheus, you can use the [`mimir.rules.kubernetes`](https://grafana.com/docs/alloy/latest/reference/components/mimir/mimir.rules.kubernetes/) component of [Grafana Alloy](https://grafana.com/docs/alloy/latest/) to deploy rules as Kubernetes resources.
+{{% /alert %}}
+
+
+## Full example
+
+The following resources construct the flow outlined in the [Grafana notification documentation](https://grafana.com/docs/grafana/latest/alerting/fundamentals/notifications/).
+
+They create:
+1. Three alert rules across two different groups
+2. Two contact points for two different teams
+3. A notification policy to route alerts to the correct team
+
+{{< figure src="notification-routing.png" title="Flowchart of alerts routed through this system" width="500" >}}
+
+{{% alert title="Note" color="primary" %}}
+If you want to try this for yourself, you can [get started with demo data in Grafana cloud](https://grafana.com/docs/grafana-cloud/get-started/#install-demo-data-sources-and-dashboards).
+The examples below utilize the data sources to give you real data to alert on.
+{{% /alert %}}
+
+### Alert rule groups
+
+The first resources in this flow are _Alert Rule Groups_.
+An alert rule group can contain multiple alert rules.
+They group together alerts to run on the same interval and are stored in a Grafana folder, alongside dashboards.
+
+First, create the folder:
+
+{{< readfile file="../examples/notifications-full/folder.yaml" code="true" lang="yaml" >}}
+
+The first alert rule group is responsible for alerting on well known Kubernetes issues:
+
+{{< readfile file="../examples/notifications-full/kubernetes-alert-rules.yaml" code="true" lang="yaml" >}}
+
+The second alert rule group is responsible for alerting on security issues:
+
+{{< readfile file="../examples/notifications-full/security-alert-rules.yaml" code="true" lang="yaml" >}}
+
+After applying the resources, you can see the created rule groups in the _Alert rules_ overview page:
+
+![Alert rules overview page](./overview-page.png)
+
+### Contact Points
+
+Before you can route alerts to the correct receivers, you need to define how these alerts should be delivered.
+[Contact points](./contact-points) specify the methods used to notify someone using different providers.
+
+Since the two different teams get notified using different email addresses, two contact points are required.
+
+{{< readfile file="../examples/notifications-full/contact-points.yaml" code="true" lang="yaml" >}}
+
+### Notification Policy
+
+Now that all parts are in place, the only missing component is the notification policy.
+The instances notification policy routes alerts to contact points based on labels.
+A Grafana instance can only have one notification policy applied at a time as it's a global object.
+
+The following notification policy routes alerts based on the team label and further configures the repetition interval for high severity alerts belonging to the operations team:
+
+{{< readfile file="../examples/notifications-full/notification-policy.yaml" code="true" lang="yaml" >}}
+
+After applying the resource, Grafana shows the following notification policy tree:
+
+![Notification policy tree after applying the resource](./notification-policy-tree.png)

docs/docs/alerting/alert-rule-groups.md

@@ -0,0 +1,12 @@
+---
+title: Alert Rule Groups
+---
+
+Alert Rule Groups contain a list of alerts which should evaluate at the same interval.
+Every rule group must belong to a folder and contain at least one rule.
+
+The easiest way to get the YAML specification for an alert rule is to use the [modify export feature](https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/export-alerting-resources/), introduced in Grafana 10.
+
+The following snippet shows an example alert rule group with a single alert that fires when the temperature is below zero degrees.
+
+{{< readfile file="../examples/alertrulegroups/resources.yaml" code="true" lang="yaml" >}}

docs/docs/alerting/contact-points.md

@@ -0,0 +1,18 @@
+---
+title: Contact Points
+---
+
+Contact points contain the configuration for sending alert notifications. You can assign a contact point either in the alert rule or notification policy options.
+For a complete explanation on notification policies, refer to the [upstream Grafana documentation](https://grafana.com/docs/grafana/latest/alerting/fundamentals/notifications/contact-points/).
+
+{{% alert title="Note" color="secondary" %}}
+The Grafana operator currently only supports a single receiver per contact point definition.
+As a workaround you can create multiple contact points with the same `spec.name` value.
+Follow issue [#1529](https://github.com/grafana/grafana-operator/issues/1529) for further updates on this topic.
+{{% /alert %}}
+
+The following snippet shows an example contact point which notifies a specific email address.
+It also highlights how secrets and config maps can utilized to externalize some of the configuration.
+This is especially useful for contact points which contain sensitive information.
+
+{{< readfile file="../examples/contactpoint_override/resources.yaml" code="true" lang="yaml" >}}

docs/docs/alerting/notification-policies.md

@@ -0,0 +1,15 @@
+---
+title: Notification Policies
+---
+
+Notification policies provide you with a flexible way of designing how to handle notifications and minimize alert noise.
+For a complete explanation on notification policies, see the [upstream Grafana documentation](https://grafana.com/docs/grafana/latest/alerting/fundamentals/notifications/notification-policies/).
+
+{{% alert title="Tip" color="secondary" %}}
+If you already know which contact point an alert should send to, you can directly set the [`receivers`]({{% relref "/docs/api/#grafanaalertrulegroupspecrulesindexnotificationsettings" %}}) property on the alert rule.
+{{% /alert %}}
+
+
+The following snippet shows an example notification policy routing to the `operations` or `security` team based on the `team` label.
+
+{{< readfile file="../examples/notification-policy/resources.yaml" code="true" lang="yaml" >}}

examples/notification-policy/resources.yaml

@@ -0,0 +1,24 @@
+apiVersion: grafana.integreatly.org/v1beta1
+kind: GrafanaNotificationPolicy
+metadata:
+  name: grafananotificationpolicy-sample
+spec:
+  instanceSelector:
+    matchLabels:
+      dashboards: "grafana"
+  route:
+    receiver: grafana-email-default
+    group_by:
+      - grafana_folder
+      - alertname
+    routes:
+      - receiver: grafana-email-operations
+        object_matchers:
+          - - team
+            - =
+            - operations
+      - receiver: grafana-email-security
+        object_matchers:
+          - - team
+            - =
+            - security

examples/notifications-full/contact-points.yaml

@@ -0,0 +1,26 @@
+---
+apiVersion: grafana.integreatly.org/v1beta1
+kind: GrafanaContactPoint
+metadata:
+  name: operations-team
+spec:
+  name: operations-team
+  type: "email"
+  instanceSelector:
+    matchLabels:
+      instance: my-grafana-stack
+  settings:
+    addresses: 'operations@example.com'
+---
+apiVersion: grafana.integreatly.org/v1beta1
+kind: GrafanaContactPoint
+metadata:
+  name: security-team
+spec:
+  name: security-team
+  type: "email"
+  instanceSelector:
+    matchLabels:
+      instance: my-grafana-stack
+  settings:
+    addresses: 'security@example.com'

examples/notifications-full/folder.yaml

@@ -0,0 +1,8 @@
+apiVersion: grafana.integreatly.org/v1beta1
+kind: GrafanaFolder
+metadata:
+  name: alerts-demo
+spec:
+  instanceSelector:
+    matchLabels:
+      instance: "my-grafana-stack"