Release the new Mimir Alertmanager

[Rotfuks]

Motivation

We know how to roll out the Mimir Alertmanager - now we need to announce and do it.

Todo

• Migration
  • Deploy Mimir Alertmanager on giantswarm installations
  • Create custom object storage on customer installations where needed
  • Deploy Mimir Alertmanager on customer installations
  • Switch alerting to Mimir Alertmanager on giantswarm installations
  • Switch alerting to Mimir Alertmanager on customer installations
  • Ensure everything runs smooth
    • Alerts are received on OpsGenie and Slack
    • Grafana Alerting shows rules, notifications policies, and active notifications
  • Disable old Alertmanager
• Add automation in mc-bootstrap
• Document the change in the architecture in the intranet
• Annonuce the change to the mimir alert manager
  • Check if customers have alerts in place (Dominik can help :))

Outcome

We use the mimir alertmanager everywhere.

[TheoBrigitte]

Giantswarm installations deployment 🟠

• gaggle
• galaxy
• gazelle
• gerbil: #inc-2025-01-16-heartbeat-gerbil-is-expired
• glasgow: same as gerbil ?
• glean: fixed with https://github.com/giantswarm/giantswarm-configs/pull/230
• glippy
• gmc: skipping, because its ephemeral
• goat
• golem
• goose: skipping, because its ephemeral
• goshawk: skipping, because its ephemeral
• goten: skipping, because its ephemeral
• grizzly
• grouse
• mammoth

Asked Panamax for object storage creation on tamarin and leopard https://gigantic.slack.com/archives/CE92C4BST/p1737368033954959:

[TheoBrigitte]

Object storage configuration for panamax: https://github.com/giantswarm/panamax-configs/pull/56

Customer installations deployment 🟢

• alba
• alligator
• armadillo
• avocet
• cedar
• enigma
• leopard
• sardine
• tamarin
• violet
• wallaby
• whale

[TheoBrigitte]

• Mimir Alertmanager is now enable on all giantswarm installations
• Usage doc is in intranet
• Announcement to giantswarm folks was done in #news-product

Parking this issue for a week before releasing this to customer installations.

[TheoBrigitte]

I checked manually OpsGenie and Slack and alerts are being received from giantswarm installations.
Releasing Mimir Alertmanager to all installations.

[TheoBrigitte]

Deployment is ok for customer installations

Check list:

• config loaded correctly (contains team_atlas_slack)
• alerts count is greater than 0
• silences count is greater than 0
• olly-op config configured with mimir-alertmanager url
• mimir ruler configured with mimir-alertmanager url
• silences-operator configured with mimir-alertmanager-url

the script

#!/bin/bash

source ~/projects/bash-magic/colors.sh

check_alertmanager() {
  kubectl -n=mimir port-forward po/mimir-alertmanager-0 8080 1>/dev/null &
  port_forward_pid=$!
  trap "kill $port_forward_pid" EXIT
  sleep 2

  alertmanager_config_ok=false
  if mimirtool alertmanager get --id anonymous --address http://localhost:8080|grep -q team_atlas_slack; then
    alertmanager_config_ok=true
  fi

  alerts_present=false
  if [ $(curl -SsH'X-Scope-OrgID: anonymous' localhost:8080/alertmanager/api/v2/alerts|jq -r '.|length') -gt 0 ]; then
    alerts_present=true
  fi

  silences_present=false
  if [ $(curl -SsH'X-Scope-OrgID: anonymous' localhost:8080/alertmanager/api/v2/silences|jq -r '.|length') -gt 0 ]; then
    silences_present=true
  fi

  olly_ok=false
  if kubectl -n=monitoring get deploy observability-operator -oyaml|grep -qE -- '- --alertmanager-enabled=true|- --alertmanager-secret-name=observability-operator-alertmanager|- --alertmanager-url=http://mimir-alertmanager-headless.mimir.svc:8080'; then
    olly_ok=true
  fi

  mimir_ok=false
  if helm -n=mimir get manifest mimir|grep -q 'alertmanager_url: dnssrvnoa+http://_http-metrics._tcp.mimir-alertmanager-headless.mimir.svc.cluster.local./alertmanager'; then
    mimir_ok=true
  fi

  silence_operator_ok=false
  if kubectl -n=monitoring get cm silence-operator -oyaml|grep -qE -- 'address: http://mimir-alertmanager-headless.mimir.svc:8080/alertmanager|tenantId: anonymous'; then
    silence_operator_ok=true
  fi

  if $alertmanager_config_ok && $olly_ok && $mimir_ok && $silence_operator_ok && $alerts_present && $silences_present; then
    echo -e "${GREEN}OK$NC"
  else
    echo -e "${RED}FAIL$NC"
  fi

  kill $port_forward_pid
}

list_capi_installation() {
  opsctl list installations|grep -v 'giantswarm'|grep -E 'capa|capz|cloud-director|vsphere' |awk '{print $1}'
}

for i in $(list_capi_installation); do
  echo "$i "
  gx "$i" &>/dev/null
  echo -n "  verify: "
  check_alertmanager
done

[TheoBrigitte]

Alerts are received on Slack (#alert) and OpsGenie

https://giantswarm.app.opsgenie.com/reports/main

[TheoBrigitte]

Alerts are coming in OpsGenie as usual

Alerts are coming on Slack

Grafana correctly shows rules, notifications policies, and active notifications