Identifying origin of Alerts sent via AlertManager #2365
Unanswered
ctwilleager-alio
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
For several days now I've been struggling with getting some sort of relevant identifier into Prometheus alerts that indicate what Kubernetes cluster an alert is being sent from.
Currently, the messages in our Slack channel are largely useless because while they do indicate a problem, there's zero indication of which cluster or environment that the alert originated from. Thus, the alerts are of no use as they are not actionable.
We are using Amazon EKS with Managed Node Groups and I have explicitly tagged the managed node group with a
cluster
label. This means that every node in the cluster that is online, or will come online due to autoscaling, will have this label attached to it. I want to use this node label in AlertManager to tag every single alert with acluster
label and display that label in the alert content so the alerts we get are actually useful.The other caveat here is that I need to keep the configuration to a single
prometheus.jsonnet
file applicable to all of our clusters. I do not want to have to deal with trying to keep a dozen differentprometheus.jsonnet
files to cover environments as this is basically needless configuration sprawl.So does anyone know how I can leverage a node label on my EKS cluster as an identifier I can use in AlertManager to tag my alerts so we know where a problem is?
Beta Was this translation helpful? Give feedback.
All reactions