Prometheus Federation
#2959
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
-
Federating Prometheus
Tony Hsieh asked about federating Prometheus on Discord.
Prometheus federation timeouts tend to happen when you're pulling too much data. Are you seeing broken pipe messages in your Prometheus logs?
A quick search indicates that a lot has been written on this topic - some folks have had better luck with Thanos as a federation solution or VictoriaMetrics via remote_write.
Increasing timeouts, reducing the number of metrics collected, or collecting the metrics at different intervals are other solutions/workarounds. The consensus seems to be that federation is not a good option if you're wanting to pull all metrics from a set of high-metric Prometheus servers.
Prometheus federation links
Beta Was this translation helpful? Give feedback.
All reactions