Health check failings in /metrics categorized per subnet

[erwindassen]

I wanted to open an issue with this request but since this is the first time I thought it would be more courteous to start a discussion first. I'll write here following the issue template to make it easier to move there after the discussion.

Context and scope

In v1.10.0 avalanchego added support for passing subnetIDs as argument to the /health endpoint and this was a very welcome addition. Unfortunately in the /metrics endpoints we still can only see the total of health checks that are failing via the avalanche_health_checks_failing metric (although there are health checks specific to the P,X and C chains). I would like to propose to add metrics for health checks per subnet. For example by tags:

avalanche_health_checks_failing{subnet_id="11111111111111111111111111111111LpoYY"} 0
avalanche_health_checks_failing{subnet_id="2wLe8Ma7YcUmxMJ57JVWETMSHz1mjXmJc5gmssvKm3Pw8GkcFq"} 1
avalanche_health_checks_failing{subnet_id="YDLrMpW9pkHPaRgRZR5fj883YUkJEoTc7XH28L8QBCY9v8FtV"} 0
...

This would allow infra to build custom alerts from prometheus metrics that identify issues with particular subnets. It is in general a good pattern to build alerts from prometheus which is solely responsible for scraping endpoints. In contrast, having to add custom logic to call JSON RPC methods to build alerts is in general an anti-pattern.

Discussion and alternatives

• Add a metric with the appropriate tag (subnet_id) when a subnet registers itself with the /health endpoint.

Open questions

None at the moment.

[StephenButtolph]

Hmmm interesting. Currently the health checks allow a list of tags (not just the subnetID). Would there be an easy way to map that into the prometheus metrics format?

[StephenButtolph]

I suppose we could just have a metric for each tag and call it a day

[aaronbuchwald]

This seems like a good use case for prometheus labels: https://prometheus.io/docs/practices/naming/#labels.

Curious what @erwindassen thinks would be the easiest format to consume

[aaronbuchwald]

Ah I see that's exactly how you implemented it 🤝

[StephenButtolph]

How does #1579 look?

[erwindassen]

Perfect! Even better. But I assume the metrics will be one per subnet tracked right?

[erwindassen]

Thanks, I see this will be merged eventually! 🙂

[StephenButtolph]

#1579 was merged, so I'm going to close this discussion. Thanks for the recommendation!