AJ-1550: logging and health check for HybridConnectionListener #56
Conversation
| hybridConnectionListener.setOfflineHandler( | ||
| t -> logger.warn("HybridConnectionListener is now offline")); | ||
| hybridConnectionListener.setOnlineHandler( | ||
| () -> logger.warn("HybridConnectionListener is now online")); |
There was a problem hiding this comment.
Maybe also have some custom counter metric emitted?
There was a problem hiding this comment.
I like this idea! It definitely blows up scope of this PR, though, since prometheus and metrics are not currently enabled in this codebase. If we wanted to enable prometheus/metrics, I'd suggest blowing up scope even further and upgrade to Spring Boot 3 first, since a bunch of metrics stuff changed between 2 and 3.
| @@ -98,5 +98,8 @@ management: | |||
| health: | |||
| enabled: true | |||
| probes.enabled: true | |||
| group: | |||
| liveness: | |||
| include: livenessState,hybridConnectionListenerHealth | |||
There was a problem hiding this comment.
the liveness probe currently returns the abbreviated {"status": "UP"} or {"status": "DOWN"}. We could add show-details: ALWAYS here so it shows the individual statuses of each of the livenessState and hybridConnectionListenerHealth subsystems.
davidangb
changed the title
|
| "Liveness state should be CORRECT", | ||
| applicationAvailability.getLivenessState(), | ||
| equalTo(LivenessState.CORRECT)); | ||
| ResultActions result = mvc.perform(get("/actuator/health/liveness")); |
There was a problem hiding this comment.
Is result leftover/redundant code? It looks identical to livenessResult but with a less specific name.
Jira Ticket: https://broadworkbench.atlassian.net/browse/AJ-1550
Summary of changes:
Adds a new
hybridConnectionListenerHealthhealth check, via Actuator, to the k8s liveness probe. This health check echoes the state ofHybridConnectionListener.isOnline(), which is part of theazure-relaylibrary.Why
It seems that if the listener runs into certain errors - specifically, we've seen problems with
java.util.concurrent.TimeoutException: DNS timeout 15000 ms- the listener's connections hang and stop responding.This PR adds the listener's online state to the actuator health check. In theory, this would allow k8s to kill the listener pod and restart a new pod. This is drastic, but it could kick the listener connection back to a good place.
Testing strategy
HybridConnectionListener.isOnline()affects the liveness probeHybridConnectionListener.isOnline()is reliable and goes down in the use cases we are seeing. See this comment