Liveness and readiness health endpoints

[pgressa]

Hi,
I've prepared initial implementation of #4005 in #4012 .

First to summarise the means of these endpoints. Not sure whether this concept was introduced by K8s but my proposal is to be inspired by the probes logic how it defines K8s :

• livenessProbe -> life-cycle of container, corrective action is restart of the container
• readinessProbe -> life-cycle of container, corrective action is that K8s doesn't send traffic to the pod the container is in
• startupProbe -> startup of container, I guess we don't need to deal with this one, or?

During the implementation I came up for some topics I would like to spun up a discussion about:

A) From backward compatibility POV, what indicator the /health endpoint returns
Problem statement:
Originally /health endpoint returned all health indicators found in the app context. By introducing Liveness and Readiness qualifiers and respectively /health/liveness and /health/readiness endpoints, it is expected that these health indicators will be adopted from now on. The adoption will take some time for sure and for some reasons for some cases the qualifiers won't be used at all. B/c of this I would like to ask:

Q: What should the /health endpoint return ?
1] All health indicators no matter it's Liveness/Readiness/Without qualifier, so it stays basically backward compatible
2] Subset of health indicators e.g. healthIndicators - livenessHealthIndicators or . healthIndicators - readinessHealthIndicators
3] something else?

Q: Should one of the liveness or readiness endpoint return also health indicators that are not qualified?
What brings me to this option is, that users may want to use new endpoints in K8s env for example, but they down't want to go trough all HealthIndicator and annotate them with new qualifiers. So the question is whether some of the endpoint should return also non-qualified indicators

B) Adoption of Liveness / Readiness qualifiers in micronaut
At this moment for 2.x there are these HealthIndicators:

Name	Module
MongoHealthIndicator.java	micronaut-mongodb/mongo-reactive
EurekaHealthIndicator.java	micronaut-discovery-client/discovery-client/
ConsulHealthIndicator.java	micronaut-discovery-client/discovery-client/
PgHealthIndicator.java	micronaut-sql/vertx-pg-client
MySQLHealthIndicator.java	micronaut-sql/vertx-mysql-client
JasyncHealthIndicator.java	micronaut-sql/jasync-sql
RedisHealthIndicator.java	micronaut-redis/redis-lettuce/
Neo4jHealthIndicator.java	micronaut-neo4j/neo4j-bolt/
ElasticsearchHealthIndicator.java	micronaut-elasticsearch/elasticsearch
DiscoveryClientHealthIndicator.java	micronaut-core/management
DiskSpaceIndicator.java	micronaut-core/management/
JdbcIndicator.java	micronaut-core/management
KafkaStreamsHealth.java	micronaut-kafka/kafka-streams/
KubernetesHealthIndicator.java	micronaut-kubernetes/kubernetes-discovery-client
RabbitMQHealthIndicator.java	micronaut-rabbitmq/rabbitmq/
NatsHealthIndicator.java	micronaut-nats/nats/
CassandraHealthIndicator.java	micronaut-cassandra/cassandra/

Q: Do we want to migrate micronaut health indicators?
Q: If we do, do we want to move them all under some specific qualifier like Readiness or do we want to distinguish them per specifically?

[graemerocher]

Q: What should the /health endpoint return ?

I would say /health should return a combination of both readiness and liveness

Q: Should one of the liveness or readiness endpoint return also health indicators that are not qualified?

/health/readiness will return just the readiness probes and /health/liveness just the liveness probes

Q: Do we want to migrate micronaut health indicators?

I think current health indicators should all be regarded as readiness

A user can optionally define a liveness probe that triggers a restart of the container themselves. In general most health indicators point to databases or messaging streams and if those go down there a restart will not resolve the situation.

Using this approach it would also mean that existing health indicators don't necessary need to be changed, although we should recommend users annotate them with @Readiness for documentation purposes.

Q: If we do, do we want to move them all under some specific qualifier like Readiness or do we want to distinguish them per specifically?

I don't think we need to require moving them but it should be recommended in the documentation for 2.1

[pgressa]

Q: What should the /health endpoint return ?
I would say /health should return a combination of both readiness and liveness
Ok, all of them will be returned

Q: Should one of the liveness or readiness endpoint return also health indicators that are not qualified?
/health/readiness will return just the readiness probes and /health/liveness just the liveness probes
Q: Do we want to migrate micronaut health indicators?
I think current health indicators should all be regarded as readiness

Ok this basically means that all health indicators except Liveness will be returned by /health/readiness endpoint, namely:

• all micronaut and user indicators without qualifier
• all micronaut and user indicators with Readiness qualifier

Ok will update documentation for 2.1 with recommendation to use those qualifiers.

[pgressa]

here's the change.. d7687cf