Core services healthchecks

This page lists the checks that should be run to validate the deployment of the core services. curl checks should be run from a container on the overlay network shared by the tested service. logs checks can be run from a manager node. The check should be tried until the timeout expires.

order	service	depends on	context	test	command	where (node label)	timeout (sec)
1	elasticsearch		single mode	cluster health should be yellow	curl -sf elasticsearch:9200/_cluster/health?wait_for_status=yellow&timeout=15s	core	15
1			cluster mode	cluster health should be green	curl -sf elasticsearch:9200/_cluster/health?wait_for_status=green&timeout=30s	core	30
2	etcd		*	endpoint health	etcdctl --endpoints "http://etcd:2379" endpoint health | grep -qw healthy	core	10
3	nats		*	api availability	curl -sf "nats:8222/subsz"	core	10
4	ampbeat	nats, elasticsearch		docker service logs amp_ampbeat 2>&1	grep -q "INFO ampbeat is running"	manager	5
5	kibana	elastisearch, ampbeat		UI availability	curl -sf "kibana:5601/app/kibana#/discover"	core	15
6	node_exporter			metrics URL	curl -sf "node-exporter:9100/metrics"	metrics	5
7	nats_exporter	nats		metrics URL	curl -sf "nats-exporter:7777/metrics"	metrics	5
8	haproxy_exporter	haproxy		metrics URL	curl -sf "haproxy-exporter:9101/metrics"	metrics	5
9	prometheus	node_exporter, nats_exporter, haproxy_exporter		status URL	curl -sf "prometheus:9090/status"	metrics	10
10	alertmanager	prometheus		metrics URL	curl -sf "alertmanager:9093/metrics"	metrics	10
11	grafana			org info	curl -sf "grafana:3000/api/org"	metrics	10
12	proxy			stats	curl -sf "http://stats:stats@proxy:1936/haproxy\?stats\;csv"	route	10
13	amplifier	nats, elasticsearch, etcd		metrics URL	curl -sf "amplifier:5100/metrics"	core	10
14	gateway	amplifier		docker service logs amp_gateway 2>&1	grep -q "gateway successfully initialized"	manager	5
15	agent	nats		docker service logs amp_agent 2>&1	grep -q "Connected to nats streaming successfully"	manager	5
16	portal	gateway		home page	curl -sf "http://portal/"	core	5