Send a notification when a health check is returning a failed status

[deviantony]

The solution should be able to notify that a node for a service is in failed state.

See: https://github.com/AcalephStorage/consul-alerts

I'm currently able to send metrics about the service_state (on/off) in Prometheus (http://prometheus.io/) using the CRN stack: https://github.com/deviantony/docker-CRN

An alert has been configured to be triggered when a service state is off. No notifications are sended though.

[deviantony]

We should use Prometheus alertmanager to send a notification (pagerduty, pushover or email support).

[deviantony]

We can ask Consul to verify the health of the application by starting it this way:

$ docker run -d -P \
    -e "SERVICE_NAME=skynet_backend" \
    -e "SERVICE_8081_IGNORE=1" \
    -e "SERVICE_8080_CHECK_CMD=/tmp/health-check.sh" \
    -e "SERVICE_8080_CHECK_INTERVAL=15s" \
    backend \
    java -jar /tmp/backend.jar

The application will parse the content of the /heatlh endpoint exposed on the 8081 and if the content is != 'UP' the service will be stated in warning on Consul and remove from the HAProxy pool.

The 8081 is not exposed on via Docker, check the result of docker inspect -f '{{ .NetworkSettings.Ports }}' container_name to retrieve the port.

Useful for chaos testing.

[deviantony]

https://github.com/prometheus/alertmanager is compliant with Slack ! We should try to send notification to the service.