Data persistence in swarm (Wiki)

[nmathew98]

there is a section in the wiki about data persistence when scaling out the elastic cluster: https://github.com/deviantony/docker-elk/wiki/Elasticsearch-cluster#data-persistence

in Elastic v8 onwards, the way to do it is to set the path.data configuration as part of env vars: https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#path-settings

in docker swarm, this would be like so:

path.data configuration: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#data-path

[antoineco]

Thanks for the hint! If you feel like contributing that change the wiki can be edited by anyone. Otherwise I'll do it sometime next week.

[nmathew98]

sweet, will get around to it tomorrow

i might add a recommendation to use nginx for port mapping and securing as well?

in swarm mode the best way is to use nginx in front for ingress I think, using target, published works but the cluster nodes can find each other but can't connect (see: https://stackoverflow.com/questions/77631583/elasticsearch-cluster-using-docker-swarm-with-three-nodes-on-three-separate-serv which is the cause I think):

and using socat to forward doesn't work so well even if you use valid ssl certs on elastic, was getting errors. best way I found was to use self signed certs on elastic and put the valid certs at nginx

[antoineco]

We had a request to add this once as part of

• Added Support for Let's Encrypt Certificates #859

In my opinion deployment concerns are out-of-scope for docker-elk. There are a lot of ways to expose web applications securely over a network, and we are better off staying away from opinionated recommendations in this project, especially since the main focus is on single-host deployments via Docker Compose.
The instructions for Swarm are a leftover from a time where we did have a Swarm configuration in the project, but this configuration didn't fit very well into the project. I left the instructions in the wiki nevertheless, for educational purposes.

[nmathew98]

wiki's been updated

[antoineco]

Thank you for your contribution, much appreciated!