[Epic] A series of improvements

[liorze]

Hello,

The repository is in maintenance mode, and currently the elasticsearch framework still lakes core features, giving no substantial added value over the alternative of running the cluster in docker via Marathon.

I think we should focus on the following features:

• Scaling up and down. Currently, the framework does not run any elasticsearch-specific logic when scaling the cluster.
  • Scaling up/down: should set the quorum size to #nodes/2+1 via the following API: https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-update-settings.html
    This will prevent a splitbrain issue.
  • Scaling down: use the allocation filtering API to drain all the shards from the node, then wait until there are no shards on that node (_cat/shards API may be helpful here). Only then it is safe to kill the task.
• Rolling-upgrades. (Restart endpoint to facilitate updates and upgrades #537) We should allow rolling-upgrades as documented in the following link: https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html.
• Persistent volumes. (Mesos persistent volumes #306) We should support mesos persistent volumes, this will have the following benefits:
  • If an elasticsearch instance fails, but the physical machine still runs, we can re-schedule a task on the same machine (Stickiness).
  • It will enable to shutdown the whole cluster and then start it again without any data lose while using local storage for performance reasons.
• Master, client and data node. (Support for different node types (data, tribe, http only, ingest (starting from vers. 5)) #578) We should be able to separately scale each node type.

I think that these are the main near-term concerns. Please share your thoughts!

[frankscholten]

Sounds good. These are indeed the important new improvements.

Perhaps rolling upgrades should be implemented first so the other features can be tested properly.

[frankscholten]

Added rolling upgrades #537 to the 1.1.0 milestone.

[tobilg]

Could you say something about the project's status, @liorze and @frankscholten? Thanks!

[frankscholten]

Hi @tobilg. Indeed the project has been in maintenance mode. I haven't been active lately because of other commiments.

I want to pick things back up again so I will take care of the current PRs.