Dynamic Cluster Membership

[redixhumayun]

I am trying to set up a dynamic cluster where every node knows of a seed node and tries to contact it to get the latest view of all the members in the cluster. The seed node here is 0.

The way I am going about this is to set up a cluster per node using the initialize method. So, my node with node_id 0 would be initialized as

raft.initialize(nodes)
where nodes is a BTreeMap with { (0, BasicNode0 } 

and similarly for node_id 1 as

raft.initialize(nodes)
where nodes is a BTreeMap with { (1, BasicNode1 } 

Now, I trigger a request from node 1 to node 0 to add 1 as a member of the cluster. I follow the pattern laid out in this example of doing add_learner followed by change_membership.

However, I notice that everything is fine until I call change_membership at which point node 0 establishes contact with node 1 but then they enter an infinite loop of new elections being triggered. More specifically, every VoteRequest of node 0 fails on node 1 with the following message

2024-03-01T16:41:13.681535Z  INFO openraft::engine::engine_impl: reject vote-request: by last_log_id: !(req.last_log_id(Some(1-0-3)) >= my_last_log_id(Some(1-1-1))
2024-03-01T16:41:21.953925Z  INFO openraft::raft: Raft::vote() rpc={vote:51-0:uncommitted, last_log:1-0-3}

It seems like the log check for VoteRequest is failing. But, why would that be?

I'm wondering if perhaps I'm going about setting up the initialisation wrong? I notice that the docs for the initialize method state that starting two clusters with different configs will result in a split brain situation. Does having separate member node at initialisation count as having separate configs?

If so, what is the correct way to go about doing what I want to do? I notice that the raft-kv-memstore code initialises nodes using new. Is that the way to go about it?

[github-actions]

👋 Thanks for opening this issue!

Get help or engage by:

• /help : to print help messages.
• /assignme : to assign this issue to you.

[drmingdrmer]

You have initialized the cluster twice with different config, which result in a brain split state.

To incrementally setup a cluster, just call Raft::initialize({0:BasicNode}) on node-0.
Then call Raft::add_learner({1:BasicNode}) on node-0 to add node-1 to the cluster.
Then call Raft::change_membership(...) to make node-1 a voter.

Or setup a cluster in oneshot: call Raft:initialize({0:BasicNode, 1:BasicNode...}) on node-0;

Refer to:
https://docs.rs/openraft/latest/openraft/docs/faq/index.html#how-to-initialize-a-cluster

[redixhumayun]

@drmingdrmer First off, thank you for being so responsive :)

Also, I saw your conversation on Discord with diptanuc - that helped quite a bit.

However, my only point of confusion is that if I want to start two nodes independently of one another - seed_node (node 0) and node 1, how do I start them? The independently is important to me here

I can start node 0 using initialize but then how do I start node 1 in a way that does not cause brain split and allow node 1 to communicate with node 0 to allow itself to be registered as part of the cluster?

All the examples I've seen just use add_learner on the seed node to add a new node assuming that the seed node already knows what nodes it wants to add. But what do I do if I want nodes to be able to start independently of the seed node and then send a message to the seed node to allow themselves to be discovered?

[drmingdrmer]

First of all, IMHO, two independent nodes just means brain split. They are equivalent concept, because these two nodes can propose different logs and there is no consensus between them.

Because they have different data, you can not add node-1 with non-empty data(i.e., node-1 has already proposed logs and applied) to the seed node node-0 without losing node-1's data.

If what you want is to let seed node-0 to discover another empty node to join into the cluster, it is not about Raft.
There should be registry service let node-1 to register itself and so that seed node-0 will discover such candidate nodes.

[drmingdrmer]

Closing this issue.

@redixhumayun
If you have any further question, please feel free to reopen it:)