Messaging using TCP

[AntonWerenberg]

From the documentation, I can read that it is possible to implement Messaging between nodes when using HTTP.
Is this the same case for TCP?
Where would be a good place to start?
Does messaging in TCP have the same capabilities regarding Leader routing?

In my setup, the different nodes all generate data that should be sent to the leader for replication.

Once again thanks for creating such a nice Raft implementation

[sakno]

@AntonWerenberg , you're welcome! Messaging is the ability to exchange messages between nodes in the cluster. The communication between the client and the cluster is not a responsibility of the library, so I would place this discussion out of the scope. You can implement any protocol and choose any framework for that, e.g. ASP.NET Core.
If we talking about node-to-node communication, there are two approaches:

• Unicast communication: request-reply or one-way
• Multicast communication, mostly known as Gossip-based messaging

For Gossip-based messaging you can use HyParView membership protocol provided out-of-the-box. Unicast communication is supported for HTTP transport only. This type of communication allows to send the message from one node to another node in the cluster, or even route message to the leader node. For more information about this type of messaging, see this article.

TCP and UDP transport for Raft do not support unicast messaging. Strictly speaking, unicast/broadcast custom messages are not the part of Raft specification. However, you still can implement this functionality on top of TCP using any appropriate serialization format: MessagePack, Protobuf, JSON or even your own wire format.

And one more thing - it looks strange that you want to generate the data on the node and the node itself is a cluster member. Typically, you have dedicated Raft cluster working as distributed data storage and the set of clients. Client typically doesn't participate in replication.

[AntonWerenberg]

Okay, thank you for your suggestions. I will build a TCP based unicast messaging system.

Yes, I know it may be bit of an unusual way of using the Raft.Next. Here is some context:
I'm working on an MSc. thesis that explores the possibility of using Raft based distributed consensus to provide controller redundancy for an industrial automation control system. The nodes will be operating in a harsh environment, and the system must be able to handle unexpected breakdowns of nodes. The system will have 5 - 20 computer nodes, each with different sensors and actuators. This means that the nodes will all act as both data generators and cluster members. My hope is to be able to use raft, not for data storage, but for sensor data replication and event synchronization.

[sakno]

For IoT, there a lot of more suitable transports. One of them is MQTT. Another one is ZeroMQ. There is perfect implementation for .NET. Event synchronization can be done easily without storing the state on the operating node. You may have dedicated cluster and write distributed lock on top of Raft. Then, client can acquire or release that lock.

Anyway, if you decided to use your current architecture, it is possible to achieve your goal in two ways:

• Organize your own messaging on top of TCP/IP network stack using Socket layer
• Contribute to .NEXT and write implementation of IMessageBus for TCP/UDP transports. The same is already done for HTTP so you have live example.

[sakno]

In case of dedicated cluster, communication between devices and dedicated cluster (client-server comm) can be organized with MQTT or ZeroMQ. In the same time, node-to-node communication can still use TCP for efficiency.

[AntonWerenberg]

It might not be a bad idea to use ZeroMQ for replication, while using Raft for leader selection. I will look into that.