Distributed Evolution and Hazelcast replacement

[tglman]

The distributed implementation has been trough quite a lot of changes in the last few versions, we do have now a coherent way to determinate what transaction have been applied to a database, and also way to request recent partial changes if a specific database on a specific node is missing them, so in terms of the protocol that keep the general data consistent OrientDB right now is in a quite good shape, so in there is no need to design or introduce a new consensus protocol in the general transaction flow.

But this affirmation contrast a bit with the feedback of the users, that struggle to use OrientDB in a distributed setup, and is true OrientDB has still issues in distributed, and I'm going to detail a bit them, spoil ahead the issues are not in the data consensus protocol.

First major issues as today is what is called in OrientDB "distributed configuration" often users stumble on this in the form of "distributed-config.json" and "default-distributed-config.json", this data actually keep the network topology state, in the specific, what nodes are online, what databases are present on each node, and if a database in a specific node is available or not for participate to transactions.

This is quite critical information because it does affect when nodes need to check if they need to sync each other and what nodes do participate in the quorum for the data transactions.

As today though the protocol for keeping this metadata update is not consistent, is one of the protocol implementation of the various distributed protocols we implemented that is still be around, and is more or less the first that need to be redesigned, the current implementation is based on hazelcast network events and database status events, that each node catch and use to more or less concurrently try to merge in a "distributed configuration", this may sounds ok, but is not, the final distributed configuration result may be different based on the order of events applied (can't apply crdt rules to the current distributed configuration) with the result that each node may have a different view of the state of the network.

The good thing is that this events are few and not too common in the general flow of a distributed environment, so even with the not consistent protocol, if the nodes come up slowly one after the other and do not jump often offline and online is possible to have a consistent topology across the network, that allow OrientDB to process transaction correctly, the bad thing is if the topology is not correct this is used to decide the quorum, with as result quorum issues that are well known in the OrientDB issue tracker.

So the solution on this are is write a consistent protocol to manage the topology, the protocol will be likely based on the data protocol, that proved to be consistent as today, and probably also a limited use of crtd that are needed on the first bootstrap of the network when a quorum cannot be reached.

This is not the only thing that need to be evolved in the distributed implementation, another thing is the initial discovery of the nodes that participate in the network, we do rely today on a quite outdated version of hazelcast for this actually the only thing hazelcast is used today in OrientDB is node discovery and network topology management, and because network topology is going to be re-designed outside hazelcast, it may make sense to introduce something a bit more specific and more modern to handle node discovery, one good candidate on this would be Jgroups that has a quite wide support and do not require to buy in any pre-designed protocols or flows.

On the data flow side, there are improvement to be done as well, as today is missing the re-coordination of a pending transaction if the original coordinator goes offline, that may cause in case of a node going offline some transaction to stall (which is still consistent) , this can be improved but is secondary to the topology work.

Also as today the amount of data kept in the distributed logs of the nodes is not decided by the network state but on the need of the specific node, this can cause to trigger full transfer of database more often than needed, and in case of introduction of data partitioning may cause consistency issues, work will be done on making sure that there is some relation between the cleaning of the distributed log and the network state, this is also secondary on the topology work .

If anyone what to help to improve the distributed, the area that an outsider can help more is probably a new implementation of node discovery, feel free to join on this conversation with questions about it.