Handle INFO messages received from the cluster

[mmmries]

@quixoten pointed out in a previous PR that when the cluster topology changes (nodes enter or leave?) nats sends INFO messages with additional information about where each node lives. This way you could do a rolling set of restarts and give the new nodes totally different addresses while keeping all clients online.

We should add support for these messages and somehow update our list of available nodes.

[mmmries]

@quixoten do you know if the username/password/tls etc settings are always required to be the same across a cluster? Seems like that's what most people will do, but just wondering if it is required

[quixoten]

It is not required. I was able to cluster two gnatsd servers with different TLS settings. One enabled and one disabled. I think the same is likely true of the auth settings because the cluster config has separate auth settings, which could be an interesting way of bridging clients in different security zones.

Not sure what this looks like to implement. Maybe some layer that will toss out invalid server uris and than crash altogether when it runs out? Could a layer above that then collect a fresh list of server uris, e.g., from etcd, and then restart the next layer down?

[mmmries]

Hmmmm, I wonder if we should make the client code handle this problem? Maybe we let the client pass in a function that is used to select a host each time we try to establish a connection?

For a basic fixed-set of nodes that function could just sample from the list each time, but for someone using etcd they could call down into that and then generate the connection settings?

Do we get notifications from the brokers when a node leaves the cluster?

[mmmries]

I tried to find some more details about this and from what I can see the INFO message for nodes joining the cluster only applies to the server protocol which is used between the brokers. But the client protocol says that INFO messages are only presented when you open a connection.

As soon as the server accepts a connection from the client, it will send information about itself and the configuration and security requirements that are necessary for the client to successfully authenticate with the server and exchange messages.

[quixoten]

I agree it's not documented very well, maybe because it's a newer feature, but it is part of the client protocol; however, the client must declare support for it in its CONNECT message. See the documentation for CONNECT.

[quixoten]

Here's where some other clients handle this:

pure-ruby-nats
go-nats
cnats
java-nats

Here is the server side implementation:

update info and broadcast to clients