Recording Connection & Disconnection events.

[msaboor35]

Overview

We are building a real-time communication system based on NATS, where the clients will connect to the 3-node NATS cluster hosted on AWS. The NATS server version is 2.10.25. Each client creates a unique consumer with filter subject patterns client.clientGroup_clientID and client.clientGroup on a stream a and consumes messages from the backend. The backend publishes messages to the whole group of clients by publishing messages on stream a with subject client.clientGroup or to some specific clients by publishing messages on stream a with subject client.clientGroup_clientID. Similarly, on stream b, the backend is consuming messages being sent by the clients. We have implemented this part without any problems.

Problem

However, we have few additional requirements as below:

We need to keep track of which client is connected, and which client is disconnected. We initially planned to use NATS System Events on the backend side to keep track of this. The plan was to listen on $SYS.ACCOUNT.*.CONNECT and $SYS.ACCOUNT.*.DISCONNECT subjects and mark the clients as connected or disconnected by checking the name in the events received. However, we have few problems with this approach.

1. As the number of clients is quite big (> 100k), we have multiple instances of backend services processing messages from NATS, and If we listen on the above NATS System Events subjects, we will get same message more than once in our system. It is not a problem as we can design our message handler to be idempotent, but the problem is it can cause unnecessarily large load considering the large number of clients.
2. We tested this approach with just 2 clients, and we were not able to reliably get the messages on our CLI running nats sub '$SYS.ACCOUNT.*.CONNECT' 2>&1 | grep client_connect | grep -v 'NATS CLI Version development'. We filter out any connection formed from the NATS CLI during this test so that we only see client connections initiated by our own client written in Golang. The connection events sometimes came quite late (around 5 minutes in 1 instance) and in 1 instance the connection event for a client never came back. Considering we were testing with just 2 clients; we fear the problem might become worse when we go to production with 100k clients.

Due to this, we changed the approach for connection event and mark the client as connected whenever we consume a message from the stream. However, this approach does not work for the disconnection event, as the client can become disconnected due to any arbitrary cause such as power loss, internet connection loss, etc.

The business requirement also requires us to filter the online/offline clients, that is why we store this information in the database along with some other client metadata. Otherwise, if the requirement was just to identify the online/offline clients, we could look at the NATS consumer (with small Inactive Threshold) to identify the client online/offline status. But this requirement makes it difficult to use this approach.

Questions

To solve the above problems, we are thinking about the following questions, which can help us in further design.

1. How can we increase the reliability of NATS System events like $SYS.ACCOUNT.*.CONNECT, if that is possible so that we can record accurate results for connection and disconnection events.
2. Are there any guarantees provided by the NATS system for its System Events?
3. Could these System Events be published on Jetstream (with WorkQueue Retention Policy)? If not, is it possible with Core NATS to filter out duplicate events, due to multiple instances of backend service running, to reduce the load on the backend?
4. Is there any other approach that we might have not considered?

If there is any additional information that is needed, please let me know.
Thank you in advance.

[msaboor35]

I realized that this can be done with NATS KV store where the clients can put some value with its unique key, and it will be expired after the set TTL. However, after checking I came to know that TTL'd messages do not get reported to watchers, yet. Is that going to be released in 2.11 release?

[derekcollison]

Yes correct and this has landed on main.

[borjaetxebarria]

Re. your original question, you can export the $SYS.ACCOUNT.*.CONNECT/DISCONNECT evens to another account, and create a stream there to capture them. Set up the stream with --no-ack as there is no point on confirming the capture.
That way you'll remove the problem of duplicate processing.
There is still a risk that some events are lost, as before the stream capture the events are just core-nats...

[borjaetxebarria]

The KV option is great, as long as the clients are willing to collaborate :-)

[borjaetxebarria]

You can also send a request to all servers to collect info on all the connections ($SYS.REQ.SERVER.PING.CONNZ) including closed ones (send payload with '{"state":1}'). But that may not be realistic in your case, with over 100k active connections!