Maintaining/Re-establising a ClientReadableStream

[JonasDoe]

Hello,
is there a proper way to resume/re-establish the connection to a server stream when it gets interrupted?
For what I can see, when a connection gets lost, I receive an error and a status in the gprc.ClientReadableStream, both reading more or less like code: 2, details: "Http response at 400 or 500 level, http status code: 0". I can use those as triggers to initiate a reconnect attempt.
What I'm wondering:

• Is there an existing utility method which allows me to enable an automatic reconnect?
• If not:
  • How reliable is this "disconnection heuristic" I'm using atm.? Should I better listen for the error or for the status? What exactly should I evaluate? Just whether code is 2, which just means UNKNOWN? For other codes as well? Is the message (Http response at 400 or 500 level, http status code: 0") itself worth be checked additionally? Or even instead of the code?
  • Is there any way to check whether a connection could be (re-)established successfully, apart from checking for errors or status 2-reports?

Nota bene: I'm relying on TypeScript files generated with the experimental --grpc-web_out=import_style=commonjs+dts,mode=grpcwebtext flag, in case that matters.

Thanks in advance for your opinions!

[sampajano]

Hi.. :)

Just a few quick thoughts..

is there a proper way to resume/re-establish the connection to a server stream when it gets interrupted?

As far as i know there is NO way to re-establish to the same server stream once the HTTP stream is broken. (they have a 1:1 relationship AFAIU.)

Should I better listen for the error or for the status?

I'd think "error" would be a better signal for when underlying connection fails.

Just whether code is 2, which just means UNKNOWN?

Here is our error handling code, where we also translate http code to status (code). You can probably experiment and find out which ones you see in practice.

Nota bene: I'm relying on TypeScript files generated with the experimental --grpc-web_out=import_style=commonjs+dts,mode=grpcwebtext flag, in case that matters.

Different codegen flavors only affects the API and generally shouldn't affect the capability of the library. :)

---

Not sure if it answers all your questions but hope it helps :)

[JonasDoe]

Well, not sure whether "retry a RPC" really covers it.

Let's presume I've successfully established a ClientReadableStream with the server and receive callback events from there. Now, all of the sudden, the internet connection is gone for whatever reason. Alright, the server receives a client cancellation of the stream, and he removes that very client from the broadcast listeners. Then, a few seconds later, the internet connection is up again. I want my client to try to establish a new server stream connection to pick up events from there again.

But how would my client even know that the connection to the server stream is gone and it should attempt to establish a new connection to the server? The problem is that it's "just" consuming the ClientReadableStream and has to make sense of the incoming events/errors/status reports here. And so have I. ;)

[sampajano]

Let's presume I've successfully established a ClientReadableStream with the server and receive callback events from there. Now, all of the sudden, the internet connection is gone for whatever reason. Alright, the server receives a client cancellation of the stream, and he removes that very client from the broadcast listeners. Then, a few seconds later, the internet connection is up again. I want my client to try to establish a new server stream connection to pick up events from there again.

Aha i see! I think i understand your use case now.. Our are probably only in terminology 😃

But how would my client even know that the connection to the server stream is gone and it should attempt to establish a new connection to the server? The problem is that it's "just" consuming the ClientReadableStream and has to make sense of the incoming events/errors/status reports here. And so have I. ;)

I guess if there is any Error received on the client, you could consider that the "connection" to server has broken, at least from the client perspective. Is that right?

As for the server, I guess there might not be a 100% reliable way to know if the server has observed the disconnect at the exact same time the client has observed it. Although, if you have a way of making sure the client tries to "re-establish" the connection to the same server (e.g. by using a stickiness cookie), and checks some form of "session id" (e.g. you could make this part of your proto) on the server, then you'd know if it's the same client who was disconnected, and could potentially "resume" the session from there.

Does that make any sense?

[JonasDoe]

Well, from the server side, the disconnection seems to be registered rather reliable - in my tests the the (Go) Context gets cancelled properly. But ofc I can't guarantee it - in the worst case the server has to ping from time to time and remove gone connections, I guess.
When it comes to missed events on the client side - well, I think the client should re-register for the server stream and fetch missed data with a query. At least for my project (a chat of some sort) I rely on broadcasts, so there are no event pipes dedicated to one connection, and therefore there's luckily no need for a session id. :) The client just listens to the broadcasted messages for a channel, and queries older/missed messages (in case of a (re-)connection etc.).

I guess if there is any Error received on the client, you could consider that the "connection" to server has broken, at least from the client perspective. Is that right?

Well, that's exactly what I'm wondering. I mean, in theory there are 17 grpc errors and plenty of http errors (which might even be undefined aka 0 as in my tests). And there are status updates - but you advocated for errors, so I decided to ignore them for that matter.

Don't get me wrong, I don't want to ask you to solve all these issues. ;) I was merely wondering whether there is some kind of best practise here, some kind of standard (of how a broken connection looks like for the client) or even some kind of built-in functionality in grpc-web to deal with that matter. I can hardly imagine to be the first person facing these issues with a client losing connection from a server stream. :)

[sampajano]

Ahh i hear you. :)

I think the problem is that you're trying to use grpc-web streaming as a long-lived "push channel", where it's not exactly designed to solve that particular use case. As you've mentioned, you need an end-to-end keep-alive mechanism (what you refer to as "ping from time to time") to ensure the other endpoint is still alive, which is not part of grpc in general.

Although, I'm not saying it's not possible to build it on top of grpc-web, but you're gonna need to build a lot more on your application layer — e.g. including things that you asked about like session management, "fetching missing data", and also potentially sticky routing, session resuming, etc. — and those are currently a layer above what grpc-web aims to solve.

Don't get me wrong, I don't want to ask you to solve all these issues. ;) I was merely wondering whether there is some kind of best practise here, some kind of standard (of how a broken connection looks like for the client) or even some kind of built-in functionality in grpc-web to deal with that matter.

Regarding this particular question, i think you should treat ANY error received on the client as a sign of a broken connection, as i understand that an error is only received when the underlying HTTP stream is broken.

---

Hope i'm able to answer your question to some degree :)

[JonasDoe]

I see. Interesting thought that I might overburden the grpc approach here. It never occurred to me, but it make sense ... I mean, server streaming is not really what one would call a remote procedure call anyway, right? So maybe I should split my service at some point into a grcp-based one providing all kind of data (old messages, user data etc.) based on requests, and a non-grpc one which broadcasts events.

Until then I'll take all errors at the client's side as reason for a "re-connect", as you adviced. :)

Thanks again for your feedback here!

[sampajano]

You're welcome :)

I mean, server streaming is not really what one would call a remote procedure call anyway, right?

Yeah i'd say generally not.. things designed for full duplex streaming like WebSocket might suit your use case better.

So maybe I should split my service at some point into a grcp-based one providing all kind of data (old messages, user data etc.) based on requests, and a non-grpc one which broadcasts events.

Sounds like a reasonable plan :) Although, even with things like WebSocket you might still need to handle things like connectivity detection (esp. if considering proxy servers and firewalls), session management, resuming, etc. yourself, so it's your decision to make on whether to introduce the extra complexity (i.e. an additional networking stack).