Proposal: PubSub Subscription Streaming

[JoshVanL]

No description provided.

[ItalyPaleAle]

How's the event ID computed?

[JoshVanL]

I believe either from the pubsub itself, or a random UUID here.

[ItalyPaleAle]

Isn't that for cloud events only?

[yaron2]

We can generate a unique id per message the same way bulk pub/sub does today

[yaron2]

Please change all endpoints (HTTP, gRPC) to be alpha

[mukundansundar]

can we have a request response sample for this endpoint?

[mukundansundar]

later on, would we want to use the same RPC for bulk subscribe as well?
If so should the TopicEventRequest stream be wrapped by another message so that we can extend to Bulk Subscribe event as well?

[JoshVanL]

Yes, we can wrap the TopicEventRequest for doing bulk messages, though don't necessarily see the use case for this.

[mukundansundar]

Suggested change

	rpc SubscribeTopicEvents(stream SubscribeTopicEventsRequest) returns (stream TopicEventRequest) {}
	rpc SubscribeTopicEvents(stream SubscribeTopicEventsRequestAck) returns (stream TopicEventRequest) {}

[JoshVanL]

Ack is not appropriate at the first message on this stream will be the initial request, not an acknowledgment.

[mukundansundar]

I was more looking for a name which covers both messages.

[mukundansundar]

@JoshVanL could you also mention what changes are needed in SDK to support this?

[JoshVanL]

@mukundansundar I'm not too sure what else needs to be added regarding SDK implementations- they need to implement the streaming client as described in the proposal .

[ItalyPaleAle]

WebSockets make things a lot more complicated, including things like authz. We support HTTP/2 now so we should just use server-sent events (since we don't need bi-di streaming). It's much simpler to implement and use, and doesn't introduce a lot more dependencies on the server.

[yaron2]

We can't use unidirectional streaming because the client needs to send back information to the server (Dapr), which is the status for each message received. From the auth perspective this wouldn't be different from how an app authenticates to Dapr today (via an access token, and unauthenticated by default) and even in browser to server scenarios a client access token is used.

[ItalyPaleAle]

The send-back can be done with regular HTTP requests. When using HTTP/2, they use the same TCP socket, so this wouldn't be an issue.

I would recommend avoiding adding another protocol as it does make a lot of things more complicated.

1. Go doesn't support WebSockets over HTTP/2, but only HTTP/1, so the app requires a separate socket: proposal: x/net/http2: support for WebSockets over HTTP/2 golang/go#49918
2. When connecting to a WS using a web browser, you can't customize the headers, so Dapr authentication wouldn't be usable (https://devcenter.heroku.com/articles/websocket-security#authentication-authorization). I understand that connecting directly to Dapr through a web browser is not officially supported, but some customers do that. Hence why I mentioned auth is more challenging.
3. WebSockets also adds complexities when dealing with proxies (this may be a problem for some Diagrid services too presumably :) ), and local development requires tooling that is WebSockets-aware.

[yaron2]

The auth remains token based auth, but it doesn't have to pass through a header in this case. The initial handshake contains the token as described in the link you posted. This seems like a standard way for client side auth and is also used by the Azure Websockets (PubSub) service, as one example.

Re: web browser, it'd actually be great to make this HTTP path usable from web browsers, where http/2 doesn't have the same support as websockets. This will provide a clearer usage pattern where backend services can connect over gRPC while web services and/or any client that doesn't have gRPC or http/2 support can use websockets.

The only issue we face with websockets is the separate socket issue. If the same port cannot be shared with the existing HTTP server and the Websockets server than this is likely a non-starter.

[ItalyPaleAle]

The auth remains token based auth, but it doesn't have to pass through a header in this case. The initial handshake contains the token as described in the link you posted.

Correct, but that's part of the application protocol we need to define, and it does add complexity.

it'd actually be great to make this HTTP path usable from web browsers, where http/2 doesn't have the same support as websockets

Support for HTTP/2 in browser is just as much as WebSockets. In any case, HTTP/2 is a nice-to-have for long-lived connections but not a requirement (using HTTP/1 would be fine, but would require one separate TCP socket).

The only issue we face with websockets is the separate socket issue. If the same port cannot be shared with the existing HTTP server and the Websockets server than this is likely a non-starter.

Separate socket, not port. So the long-lived connection requires its own TCP socket between the app and Dapr (1 TCP socket for the WebSockets long-lived stream, and then each request from the app will have its own). While in the case of HTTP/2 there's multiplexing being used, so there is at most 1 TCP socket being used (the long-lived stream can share the same socket as every request going to Dapr). This does improve efficiency. But it doesn't require a separate port - the same Go server can serve HTTP/1, HTTP/2, and WebSockets (but this only over HTTP/1 and not HTTP/2, per issue above)

[yaron2]

Ok, so if we don't need a separate port and there's a path forward for HTTP/2 support for WebSockets then I'm comfortable using WebSockets HTTP 1.1 to start with

[ItalyPaleAle]

The issue for adding WebSockets over HTTP/2 was opened in 2021 and looks like it's not being worked on. It seems that it is not a priority (understandably, as WebSockets is not as popular as it once was).

I think enabling a new communication protocol should be something not done lightly, as we'll be on the hook to support that for a very long time. There's nothing inherently wrong with HTTP/2, which is already supported and we have all tooling and test in place.

[yaron2]

The user experience that comes out of having a full-duplex bidi stream with the receive message/reply to server commands encapsulated within WebSockets is preferable to me than what the experience would be like with SSE, moreover the latter being limited to a text format. We can always change to HTTP2/SSE if we're seeing a reason to do so before moving the API to stable

[daixiang0]

Although WebSocket is not that popular, it is still very useful in full-duplex bidi stream case. I think it is a good start and we can Iterate it based on users' response and so on.

[mukundansundar]

I am open to looking at the maintainability aspect of Websockets and we could also potentially ping for the interest in the duplex stream from the community.
But this proposal looks good for gRPC as a starter.

[olitomlinson]

A suggestion... rather than encouraging end-users to make many calls to Dapr to programmatically add and remove subscriptions throughout the lifetime of the App, please consider flipping this over to a pull model.

From an end-users perspective, it would be safer for the App to maintain its target state of desired subscriptions, and when ready, then the App signals to Dapr that it requests Dapr to refresh (or pull) the subscriptions.

I've tried to express what I'm suggesting with a sequence diagram. I've kept the diagram to non-streaming to Keep It Simple, but I don't see why this pattern couldn't map to a streaming use-case as originally intended by this Proposal.

Pros

• Builds on-top of existing concept already known and used by customers (Programmatic Subscriptions)
• Safer - End users are forced to maintain a single source of truth / desired target state of subscriptions for the App
  • For ephemeral use-cases, the source of truth can be kept in memory of the App
  • For long-running use-cases (which must persist a process restart), the end-user can store the subscriptions in any persistant store, even Dapr State Store.
• Guaranteed eventual reconciliation
  • Reconciliation is just a call to /dapr/subscribe (or streaming equivalent)
  • Dapr always reconciles on init (Todays behavior)
  • New : Reconciliation can be forced by App calling /v1.0/refresh-subscriptions (or streaming equivalent) on Dapr
  • New : Optionally Dapr can periodically reconcile via /dapr/subscribe(or streaming equivalent) to ensure that progress is always moving forward to the target state.
• Debounce / Batch many changes to a PubSub component
  • Debounce many change requests into a single operation, which is safer for PubSub components such as Kafka which do not like high-frequency change (rebalancing Consumer Groups) - The debounce rate could be configurable/adaptive to accomodate different PubSub component types

Cons

• Eventually consistent
  • User needs to understand that the reconciliation will not be immediate, and may take at-most up to the next periodic reconcile moment (if periodic reconcile is enabled)

[yaron2]

A suggestion... rather than encouraging end-users to make many calls to Dapr to programmatically add and remove subscriptions throughout the lifetime of the App, please consider flipping this over to a pull model.

From an end-users perspective, it would be safer for the App to maintain its target state of desired subscriptions, and when ready, then the App signals to Dapr that it requests Dapr to refresh (or pull) the subscriptions.

I've tried to express what I'm suggesting with a sequence diagram. I've kept the diagram to non-streaming to Keep It Simple, but I don't see why this pattern couldn't map to a streaming use-case as originally intended by this Proposal.

Pros

• Builds on-top of existing concept already known and used by customers (Programmatic Subscriptions)

• Safer - End users are forced to maintain a single source of truth / desired target state of subscriptions for the App

  • For ephemeral use-cases, the source of truth can be kept in memory of the App
  • For long-running use-cases (which must persist a process restart), the end-user can store the subscriptions in any persistant store, even Dapr State Store.

• Guaranteed eventual reconciliation

  • Reconciliation is just a call to /dapr/subscribe (or streaming equivalent)
  • Dapr always reconciles on init (Todays behavior)
  • New : Reconciliation can be forced by App calling /v1.0/refresh-subscriptions (or streaming equivalent) on Dapr
  • New : Optionally Dapr can periodically reconcile via /dapr/subscribe(or streaming equivalent) to ensure that progress is always moving forward to the target state.

• Debounce / Batch many changes to a PubSub component

  • Debounce many change requests into a single operation, which is safer for PubSub components such as Kafka which do not like high-frequency change (rebalancing Consumer Groups) - The debounce rate could be configurable/adaptive to accomodate different PubSub component types

Cons

• Eventually consistent

  • User needs to understand that the reconciliation will not be immediate, and may take at-most up to the next periodic reconcile moment (if periodic reconcile is enabled)

The scope of this proposal is to solve the highly requested feature to allow apps to subscribe dynamically without needing to listen on a port, thereby simplifying the user experience, elevating security and working around network limitations. Most of not all messaging SDKs including the ones used by Dapr encourage users to add subscriptions programmatically

[olitomlinson]

@yaron2 would the sequence of what I've suggested not fit into the original framework of the proposal for streaming too?

[yaron2]

@yaron2 would the sequence of what I've suggested not fit into the original framework of the proposal for streaming too?

Not as long as it entails Dapr needing to reach the app. I think what you introduced should be coupled with introducing a pull based messaging mode to Dapr, which is highly useful in and as of itself, albeit needing a separate proposal

[olitomlinson]

Not as long as it entails Dapr needing to reach the app.

I fully understand the proposals need to not open inbound ports on the App - I just used the non-streaming example to present the sequence of events between Dapr & App. This is why I wrote :

I've kept the diagram to non-streaming to Keep It Simple, but I don't see why this pattern couldn't map to a streaming use-case as originally intended by this Proposal.

My assumption was that the same sequence of events could be transferred to grpc/WS? Was my assumption wrong? :)

[mikeee]

Suggested change

	- [ ] SDK implementations
	- [ ] .Net
	- [ ] Java
	- [ ] Go
	- [ ] Python
	- [ ] JavaScript
	- [ ] SDK implementations
	- [ ] .Net
	- [ ] Java
	- [ ] Go
	- [ ] Python
	- [ ] JavaScript
	- [ ] Rust

Please can I propose that this proposal tracks the Rust-SDK too :)

[mikeee]

TopicEventResponse refers to a response on a published message am I right in reading this as it being used as a response for the subscription response too? If so seems a bit too tightly coupled in my opinion

[artursouza]

+1 binding

[daixiang0]

+1 binding

[ItalyPaleAle]

I'm opposed until websockets are removed

[JoshVanL]

Recusing myself from voting.

[mukundansundar]

I'm opposed until websockets are removed

Is there any specific reason why web sockets should not be supported?

[mukundansundar]

+1 binding with the understanding that we might reconsider web sockets based implementation if needed.

[yaron2]

I am recusing my vote.

[yaron2]

We have enough binding votes for this proposal.