Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Pub/sub] overview refresh #2435

Closed
wants to merge 6 commits into from
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -6,48 +6,84 @@ weight: 1000
description: "Overview of the Pub/Sub API building block"
---

## Introduction
## Publish and subscribe pattern

The [publish/subscribe pattern](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern) allows microservices to communicate with each other using messages. The **producer or publisher** sends messages to a **topic** without knowledge of what application will receive them. This involves writing them to an input channel. Similarly, a **consumer or subscriber** subscribes to the topic and receive its messages without any knowledge of what service produced these messages. This involves receiving messages from an output channel. An intermediary message broker is responsible for copying each message from an input channel to an output channels for all subscribers interested in that message. This pattern is especially useful when you need to decouple microservices from one another.
The publish and subscribe pattern (Pub/Sub) enables microservices to communicate with each other using messages for event-driven architectures.

The publish/subscribe API in Dapr provides an at-least-once guarantee and integrates with various message brokers and queuing systems. The specific implementation used by your service is pluggable and configured as a Dapr pub/sub component at runtime. This approach removes the dependency from your service and, as a result, makes your service more portable and flexible to changes.
- The producer, or **publisher**, writes messages to an input channel and sends them to a topic, unaware which application will receive them.
- The consumer, or **subscriber**, subscribes to the topic and receives messages from an output channel, unaware which service produced these messages.

The complete list of Dapr pub/sub components is [here]({{< ref supported-pubsub >}}).
An intermediary message broker copies each message from a publisher's input channel to an output channel for all subscribers interested in that message. This pattern is especially useful when you need to decouple microservices from one another.

<img src="/images/pubsub-overview-pattern.png" width=1000>

<br></br>

The Dapr pub/sub building block provides a platform-agnostic API to send and receive messages. Your services publish messages to a named topic and also subscribe to a topic to consume the messages.
## Pub/Sub API in Dapr

The service makes a network call to a Dapr pub/sub building block, exposed as a sidecar. This building block then makes calls into a Dapr pub/sub component that encapsulates a specific message broker product. To receive topics, Dapr subscribes to the Dapr pub/sub component on behalf of your service and delivers the messages to an endpoint when they arrive.
The Pub/Sub API in Dapr:
- Provides a platform-agnostic API to send and receive messages.
- Offers at-least-once message delivery guarantee.
- Integrates with various message brokers and queuing systems.

The diagram below shows an example of a "shipping" service and an "email" service that have both subscribed to topics that are published by the "cart" service. Each service loads pub/sub component configuration files that point to the same pub/sub message bus component, for example Redis Streams, NATS Streaming, Azure Service Bus, or GCP Pub/Sub.
The specific message broker used by your service is pluggable and configured as a Dapr Pub/Sub component at runtime. This removes the dependency from your service and makes your service more portable and flexible to changes.

When using Pub/Sub in Dapr:

1. Your service makes a network call to a Dapr Pub/Sub building block API.
1. The Pub/Sub building block makes calls into a Dapr Pub/Sub component that encapsulates a specific message broker.
1. To receive messages on a topic, Dapr subscribes to the Pub/Sub component on behalf of your service with a topic and delivers the messages to an endpoint on your service when they arrive.

In the diagram below, a "shipping" service and an "email" service have both subscribed to topics published by a "cart" service. Each service loads Pub/Sub component configuration files that point to the same Pub/Sub message bus component; for example: Redis Streams, NATS Streaming, Azure Service Bus, or GCP Pub/Sub.

<img src="/images/pubsub-overview-components.png" width=1000>
<br></br>

The diagram below has the same services, however this time showing the Dapr publish API that sends an "order" topic and order endpoints on the subscribing services that these topic messages are delivered posted to by Dapr.
In the diagram below, the Dapr API posts an "order" topic from the publishing "cart" service to "order" endpoints on the "shipping" and "email" subscribing services.

<img src="/images/pubsub-overview-publish-API.png" width=1000>
<br></br>

## Features
The pub/sub building block provides several features to your application.
[View the complete list of Pub/Sub components that Dapr supports]({{< ref supported-pubsub >}}).

### Cloud Events message format
## Dapr Pub/Sub API features

To enable message routing and to provide additional context with each message, Dapr uses the [CloudEvents 1.0 specification](https://github.com/cloudevents/spec/tree/v1.0) as its message format. Any message sent by an application to a topic using Dapr is automatically "wrapped" in a Cloud Events envelope, using `Content-Type` header value for `datacontenttype` attribute.
The Pub/Sub building block brings several features to your application.

Dapr implements the following Cloud Events fields:
### Sending messages using Cloud Events

Dapr Pub/Sub sends messages between services. To enable message routing and provide additional context with each message, Dapr uses the [CloudEvents 1.0 specification](https://github.com/cloudevents/spec/tree/v1.0) as its message format. Any message sent by an application to a topic using Dapr is automatically wrapped in a Cloud Events envelope, using [`Content-Type` header value]({{< ref "pubsub-overview.md#content-types" >}}) for `datacontenttype` attribute.

Dapr implements the following Cloud Events fields when creating a message topic.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We talked about having the rest of this section in its own Pub/Sub using Cloud Events topic.


* `id`
* `source`
* `specversion`
* `type`
hhunter-ms marked this conversation as resolved.
Show resolved Hide resolved
* `datacontenttype` (Optional)
* `traceparent`
* `datacontenttype` (optional)

The following example demonstrates an `orders` topic message sent by Dapr that includes a W3C `traceid` unique to the message, the `data` and the fields for the CloudEvent where the data content is serialized as JSON.

The following example shows an XML content in CloudEvent v1.0 serialized as JSON:
```json
{
"topic": "orders",
"pubsubname": "order_pub_sub",
"traceid": "00-113ad9c4e42b27583ae98ba698d54255-e3743e35ff56f219-01",
"tracestate": "",
"data": {
"orderId": 1
},
"id": "5929aaac-a5e2-4ca1-859c-edfe73f11565",
"specversion": "1.0",
"datacontenttype": "application/json; charset=utf-8",
"source": "checkout",
"type": "com.dapr.event.sent",
"traceparent": "00-113ad9c4e42b27583ae98ba698d54255-e3743e35ff56f219-01"
}
```

As another example of a v1.0 CloudEvent, the following shows data as XML content in a CloudEvent message serialized as JSON:

```json
{
Expand All @@ -62,59 +98,88 @@ The following example shows an XML content in CloudEvent v1.0 serialized as JSON
}
```

### Message subscription
#### Setting message content types

When publishing a message, it's important to specify the content type of the data being sent.
Unless specified, Dapr will assume `text/plain`.

Dapr applications can subscribe to published topics. Dapr allows two methods by which your applications can subscribe to topics:
For Dapr's HTTP API, the content type can be set in a `Content-Type` header.

- **Declarative**, where a subscription is defined in an external file,
- **Programmatic**, where a subscription is defined in the user code.
gRPC clients and SDKs have a dedicated content type parameter.

Both declarative and programmatic approaches support the same features. The declarative approach removes the Dapr dependency from your code and allows for existing applications to subscribe to topics, without having to change code. The programmatic approach implements the subscription in your code.
#### Message delivery

For more information read [How-To: Publish a message and subscribe to a topic]({{< ref howto-publish-subscribe >}}).
In principle, Dapr considers a message successfully delivered once the subscriber processes the message and responds with a non-error response. For more granular control, Dapr's Pub/Sub API also provides explicit statuses, defined in the response payload, with which the subscriber indicates specific handling instructions to Dapr (for example, `RETRY` or `DROP`).

### Receiving messages with topic subscriptions

### Message delivery
Dapr applications can subscribe to published topics via two methods that support the same features: declarative and programmatic.

In principle, Dapr considers message successfully delivered when the subscriber responds with a non-error response after processing the message. For more granular control, Dapr's publish/subscribe API also provides explicit statuses, defined in the response payload, which the subscriber can use to indicate the specific handling instructions to Dapr (e.g. `RETRY` or `DROP`). For more information on message routing read [Dapr publish/subscribe API documentation]({{< ref "pubsub_api.md#provide-routes-for-dapr-to-deliver-topic-events" >}})
| Subscription method | Description |
| ------------------- | ----------- |
| **Declarative** | Subscription is defined in an **external file**. The declarative approach removes the Dapr dependency from your code and allows for existing applications to subscribe to topics, without having to change code. |
| **Programmatic** | Subscription is defined in the **user code**. The programmatic approach implements the subscription in your code. |

For more information, read [about the subscriptions in the how-to]({{< ref howto-publish-subscribe.md >}}).

### Message routing

Dapr provides [content-based routing](https://www.enterpriseintegrationpatterns.com/ContentBasedRouter.html) pattern. [Pub/Sub routing]({{< ref howto-route-messages.md >}}) is an implementation of this pattern that allows developers to use expressions to route [CloudEvents](https://cloudevents.io) based on their contents to different URIs/paths and event handlers in your application. If no route matches, an optional default route is used. This is useful as your applications expands to support multiple event versions or special cases.

This feature is available to both the declarative and programmatic subscription approaches.

For more information on message routing, read [Dapr Pub/Sub API reference]({{< ref "pubsub_api.md#provide-routes-for-dapr-to-deliver-topic-events" >}})

### At-least-once guarantee

Dapr guarantees "At-Least-Once" semantics for message delivery. That means that when an application publishes a message to a topic using the publish/subscribe API, Dapr ensures that this message will be delivered at least once to every subscriber.
Dapr guarantees at-least-once semantics for message delivery. When an application publishes a message to a topic using the Pub/Sub API, Dapr ensures the message is delivered *at least once* to every subscriber.

### Consumer groups and competing consumers pattern

The burden of dealing with concepts like consumer groups and multiple application instances using a single consumer group is all handled automatically by Dapr. When multiple instances of the same application (running same app-IDs) subscribe to a topic, Dapr delivers each message to *only one instance of **that** application*. This is commonly known as the competing consumers pattern and is illustrated in the diagram below.
Dapr automatically handles the burden of dealing with concepts like consumer groups and competing consumers pattern. The competing consumers pattern refers to multiple application instances using a single consumer group. When multiple instances of the same application (running same app IDs) subscribe to a topic, Dapr delivers each message to *only one instance of **that** application*. This concept is illustrated in the diagram below.

<img src="/images/pubsub-overview-pattern-competing-consumers.png" width=1000>
<br></br>

Similarly, if two different applications (different app-IDs) subscribe to the same topic, Dapr deliver each message to *only one instance of **each** application*.
Similarly, if two different applications (with different app-IDs) subscribe to the same topic, Dapr delivers each message to *only one instance of **each** application*.

### Topic scoping
### Scoping topics for added security

By default, all topics backing the Dapr pub/sub component (e.g. Kafka, Redis Stream, RabbitMQ) are available to every application configured with that component. To limit which application can publish or subscribe to topics, Dapr provides topic scoping. This enables you to say which topics an application is allowed to publish and which topics an application is allowed to subscribe to. For more information read [publish/subscribe topic scoping]({{< ref pubsub-scopes.md >}}).
By default, all topics backing the Dapr Pub/Sub component are available to every application configured with that component. You can limit which application can publish or subscribe to topics with Dapr topic scoping. For more information, read: [Pub/Sub topic scoping]({{< ref pubsub-scopes.md >}}).

### Message Time-to-Live (TTL)
Dapr can set a timeout message on a per message basis, meaning that if the message is not read from the pub/sub component, then the message is discarded. This is to prevent the build up of messages that are not read. A message that has been in the queue for longer than the configured TTL is said to be dead. For more information read [publish/subscribe message time-to-live]({{< ref pubsub-message-ttl.md >}}).

- Note: Message TTL can also be set for a given queue at the time of component creation. Look at the specific characteristic of the component that you are using.
Dapr can set a timeout message on a per-message basis, meaning that if the message is not read from the Pub/Sub component, then the message is discarded. This timeout message prevents a build up of unread messages. If a message has been in the queue longer than the configured TTL, it is marked as dead.

For more information, read [Pub/Sub message TTL]({{< ref pubsub-message-ttl.md >}}).

{{% alert title="Note" color="primary" %}}
You can also set message TTL for a given queue at component creation. Look at the specific characteristic of the component that you are using.

{{% /alert %}}

### Communication with applications not using Dapr and CloudEvents
For scenarios where one application uses Dapr but another doesn't, CloudEvent wrapping can be disabled for a publisher or subscriber. This allows partial adoption of Dapr pubsub in applications that cannot adopt Dapr all at once. For more information read [how to use pubsub without CloudEvent]({{< ref pubsub-raw.md >}}).

### Publish/Subscribe API
If one of your applications uses Dapr while another doesn't, you can disable the CloudEvent wrapping for a publisher or subscriber. This allows partial adoption of Dapr Pub/Sub in applications that cannot adopt Dapr all at once.

For more information, read [how to use Pub/Sub without CloudEvents]({{< ref pubsub-raw.md >}}).

The publish/subscribe API is located in the [API reference]({{< ref pubsub_api.md >}}).
## Try it out

Want to put the Dapr Pub/Sub API to the test? Walk through the following quickstart and tutorials to see Pub/Sub in action:

| Quickstart/tutorial | Description |
| ------------------- | ----------- |
| [Pub/Sub quickstart]({{< ref pubsub-quickstart.md >}}) | Send and receive messages using the publish and subscribe API. |
| [Pub/Sub tutorial](https://github.com/dapr/quickstarts/tree/master/tutorials/pub-sub) | Demonstrates how to use Dapr to enable pub-sub applications. Uses Redis as a pub-sub component. |

## Next steps

* Follow these guides on:
* [How-To: Publish a message and subscribe to a topic]({{< ref howto-publish-subscribe.md >}})
* [How-To: Configure Pub/Sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}})
* Try out the [Pub/Sub quickstart sample](https://github.com/dapr/quickstarts/tree/master/tutorials/pub-sub)
* Learn [how to use Pub/Sub]({{< ref howto-publish-subscribe.md >}}).
* Follow the [How-To: Configure Pub/Sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}})
* Learn about [topic scoping]({{< ref pubsub-scopes.md >}})
* Learn about [message time-to-live (TTL)]({{< ref pubsub-message-ttl.md >}})
* Learn about [pubsub without CloudEvent]({{< ref pubsub-raw.md >}})
* List of [pub/sub components]({{< ref supported-pubsub.md >}})
* Read the [pub/sub API reference]({{< ref pubsub_api.md >}})
* Learn about [message TTL]({{< ref pubsub-message-ttl.md >}})
* Learn about [Pub/Sub without CloudEvent]({{< ref pubsub-raw.md >}})
* List of [Pub/Sub components]({{< ref supported-pubsub.md >}})
* Read the [Pub/Sub API reference]({{< ref pubsub_api.md >}})