diff --git a/site/conversions.md b/site/conversions.md index 58370d1e5d..6f2e14171e 100644 --- a/site/conversions.md +++ b/site/conversions.md @@ -31,19 +31,27 @@ limitations under the License. ## Overview -RabbitMQ is a multi-protocol broker supporting AMQP, AMQP (0.9.1), MQTT (3-5) and the STOMP -protocols. Messages can readily be received from one protocol and consumed by others. +RabbitMQ is a multi-protocol broker supporting [AMQP 1.0](https://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-overview-v1.0-os.html) +(henceforth AMQP), [AMQP 0.9.1](https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf), +[MQTT](https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html), [STOMP](https://stomp.github.io/), +and the [RabbitMQ Streams](https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbitmq_stream/docs/PROTOCOL.adoc) protocols. +Messages can readily be published by one protocol and consumed by others. This necessitates data format conversions between different protocol formats. Prior to RabbitMQ 3.13 all messages were converted into an internal format based -on the AMQP 0.9.1 protocol when received. This approach often resulted in unnecessary +on the AMQP 0.9.1 protocol when published to RabbitMQ. This approach often resulted in unnecessary conversions and/or data fidelity issues. Since RabbitMQ 3.13 the approach has changed for the AMQP 0.9.1 and MQTT protocols -as well as messages written to RabbitMQ streams in that messages will always be stored in their original format and only converted -_if_ consumed by a protocol different from the originating protocol. This means all -conversions have been re-written and formalised. +_if_ consumed by a protocol different from the originating protocol. This has the advantage +that no protocol specific information is lost when the originating and the consuming protocols +are the same. +The only exception to this rule are [streams](https://www.rabbitmq.com/streams.html): No matter which protocol published, +streams internally store their messages as AMQP 1.0 encoded data. Hence, when for example publishing using AMQP 0.9.1 +a conversion to AMQP takes place. + +All conversions have been re-written and formalised. This document aims to capture the conversions rules that are implemented so that application developers can look back at this to understand how their messages will be transformed in their multi-protocol messaging applications. @@ -52,18 +60,19 @@ will be transformed in their multi-protocol messaging applications. ## Conventions `shortstr`: (condition) a string that is smaller than 256 bytes and contains only valid -utf8 data with no NUL characters. +UTF-8 encoded data with no `NUL` (binary zero octet) characters. `*`: is read as "any" (field/type etc). -Conditions can be read to be evaluated in the order they are written. E.g. -and AMQP message_id of type `utf8` with more than 255 bytes will fail the first +Conditions can be read to be evaluated in the order they are written. For example +an AMQP `message_id` of type `utf8` with more than 255 bytes will fail the first `shortstr` condition and then fall through to the next line. -If there is no matching line then the data will not be converted. +If there is no matching line then the data will not be converted (and therefore not be +available to the consuming application). ## AMQP -> AMQP 0.9.1 -In RabbitMQ 3.13 the AMQP (1.0) plugin does not implement these conversion rules +In RabbitMQ 3.13 the AMQP (1.0) plugin does not implement these conversion rules yet and they are only used when reading from a stream. However they will be used as of RabbitMQ 4.0+. @@ -93,7 +102,7 @@ as of RabbitMQ 4.0+. | properties | group_id | utf8 | `shortstr` | properties | app_id | shortstr | | | properties | group_sequence | sequence-no | | | | | | | properties | reply_to_group_id | utf8 | | | | | | -| application properties | \* | \* (see type conversions) | `shortstr` | properties | headers | \* | | +| application properties | \* | \* (see type conversions) | Key: `shortstr` | properties | headers | \* | | | message annotations | \* (symbol - x-cc\*) | | Key: x-cc | properties | headers, Key: "CC" | longstr | | | message annotations | \* (symbol - x-\*) | \* (see type conversions) | Key: `shortstr` & x-* | properties | headers | \* | Typically this means x- headers | | data | | data | single data section | payload | | binary | plain binary extracted from data section | @@ -106,7 +115,7 @@ as of RabbitMQ 4.0+. | AMQP 1.0 type | Condition | AMQP 0.9.1 type | Comment | | ------------- | ------------ | -------------------- | --------------------------------------------------- | -| utf8 | | longstr | RabbitMQ does not support shortstr header values so all utf8 inputs are converted to longstr unless it is a field in the basic properties | +| utf8 (string) | | longstr | RabbitMQ does not support shortstr header values so all utf8 inputs are converted to longstr unless it is a field in the basic properties | | binary | | binary | | | long | | long | | | ulong | | long | risk of overflow | @@ -184,20 +193,19 @@ as of RabbitMQ 4.0+. | MQTT 5.0 Section | MQTT Field | MQTT 5.0 Type | Condition | AMQP 1.0 Section | AMQP 1.0 Field | AMQP 1.0 Type | Comment | | ---------------------------- | -------------------------- | --------------------- | ------------------------------------- | ---------------------- | -------------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Fixed Header | Dup | Bits | | | | | Setting to header first-acquirer does not make sense because the DUP flag only applies from client to server (consumption from server to client is determined by the Redeliverd flag sent from the queues) | +| Fixed Header | Dup | Bits | | | | | Setting to header first-acquirer does not make sense because the DUP flag only applies from client to server (consumption from server to client is determined by the Redelivered flag sent from the queues) | | Fixed Header | QoS | Bits | | header | durable | boolean | durable is true if QoS > 0 | | Fixed Header | Retain | Bits | | | | | | | Variable Header | Payload Format Indicator | Bits | | | | | see row "Payload" with condition "Payload Format Indicator set" | | Variable Header | Message Expiry Interval | uint | | header | ttl | milliseconds | seconds to milliseconds | | Variable Header | Topic Alias | ushort | | | | | | -| Variable Header | Response Topic | utf8 | | properties | reply_to | utf8 | Translate MQTT topic to AMQP topic. If operator did not mofidy mqtt.exchange, set reply-to address to "/topic/" RK. If operator modified mqtt.exchange, set reply-to address to "/exchange/" X "/" RK. | +| Variable Header | Response Topic | utf8 | | properties | reply_to | utf8 | Translate MQTT topic to AMQP routing key (RK). Set reply-to address to "/exchange/" X "/" RK. | | Variable Header | Correlation Data | binary | `urn:uuid` | properties | correlation_id | uuid | | | Variable Header | Correlation Data | binary | | properties | correlation_id | binary | | -| | | | | | | | | -| Variable Header | User Property | utf8 | Key begins with "x-" and key is ASCII | message annotation | Key | map(symbol) | (as done for AMQP 0.9.1 -> AMQP 1.0) | -| | | | Key does not begin with "x-" | application properties | Key | map(utf8) | | -| Variable Header | Subscription Identifier | variable byte integer | | | | | | -| Variable Header | Content Type | utf8 | Content Type is ASCII | properties | content_type | symbol | MQTT content type is UTF-8 whereas AMQP 1.0 content type is only ASCII | +| Variable Header | User Property | utf8 | Key begins with "x-" and key is ASCII | message annotation | | value: utf8 | Key is of type symbol | +| | | | Key does not begin with "x-" | application properties | | value: utf8 | Key is of type utf8 | +| Variable Header | Subscription Identifier | uint | | | | | | +| Variable Header | Content Type | utf8 | valid ASCII | properties | content_type | symbol | MQTT content type is UTF-8 whereas AMQP 1.0 content type is only ASCII | | Payload | | | Payload Format Indicator unset | data | | data | | | Payload | | | Payload Format Indicator set | amqp.value | | utf8 | If the Payload Format Indicator is set, convert the MQTT payload to a string (i.e. single AMQP value section) because an AMQP string is UTF-8 encoded | @@ -254,7 +262,6 @@ as of RabbitMQ 4.0+. | ---------------------- | -------------------- | ------------------------- | ------------------------------------------------------------------------- | -------------------| --------------------------| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | header | durable | boolean | | Fixed Header | QoS | Bits | durable=true maps to QoS 1 durable=false maps to QoS 0 | | properties | message_id | utf8 | | | | | | -| properties | message_id | utf8 | | | | | | | properties | message_id | uuid | | | | | | | properties | message_id | ulong | | | | | | | properties | message_id | binary | | | | | |