Get information about missing consumer acknowledgements in publisher #12017
-
|
Hi, we want to use transactional outbox and inbox message pattern for sending and receiving messages from RabbitMQ. We plan to use a cluster in one datacenter and if the datacenter fails, the application will switch to the second datacenter. Inside the second datacenter a fresh standby RabbitMQ cluster gets started but without any kind of message forwarding etc. The idea is to replay the messages in the (replicated) outbox tables. In case of duplicates, that could be detected by (replicated) inbox tables. Our question is, how long we should keep publisher confirmed messages in the outbox tables to be sure, that they have been consumed by all potential consumers. Getting only the ACK for publishing won't be sufficient to decide as that messages will be lost if the datacenter is switched. Is there any way (RabbitMQ API call?) to detect, if all target consumers have succesfully consumed the message? Without introducing explicit response messages and need to track who should actually get the messages etc. ... That old thread does not seem promising unfortunately: Thanks in advance, |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 5 replies
-
|
Publishers are completely separated from consumers in every messaging (and to a lesser extent, streaming) protocol RabbitMQ supports, by design. Our team cannot know for how long you should to keep messages "unconfirmed". I'd say twice as long as your 95% percentile message processing time on the consumer. If you don't have those metrics, a good place to start would be to start collecting them. A few minutes should be fine for most systems but it's just a guess. |
Beta Was this translation helpful? Give feedback.
-
|
Explicit response messages is the only way for publishers to know that the consumer has consumed and processed a message. You don't have to acknowledge each message individually. The same approaches for publisher confirms can work here, namely batching of confirmations, and redelivery of batches when you must redeliver. The moment when you start coupling publishers and consumers in any "synchronous way", your availability goes out the window because a single stuck consumer will block some or all publishers. If multiple data centers are involved, that could be followed by a false positive promotion. |
Beta Was this translation helpful? Give feedback.
-
|
Ok, thank you for answering. We also really would like to avoid response messages and coupling, because then we would need to keep track, which consumers at all should get messages depending on the message types (for example fanout sending copies to multiple consumers). Definitely something we would like to avoid. Therefore we were thinking about asking RabbitMQ later on by a timerjob if a particular message identified by an unique ID is still waiting in at least one queue for processing. If yes, we would not drop that message from outbox. If no, we can safely remove it from the outbox. Of course we would only check already published confirmed messages in our outbox. In the RabbitMQ management UI there is an overview of waiting messages, therefore I had the hope, that there is an API call to check for messages by a given message ID.
Working with best guess might work here, but not sure, if we could rely on that for our application. For example if a consumer is completely down for one hour due to failure, a few minutes would be a wrong guess then. But nevertheless thank you again very much for your fast response! :) Best |
Beta Was this translation helpful? Give feedback.
-
|
Asynchronous request-reply coupling is the "loosest coupling" form that exists. A published message can be routed to N queues, can expire and be dead-lettered, can be dropped because of a length limit, and so on. It's not realistic to keep track of what happened to a specific published message because it can be dropped in one queue, consumed and requeued in another, and consumed and acknowledged in the third. This is besides the aforementioned synchronous-in-nature coupling such operation would immediately create. |
Beta Was this translation helpful? Give feedback.
-
|
Ok, I see. Then I think we have two options:
Both options doesn't look very satisying :/ Best |
Beta Was this translation helpful? Give feedback.
-
|
Just like with warm standbys (a feature available in Tanzu RabbitMQ, the commercial edition), you must decide on what's your margin of safety, and the best way to do that is by using consumer metrics. The upside of this is (potentially much) simpler applications. But I can think of a 3rd option: using streams where consumers can consume repeatedly and keep track of their progress using a cursor, that's it. The data deletion is handled using an expiration interval, which is very similar to the 2nd option above. |
Beta Was this translation helpful? Give feedback.
-
|
Ok, thanks for the hint. I'll take a look. Best |
Beta Was this translation helpful? Give feedback.
Explicit response messages is the only way for publishers to know that the consumer has consumed and processed a message.
You don't have to acknowledge each message individually. The same approaches for publisher confirms can work here, namely batching of confirmations, and redelivery of batches when you must redeliver.
The moment when you start coupling publishers and consumers in any "synchronous way", your availability goes out the window because a single stuck consumer will block some or all publishers. If multiple data centers are involved, that could be followed by a false positive promotion.