Add message size metrics export to Prometheus

[fmarani]

Is your feature request related to a problem? Please describe.

We are trying to decrease the max_message_size from the very generous (for us) default to something much smaller. We are not able to easily pick a new max because we don't have metrics on it.

Describe the solution you'd like

Ideally i'd be able to get the message size info through the Prometheus exporter, and the metric would be at queue level.

Describe alternatives you've considered

No response

Additional context

We are using Celery

[michaelklishin]

It is not a metric that exists in RabbitMQ, so there's nothing to expose to Prometheus. Our team won't have the capacity to work on this metric for 4.0 but you are welcome to investigate it and share a design before you spend any time actually implementing it.

[gomoripeti]

We are interested in this metric too and would like to implement it

To tune max_message_size setting it would be most useful to record
message size at the point where it is checked, which is the AMQP 0.9.1
channel or AMQP 1.0 session processes.

Keeping track of exact max value is a bit trickier than simple
counters because of parallel updates from multiple processes. One
option is that the new metric is handled by rabbit_core_metrics. A
new ETS table message_size is created with one entry initialized per
protocol {Proto, MaxValue} (amqp091, amqp10). Then
ets:select_replace/3 can be used to atomically update the max value.

Alternatively instead a histogram of message sizes can be tracked with
~10-20 buckets. Each interval's upper bound is 4 or 16 times as high
as the one before it and the last interval has no upper bound. For
example first bucket counts messages with a size between 0..256B,
second 257B..1024B, ..., 64MiB+1..256MiB and the last one 256MiB+1 and
above. A tuple with arity equal to the bucket count can be stored in
an ETS table and can be atomically updated with
ets:update_counter/4. The histogram gives richer (but less exact)
information about message sizes coming into the broker.

Although MQTT does not enforce the max_message_size limit (if I am
not mistaken) it is still valuable to track message sizes from
rabbit_mqtt_processor as well.

STOMP currently proxies through AMQP 0.9.1 (so it should be covered)
but the same applies when native STOMP will be implemented.

Keeping track of max message size per-queue is not strictly necessary
to set an appropriate limit but could help in investigations. (Storing
a histogram for each queue is too much data and overkill in my
opinion.) One way is to update max message size from within the queue
(leader) process. In this case code for each queue type needs to be
updated:

• classic queues: rabbit_amqqueue_process
• quorum queues: rabbit_fifo
• stream queues: osiris lib
  The last one is a bit problematic as it lives in a dependency which
  cannot call rabbit_core_metrics so would have to have its own
  implementation.

Alternatively per-queue metrics can be updated outside the queue
process from the publishing process (rabbit_channel,
rabbit_amqp_sessions, rabbit_mqtt_processor), just before or after
they call rabbit_queue_type:deliver/4.

Another idea is instead of per-queue only track per-queue-type sizes.
(Different queue storages are able to handle large messages differently)

I'd appreciate all feedback from the Core Team, especially in these open questions:

• should we have per-protocol or per-queue metrics or both?
• should we have histogram or max value for per-protocol or both?
• is it ok to update per-queue metrics outside queue processes?
• is it ok to update per-queue metrics even if message is dropped by
  the queue or delivery fails?

(Of course if the answer is "both" for any of the questions they can
be implemented step-by-step in multiple PRs)

[kjnilsson]

I considered seshat counters, but the advantage of ets is that the implementation can be identical for per-protocol and per-queue metrics. If I understand correctly seshat involves using persisten_terms which don't play well with potentially high number of transient objects as keys like queues which can come and go.

No seshat doesn't mandate use of persistent term - that was just a choice that was made in the global metrics - each process that wants to increment counters can just get the counter ref from the seshat tables in their init function.

I am not sure per queue metrics will tell you more than you'll know from the protocol metrics? or are you looking for something else here?

[johanrhodin]

I am not sure per queue metrics will tell you more than you'll know from the protocol metrics? or are you looking for something else here?

Per-queue metrics would solve for the case where the user isn't sure which publishers are generating big messages. Think thousands of queues and you only see after the fact that network usage/disk bandwidth was high. Consumers chugged along, maybe even have been auto-scaled, so the messages are gone and you can't check message body size after the fact (or parse the .qi/.qs/.rdq files to find out the histogram of messages). But #12087 will help you with setting max_message_size.

[ansd]

I'm not sure a histogram helps much with achieving the goal to set a new lower max_message_size. For example, what do you do when you see a few dozens of messages in the range 64-256 MB (but none above 256MB) over the course of the last day? The histogram won't tell which connections these large messages are coming from or which queues they are going into. They might be coming from a single client or from many different clients. Do you then leave your max_message_size limit at 256MB?

Probably, if I were to set a new max_message_size, I would implement an event being emitted to the event exchange per large message with information how large this message is, which connection and protocol this message was sent from with a timestamp and whatever other info might be interesting....
The threshold of "large message" (i.e. when to emit this event) can be configurable and put into persistent_term.
Over the course of a few days, you can then gradually decrease this limit and pinpoint any large message you encounter.

That said, I have nothing against a histogram of message sizes exposed via Prometheus. We can add it. It looks nice and might be interesting to learn about message size distributions (but probably for reasons other than setting max_message_size).

[johanrhodin]

what do you do when you see a few dozens of messages in the range 64-256 MB (but none above 256MB) over the course of the last day?

If you see that 99.99% of messages are in the smallest buckets and just a few dozen in the 64-256MB range you set max_message_size to say 8MiB and let the applications tell you that they can't publish :)

I would implement an event being emitted to the event exchange per large message with information how large this message is, which connection and protocol this message was sent from with a timestamp and whatever other info might be interesting....

That sounds like a great idea for a possible follow-up PR!

[michaelklishin]

As long as we don't overwhelm the single process event manager behind the internal event mechanism, I like the idea.

[johanrhodin]

This is being implemented in #12087 (on protocol level, not queue-level, and not for streams)