How can a RabbitMQ cluster sustain a throughput of 100K messages per second?

[TaheJdidi]

Describe the bug

Hi there,

I've created a RabbitMQ client using the rabbitmq.client package. I'm receiving 100,000 messages from a third party every second, and I want to write all of these messages to the broker. I'm sending a batch publish to the broker every second. However, it seems like the broker can't handle this volume of messages every second.

Reproduction steps

1.Send write per batch every 1s containing 100k messages

Expected behavior

The broker could write the 100k messages every 1s

Additional context

No response

[michaelklishin]

@TaheJdidi do you really expect that there is a single number that would work for every workload and every environment?

The answer depends on the infrastructure used. A single queue, including quorum queues in 3.13, usually do not reach 100K but several queues collectively can reach hundreds of thousands given enough resources.

In addition to RabbitMQ itself, the answer will vary from client library to client library. A single connection (app instance) that uses Bunny, Pika won't usually go above 10K. Java, .NET, Rust and C++ clients reach tends of thousands.

Throughput will be proportionally lower with larger messages for fairly obvious network and disk I/O reasons.

Don't Guess

When in doubt, don't guess. Benchmark your actual deployment with multiple queues and streams,
PerfTest and Stream PerfTest can simulate a wide range of workloads and use pre-configured topologies.

Streams

A single stream with a fast RabbitMQ stream protocol client can reach a million. Several streams — a few millions, provided the disks are fast enough. A superstream (a partitioned stream) is a way of using multiple streams.

RabbitMQ blog contains many benchmark results that include not only some numbers but also a description of the environment.