[Q&A] Controlling Kafka Consumer Speed and Back Pressure Mechanisms in Fluentd

[Mingsicogi]

What is a problem?

I am currently using the Fluentd Kafka plugin to consume data from Kafka. In high-throughput environments, I am encountering issues where the output processing rate cannot keep up with the incoming data rate from Kafka. This leads to potential data loss or increased memory usage. Specifically, I have the following questions and feature requests:

1. Is there a way to control the speed at which the consumer consumes data from Kafka, without relying solely on memory buffering?
2. Are there any plans to implement a back pressure mechanism in Fluentd that would adjust the input rate based on the output processing capacity? I found a related issue discussing the need for back pressure support in Fluentd: Fluentd Issue #567. It would be beneficial to know if there are any updates or plans to address this issue.

Describe the configuration of Fluentd

No response

Describe the logs of Fluentd

No response

Environment

- Fluentd version: fluentd 1.16.2
- TD Agent version:
- Fluent Package version: fluent-plugin-kafka 0.19.2, fluent-plugin-kinesis-aggregation 3.5.0
- Docker image (tag):
- Operating system:
- Kernel version:

[cosmo0920]

2. Are there any plans to implement a back pressure mechanism in Fluentd that would adjust the input rate based on the output processing capacity? I found a related issue discussing the need for back pressure support in Fluentd: Fluentd Issue #567. It would be beneficial to know if there are any updates or plans to address this issue.

Fluent Bit already implemented back pressure mechanism. So, it's time to move out from Fluentd. 🫥