Kafka producer configurations

[Miguel-Garcia-TH]

Hi,

I have a question/issue to raise and I hope this is the right place to do it, as Slack seems a bit quiet.

I’m using Sequin with a Kafka sink and am encountering duplicate messages, particularly when Kafka brokers crash.

After some research and debugging, I believe the duplicates are caused by an acknowledgment that is never received. The Kafka producer inside Sequin appears to retry before another broker responds with the ack. This seems related to the behavior described in your blog post on at-most-once, at-least-once, and exactly-once delivery.

To make my setup more resilient, I would need to configure additional Kafka producer settings. However, it seems that Sequin does not support overriding producer configurations beyond the options available in the web form or the Kafka sink configuration example.

Is there a way to override Kafka producer configs? For example, I would expect to be able to configure settings such as:

destination:
    type: "kafka"
    hosts: "<kafka_bootstrap_servers>"
    topic: "my-topic"
    acks: "all"
    retries: 3
    max_in_flight_requests_per_connection: 5
    enable_idempotence: true

[dosubot]

Hi @Miguel-Garcia-TH! I'm Dosu and I’m helping the sequin team.

You're correct that Sequin currently doesn't expose the Kafka producer settings you're looking for. Looking at the codebase, the only configurable producer setting is compression (none, gzip, snappy, lz4, zstd). Settings like acks, retries, max_in_flight_requests_per_connection, and enable_idempotence are not exposed and rely on the defaults from the Brod library.

Sequin does implement its own application-level retry logic (5 retries with exponential backoff starting at 200ms), which is separate from Kafka producer-level retries. This combined with Brod's defaults (which may allow multiple in-flight requests without idempotence enabled) likely explains the duplicate messages you're seeing during broker crashes.

There's no workaround to configure these settings currently—this would be a valid feature request. For now, consumer-side deduplication may be your best option to handle duplicates.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}