You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm creating a wrapper on top of confluent-kafka-python to poll and process messages concurrently. They are IO-bound so this can improve throughput without increasing the number of partitions. Do I need to set enable.auto.commit to false and locally track offsets and commit them manually? Or does confluent-kafka-python already handle this and I can just set enable.auto.commit to True?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
I'm creating a wrapper on top of confluent-kafka-python to poll and process messages concurrently. They are IO-bound so this can improve throughput without increasing the number of partitions. Do I need to set enable.auto.commit to false and locally track offsets and commit them manually? Or does confluent-kafka-python already handle this and I can just set enable.auto.commit to True?
The doc says The Producer, Consumer and AdminClient are all thread safe.. Does this mean I don't need to manage offsets manually and can just run multiple threads in a consumer to process messages concurrently?
conflicting information
https://www.confluent.io/blog/kafka-consumer-multi-threaded-messaging/ says
When implementing a multi-threaded consumer architecture, it is important to note that the Kafka consumer is not thread safe.
but
https://github.com/confluentinc/confluent-kafka-python?tab=readme-ov-file#thread-safety says
The Producer, Consumer and AdminClient are all thread safe.
Is the consumer thread safe or not?
Beta Was this translation helpful? Give feedback.
All reactions