confluent_kafka.Consumer.consume Method Takes Excessive Time to Respond #1752
abdelghanimeliani
started this conversation in
General
Replies: 1 comment
-
the issue arose because when auto.offset.reset is set to latest, the consumer cannot fetch events produced before its subscription and waits for the producer to generate a new event. By setting 'auto.offset.reset' to earliest, the consumer can now access the old events, but this results in retrieving the oldest event in the topic, which is not suitable for my use case, for me, i want the LAST available event. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I am writing a Flask API to retrieve data from Apache Kafka topics. The consumer subscribes to a different topic each time according to the API path and uses the
confluent_kafka.Consumer.consume
method to fetch a given number of events each time. Here is the consumer configuration I am using:The code to consume events from the given topic is as follows:
the
assignment_callback
and themessages_parser
are other methods that i built .The problem is that it takes a long time (1-2 minutes) to return the answer. Is this normal behavior, or is there something wrong with the configuration or usage?
additional infomration: I am using kafka as a docker container for an experimental environment
Beta Was this translation helpful? Give feedback.
All reactions