Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug Report]: KafkaFlow doesn't work with cooperative-sticky protocol #456

Open
1 task done
AlexeyRaga opened this issue Oct 18, 2023 · 3 comments
Open
1 task done
Assignees
Labels
bug Something isn't working

Comments

@AlexeyRaga
Copy link

Prerequisites

  • I have searched issues to ensure it has not already been reported

Description

As discussed in this thread KafkaFlow doesn't behave correctly when the consumer is configured to use cooperative-sticky rebalancing protocol.

The protocol is described here

Steps to reproduce

Configure the consumer using WithConsumerConfig(...) and set the the strategy:

PartitionAssignmentStrategy = PartitionAssignmentStrategy.CooperativeSticky

Expected behavior

The consumer is expected to work as usual/normal, except that rebalancing would not cause "stop-the-world" behaviour.

Actual behavior

When KafkaFlow is configured to use PartitionAssignmentStrategy.CooperativeSticky the consumer seems to be working (processes messages), but does not commit any offsets.

KafkaFlow version

2.4.1

@AlexeyRaga AlexeyRaga added the bug Something isn't working label Oct 18, 2023
@lpcouto lpcouto self-assigned this Feb 6, 2024
@lpcouto
Copy link
Contributor

lpcouto commented Feb 8, 2024

Hi @AlexeyRaga,
I'm starting to look into this issue. As we no longer have access to the mentioned thread, can you give some more context, on what was said there?
Also, have you updated Kafkaflow to version 3? If so, is this still an issue in that version?

@AlexeyRaga
Copy link
Author

@lpcouto I equally have no access to the thread anymore, but I don't believe that anything useful was mentioned in there.
As far as I can remember, there was something like "oh, it should work because an underlying librdkafka does" and then "oh, no, it doesn't indeed because we use the callbacks that are different in the cooperative mode".
But not much more.

I haven't tried using cooperative-sticky mode in version 3, but I don't believe it'll work.
It is extremely easy to test though, just set the mode to be CooperativeSticky and observe no offsets committed.

@lpcouto
Copy link
Contributor

lpcouto commented Feb 13, 2024

@AlexeyRaga
When the cooperative-sticky rebalancing protocol is set, Kafka's response to the rebalance is different - we get the incremented partitions, and not the consumer's total list of partitions. This causes an unexpected behavior in Kafkaflow - it cannot commit offsets to partitions that it supposedly doesn't have.
We are working on a solution and will let you know when done.

golanbz added a commit to golanbz/kafkaflow that referenced this issue Sep 2, 2024
…world" scenarios (e.g., cooperative sticky). Fixes issue Farfetch#557 and Fixes issue Farfetch#456

* Enabled automatic committing with `confluent auto commit: true` instead of relying solely on manual commits, but only when the consumer strategy is cooperative sticky. (Refer to the open librdkafka issue at confluentinc/librdkafka#4059).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Development

No branches or pull requests

2 participants