Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[close #395] Batch writes in Kafka consumer #396

Merged
merged 31 commits into from
Mar 5, 2024

Conversation

pingyu
Copy link
Collaborator

@pingyu pingyu commented Feb 25, 2024

What problem does this PR solve?

Issue Number: close #395

Problem Description: cdc: Batch writes to sinks in Kafka consumer

What is changed and how does it work?

  • Batch writes to sinks in Kafka consumer.
  • Fix possible bugs:
    • MarkMessage should not be inside the loop of handling kv entries of a message.
    • break ClaimMessages when meet fallback kv seems to make no sense.

Depends on #392

Code changes

  • No.

Check List for Tests

This PR has been tested by at least one of the following methods:

  • Integration test

Side effects

  • No side effects

Related changes

  • No related changes

To reviewers

Please follow these principles to check this pull requests:

  • Concentration. One pull request should only do one thing. No matter how small it is, the change does exactly one thing and gets it right. Don't mix other changes into it.
  • Tests. A pull request should be test covered, whether the tests are unit tests, integration tests, or end-to-end tests. Tests should be sufficient, correct and don't slow down the CI pipeline largely.
  • Functionality. The pull request should implement what the author intends to do and fit well in the existing code base, resolve a real problem for TiDB/TiKV users. To get the author's intention and the pull request design, you could follow the discussions in the corresponding GitHub issue or internal.tidb.io topic.
  • Style. Code in the pull request should follow common programming style. (For Go, we have style checkers in CI). However, sometimes the existing code is inconsistent with the style guide, you should maintain consistency with the existing code or file a new issue to fix the existing code style first.
  • Documentation. If a pull request changes how users build, test, interact with, or release code, you must check whether it also updates the related documentation such as READMEs and any generated reference docs. Similarly, if a pull request deletes or deprecates code, you must check whether or not the corresponding documentation should also be deleted.
  • Performance. If you find the pull request may affect performance, you could ask the author to provide a benchmark result.

(The above text mainly refers to TiDB Development Guide. It's also highly recommended to read about Writing code review comments)

Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
@pingyu pingyu changed the title [close #395] Kafka consumer batch [close #395] Batch writes in Kafka consumer Feb 25, 2024
@ti-chi-bot ti-chi-bot bot added the size/L label Feb 25, 2024
Signed-off-by: Ping Yu <yuping@pingcap.com>
@ti-chi-bot ti-chi-bot bot added size/XL and removed size/L labels Feb 25, 2024
@pingyu
Copy link
Collaborator Author

pingyu commented Feb 25, 2024

/test pull-integration-kafka-test

2 similar comments
@pingyu
Copy link
Collaborator Author

pingyu commented Feb 25, 2024

/test pull-integration-kafka-test

@pingyu
Copy link
Collaborator Author

pingyu commented Feb 25, 2024

/test pull-integration-kafka-test

Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
@pingyu
Copy link
Collaborator Author

pingyu commented Feb 25, 2024

/test pull-integration-kafka-test

Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
…sumer-batch

Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
…sumer-batch

Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
…sumer-batch

Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
…sumer-batch

Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
@ti-chi-bot ti-chi-bot bot added size/L and removed size/XL labels Mar 4, 2024
Signed-off-by: Ping Yu <yuping@pingcap.com>
@pingyu pingyu requested a review from zeminzhou March 5, 2024 00:07
Copy link
Contributor

@zeminzhou zeminzhou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pingyu pingyu merged commit 50e26d5 into tikv:main Mar 5, 2024
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

cdc: Batch writes to sinks in Kafka consumer
2 participants