Skip to content

Conversation

beliefer
Copy link
Contributor

@beliefer beliefer commented Sep 25, 2025

This PR proposes to improve perf for Elasticsearch 8 connector.
Currently, Elasticsearch 8 connector create a new OperationSerializer for each Operation.
A large amount of OperationSerializer's instances brings big overhead.

There is a benchmark cases.

  1. Flink application with single parallelism.
    The Kafka Topic consumption rate show below.
Before this PR After this PR
1.5 K strips/s(min) 2.3 K strips/s(min)
2.0 K strips/s(max) 2.9 K strips/s(max)
single
  1. Flink application with 32 parallelisms.
    The Kafka Topic consumption rate show below.
Before this PR After this PR
58.3 K strips/s(min) 70.3 K strips/s(min)
60.0 K strips/s(max) 74.5 K strips/s(max)
32

@beliefer
Copy link
Contributor Author

beliefer commented Sep 25, 2025

ping @mtfelisb @liuml07 cc @reswqa

Copy link
Member

@liuml07 liuml07 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR makes sense to me. Thanks for fixing this.

I think we can make the commit / PR title more descriptive. How about something like Reuse OperationSerializer in Elasticsearch 8 connector to improve perf

@beliefer beliefer changed the title [FLINK-38429] Improve perf for Elasticsearch 8 connector [FLINK-38429] Reuse OperationSerializer in Elasticsearch 8 connector to improve perf Sep 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants