Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix] Fix the key-based batch can't guarantee the ordering when flushing #1252

Merged
merged 5 commits into from
Jul 19, 2024

Conversation

RobertIndie
Copy link
Member

@RobertIndie RobertIndie commented Jul 19, 2024

Motivation

When using key-based producing, the key-based batcher cannot guarantee the sequence ID order when flushing messages. This leads to frequent disconnections between the client and the broker, resulting in poor performance.

Here is the related log:

time="2024-07-19T03:38:07Z" level=warning msg="Received send error from server: [PersistenceError] : [Cannot determine whether the message is a duplicate at this time]"
time="2024-07-19T03:38:07Z" level=warning msg="Connection was closed" cnx="xxx" producerID=1 producer_name=xxx topic="xxx"
time="2024-07-19T03:38:07Z" level=warning msg="Failed to write on connection" error="use of closed network connection" local_addr="xxx" remote_addr="xxx"

The broker receives unordered messages and returns a SendError to the client. This causes the client to reconnect to the broker.

Modifications

  • Refactor the BatchBuilder interface.
  • Sort batches before flushing in the key-based batcher builder.

Verifying this change

This change added tests

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API: (yes / no)
  • The schema: (yes / no / don't know)
  • The default values of configurations: (yes / no)
  • The wire protocol: (yes / no)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / GoDocs / not documented)
  • If a feature is not applicable for documentation, explain why?
  • If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

@RobertIndie RobertIndie added this to the v0.14.0 milestone Jul 19, 2024
@RobertIndie RobertIndie self-assigned this Jul 19, 2024
@BewareMyPower BewareMyPower merged commit e2fd4c9 into apache:master Jul 19, 2024
9 checks passed
@RobertIndie RobertIndie deleted the fix-key-based-ordering branch July 19, 2024 14:07
RobertIndie added a commit that referenced this pull request Jul 24, 2024
…ing (#1252)

### Motivation

When using key-based producing, the key-based batcher cannot guarantee the sequence ID order when flushing messages. This leads to frequent disconnections between the client and the broker, resulting in poor performance.

Here is the related log:
```
time="2024-07-19T03:38:07Z" level=warning msg="Received send error from server: [PersistenceError] : [Cannot determine whether the message is a duplicate at this time]"
time="2024-07-19T03:38:07Z" level=warning msg="Connection was closed" cnx="xxx" producerID=1 producer_name=xxx topic="xxx"
time="2024-07-19T03:38:07Z" level=warning msg="Failed to write on connection" error="use of closed network connection" local_addr="xxx" remote_addr="xxx"
```

The broker receives unordered messages and returns a SendError to the client. This causes the client to reconnect to the broker.

### Modifications

- Refactor the BatchBuilder interface.
- Sort batches before flushing in the key-based batcher builder.

(cherry picked from commit e2fd4c9)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants