#1360 Support for poll maximum wait time #1417
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
The JDBC source task stoppage often results in the ungraceful shutdown, due to a long duration time for the polling operation which blocks the worker task thread.
If the connector is restarted and re-deployed on the same node, it may result in a missing JMX bean for the connector as the newly registered metrics will be removed once the task is finally stopped after graceful timeout.
This can case issues with monitoring as the metrics needs to be exported from the JMX.
Solution
The proposed solution is to introduce a new configuration property
poll.max.wait.time.ms
which limits the duration for which the worker task is blocked for a single polling call.To make it possible, the polling operation should be run in a separated thread, where the task awaits for it to finish w up to the configured
poll.max.wait.time.ms
duration.If the operation is not finished within
poll.max.wait.time.ms
the source task will returnnull
list of records signaling no-data to the worker.As the worker task runs in a loop, where its continuously polling the source task, the task will check if there is a previously started poll operation, and try to wait for it to finish each time. If there is no previous operation started, it will start new one, and apply the same logic.
This in consequence should limit the total duration for which the worker task thread can be blocked, thus allowing the graceful shutdown, regardless of the
poll.interval.ms
value nor the records fetching time.For the backward compatibility it should be possible to set the
poll.max.wait.time.ms
to0
where the task waits indefinitely for the polling result, without a separated thread.Does this solution apply anywhere else?
If yes, where?
Similar strategy can be applied to other types of source connectors where worker thread is blocked for extended time period during the
SourceTask::poll
call.Test Strategy
To allow easier testing, I've extracted the related logic into
JdbcSourceTaskPollExecutor
class.Testing done:
Release Plan