You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently it persists the state without updating committable offsets, so after shutdown we have updated state in persistent storage but no committed offsets, which is not a critical issue by itself as any sane application should support deduplication of input messages, meaning that when we start an app again it would load latest state and re-process some messages by simply ignoring them.
Improved graceful shutdown sequence should look like
finish processing of current messages we got from consumer.poll
persist the state
collect committable offsets after state is persisted (currently TimerFlowOf's release does nothing related to offsets on release)
with following conditions
if messages' processing fails, no persistence and no commit is required
if state's persisting fails, committable offsets cannot be updated and no commit is required
A minimum set of tests needs to be added
if state is persisted, correct offsets are committed, so that on next app run there's no re-processing of messages
if state's persistence failed, then commit should not happen
Also it might affect a special case of graceful shutdown for a single PartitionFlow instead of the whole app
on consumer group's rebalance some partitions might got removed from current consumer instance
as a result corresponding PartitionFlow would be removed from TopicFlow's cache, triggering the release of PartitionFlow, which would flush/persist the state if corresponding config is enabled (flushOnRevoke)
current implementation does not try to commit any offsets afterwards, it simply removes/forgets pending commits
Chances are quite high that it won't be an easy task as
we don't have access to pending offsets to be committed in TimerFlowOf
shutdown sequence is a result of a composition of different resources, so multiple files have to be adjusted, mb even some considerable part of the processing flow and the way we assemble the structure of Topic/Partition/Key/Timer flows
The text was updated successfully, but these errors were encountered:
Currently it persists the state without updating committable offsets, so after shutdown we have updated state in persistent storage but no committed offsets, which is not a critical issue by itself as any sane application should support deduplication of input messages, meaning that when we start an app again it would load latest state and re-process some messages by simply ignoring them.
Improved graceful shutdown sequence should look like
consumer.poll
TimerFlowOf
's release does nothing related to offsets on release)with following conditions
A minimum set of tests needs to be added
Also it might affect a special case of graceful shutdown for a single PartitionFlow instead of the whole app
PartitionFlow
would be removed fromTopicFlow
's cache, triggering the release ofPartitionFlow
, which would flush/persist the state if corresponding config is enabled (flushOnRevoke)Chances are quite high that it won't be an easy task as
TimerFlowOf
The text was updated successfully, but these errors were encountered: