You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Backfilling can backpressure upstream, causing the existing streaming jobs to be slower or even stuck. There are three cases where backfilling can happen:
CREATE MV
CREATE SINK with connector
CREATE SINK INTO TABLE
The current way to mitigate backfilling effect on upstream
SET BACKFILL_RATE_LIMIT to xxx. Supported for 1, 2, 3.
SET sink_decouple to true (default on). Supported for 2.
SET streaming_use_snapshot_backfill to true (default off, experimental now). Supported for 1.
The only effective way for 3 is use rate limit, which requires manual operation and understanding on the workload before determining a good value. Therefore, I think we should also support sink decoupling for sink into table as well. This is also a perquisite of doing severless backfill for sink into table.
Describe the solution you'd like
There are two ways to implement sink decoupling for sink into table:
Use kv log store for SINK INTO TABLE, similar to what we did for sink with connector.
Record L0 changelog and support snapshot backfilling for SINK INTO TABLE, similar to what we did for MV.
Describe alternatives you've considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
Backfilling can backpressure upstream, causing the existing streaming jobs to be slower or even stuck. There are three cases where backfilling can happen:
The current way to mitigate backfilling effect on upstream
SET BACKFILL_RATE_LIMIT to xxx
. Supported for 1, 2, 3.SET sink_decouple to true
(default on). Supported for 2.SET streaming_use_snapshot_backfill to true
(default off, experimental now). Supported for 1.The only effective way for 3 is use rate limit, which requires manual operation and understanding on the workload before determining a good value. Therefore, I think we should also support sink decoupling for sink into table as well. This is also a perquisite of doing severless backfill for sink into table.
Describe the solution you'd like
There are two ways to implement sink decoupling for sink into table:
Describe alternatives you've considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: