You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have a streaming job to ingest a table which has ~100 million records. I set the parallelism to 25 which pulls all the records. However, once the job ingests the initial load the resources are still blocked by this job. The incremental data does not require the configured parallelism after the initial load.
I tried setting jobmanager.scheduler=adaptive to check if it would auto scale the resources but that didn't work.
I am aware of stopping the job after initial load, setting the parallelism to a lower value and then restarting the job from the last savepoint approach.
But I want to know if there is any other approach to auto scale the job without stopping it using the savepoint method.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
I have a streaming job to ingest a table which has ~100 million records. I set the parallelism to 25 which pulls all the records. However, once the job ingests the initial load the resources are still blocked by this job. The incremental data does not require the configured parallelism after the initial load.
I tried setting jobmanager.scheduler=adaptive to check if it would auto scale the resources but that didn't work.
I am aware of stopping the job after initial load, setting the parallelism to a lower value and then restarting the job from the last savepoint approach.
But I want to know if there is any other approach to auto scale the job without stopping it using the savepoint method.
Beta Was this translation helpful? Give feedback.
All reactions