Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

max parallelism is static to 128 unless you pass in write.parallelism #206

Closed
ryanvanhuuksloot opened this issue Jan 28, 2025 · 1 comment

Comments

@ryanvanhuuksloot
Copy link

ryanvanhuuksloot commented Jan 28, 2025

I was expecting with #203 that if your dataset was in region us then you'd be able to use up to the new cap of 512 without specifically setting the write.parallelism

What I found was that you need to pass write.parallelism
Even adding write.region wasn't enough to use a higher parallelism than 128 (without the write.parallelism)

@jayehwhyehentee
Copy link
Collaborator

128 is the Flink default. The connector enforces a max parallelism, but does not set the current job's parallelism. Passing write.parallelism is the expected behavior here.
Btw, I gope you're setting the region here (for datastream API) or here (for table API).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants