Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: give more memory to the Delta Like workers #271

Merged
merged 1 commit into from
Aug 29, 2023
Merged

feat: give more memory to the Delta Like workers #271

merged 1 commit into from
Aug 29, 2023

Commits on Aug 29, 2023

  1. feat: give more memory to the Delta Like workers 

    We were noticing that after about 60M rows, a delta lake table can no
longer be merged to, with only 2G of driver memory.

So bumping it to 4G for extra room. This may mean that batch sizes
should be adjusted downward, as the driver takes more system memory.
    @mikix
    mikix committed Aug 29, 2023
    Configuration menu
    Copy the full SHA
    6c2d5e2 View commit details
    Browse the repository at this point in the history