Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Implement SplitAndRetry for ProjectExec #12242

Open
revans2 opened this issue Feb 28, 2025 · 0 comments
Open

[FEA] Implement SplitAndRetry for ProjectExec #12242

revans2 opened this issue Feb 28, 2025 · 0 comments
Labels
? - Needs Triage Need team to review and classify feature request New feature or request reliability Features to improve reliability or bugs that severly impact the reliability of the plugin

Comments

@revans2
Copy link
Collaborator

revans2 commented Feb 28, 2025

Is your feature request related to a problem? Please describe.
We recently put in some code to do a pre-split on ProjectExec

#12148

The idea there was to try and avoid running out of memory by estimating how large the output is likely to be, just looking at the data types and a few simple operations.

But this is not perfect and we can still run out of memory.

Describe the solution you'd like
We didn't implement splitting initially because some project operations used by windowing that we didn't want to split up. #12148 made it so that they do the right thing and when creating the project, the disable the split. So now we can enable spilt when it will not break anything.

The big issue with split in tiered project is splitting between multiple tieres. The only time that we run into that kind of a situation is when we have non-deterministic expressions, which are really rare. So at least in the common case we can handle the split an retry with no issues at all.

@revans2 revans2 added ? - Needs Triage Need team to review and classify feature request New feature or request reliability Features to improve reliability or bugs that severly impact the reliability of the plugin labels Feb 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
? - Needs Triage Need team to review and classify feature request New feature or request reliability Features to improve reliability or bugs that severly impact the reliability of the plugin
Projects
None yet
Development

No branches or pull requests

1 participant