[FEA] Implement SplitAndRetry for ProjectExec #12242
Labels
? - Needs Triage
Need team to review and classify
feature request
New feature or request
reliability
Features to improve reliability or bugs that severly impact the reliability of the plugin
Is your feature request related to a problem? Please describe.
We recently put in some code to do a pre-split on ProjectExec
#12148
The idea there was to try and avoid running out of memory by estimating how large the output is likely to be, just looking at the data types and a few simple operations.
But this is not perfect and we can still run out of memory.
Describe the solution you'd like
We didn't implement splitting initially because some project operations used by windowing that we didn't want to split up. #12148 made it so that they do the right thing and when creating the project, the disable the split. So now we can enable spilt when it will not break anything.
The big issue with split in tiered project is splitting between multiple tieres. The only time that we run into that kind of a situation is when we have non-deterministic expressions, which are really rare. So at least in the common case we can handle the split an retry with no issues at all.
The text was updated successfully, but these errors were encountered: