Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Time Pruner should support queries where we're operating on the time column itself #14362

Open
jadami10 opened this issue Nov 1, 2024 · 4 comments · May be fixed by #14385
Open

Time Pruner should support queries where we're operating on the time column itself #14362

jadami10 opened this issue Nov 1, 2024 · 4 comments · May be fixed by #14385
Assignees

Comments

@jadami10
Copy link
Contributor

jadami10 commented Nov 1, 2024

Today, we only apply time pruning when the left hand side of an operand is strictly the time colum, https://github.com/apache/pinot/blob/master/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/TimeSegmentPruner.java#L370-L373

But we should still be able to apply time pruning when an operand is applied to the time column. The main case we've seen is folks will run a query with where date_trunc('day', <ts_column>) < <static_time>. This causes the time pruner to not be used.

The most naive way here would be to rebuild the interval tree in the broker with the min/max time columns for each segment also truncated. Though this would be problematic for tables with tens of thousands of segments.

I'm opening this as an issue for now to brainstorm if there's a universal solution or if there's easy operands that can be supported.

@Jackie-Jiang
Copy link
Contributor

We do have TimePredicateFilterOptimizer introduced in #6957. We should be able to enhance that to also support DATE_TRUNC

@jadami10
Copy link
Contributor Author

jadami10 commented Nov 3, 2024

great link, ty jackie. I won't have time to work on this immediately (our user also just modified their query instead), but I'll try to come back to it in free time

@ashishjayamohan
Copy link
Contributor

@jadami10
Mind if I take a stab at this?

@jadami10
Copy link
Contributor Author

jadami10 commented Nov 5, 2024

absolutely! feel free to tag me in to review

@ashishjayamohan ashishjayamohan linked a pull request Nov 5, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants