Support simplify not for physical expr #18970

xudong963 · 2025-11-27T13:37:33Z

Part of the #18868

martin-g · 2025-11-27T20:55:53Z

datafusion/physical-expr/src/simplifier/not.rs

+            panic!("Expected binary OR expression result");
+        }
+
+        Ok(())


Please add a test for NOT(NOT(a) AND NOT(b)).
It should be simplified to a OR b but let's confirm.

done a0116ce

datafusion/physical-expr/src/simplifier/not.rs

martin-g · 2025-11-27T21:04:32Z

datafusion/physical-expr/src/simplifier/mod.rs


    fn f_up(&mut self, node: Self::Node) -> Result<Transformed<Self::Node>> {
+        // Apply NOT expression simplification first
+        let not_simplified = simplify_not_expr_recursive(&node, self.schema)?;


The name of the variable is a bit confusing - not_simplified. It sounds like it is not simplified.
Maybe rename it to not_expr_simplified ?!

Good point, agree

martin-g · 2025-11-27T21:08:36Z

datafusion/physical-expr/src/simplifier/not.rs

+            return Ok(Transformed::yes(lit(ScalarValue::Boolean(Some(!val)))));
+        }
+        if let ScalarValue::Boolean(None) = literal.value() {
+            return Ok(Transformed::yes(lit(ScalarValue::Boolean(None))));


Should this return Transformed::yes ?
It returns the same value.

Actually this is NOT(NULL) -> NULL. All is good!

martin-g · 2025-11-27T21:22:45Z

datafusion/physical-expr/src/simplifier/not.rs

+    if not_simplified.transformed {
+        // Recursively simplify the result
+        let further_simplified =
+            simplify_not_expr_recursive(&not_simplified.data, schema)?;


I wonder whether this could be a vector of an attack.
Someone may craft a query with many not(not(not(not(...)))) to cause a stack overflow due to the usage of recursion here.

I changed the recursion to iteration, and added a test for deep not, I verified the test under the old implemtaion, it will hang, but with the new way, it's okay. ceb85ff

xudong963 · 2025-12-01T11:31:16Z

Thanks for the review @martin-g

alamb

Thank you @xudong963 and @martin-g

I think prior to releasing this code we should make the following changes:

Avoid explicit recursion in not simplification (and instead use the tree node API)
Avoid the potential infinite loop

It would be nice to make them as part of this PR, but I also think it would be ok to do it as a follow on

Other things I think would be good to do but not strictly necessary:

Move the tests to the public interface (PhysicalExprSimplifier)
Avoid duplications of Operator::negate

alamb · 2025-12-01T17:14:20Z

datafusion/physical-expr/src/simplifier/mod.rs

        );
-        Ok(unwrapped)
+        // Combine transformation results
+        let final_transformed = transformed || unwrapped.transformed;


I think you can use transform_data here instead: https://docs.rs/datafusion/latest/datafusion/common/tree_node/struct.Transformed.html#method.transform_data

So something like

// Apply NOT expression simplification first let rewritten = simplify_not_expr(&node, self.schema)?.transform_data(|node| { unwrap_cast::unwrap_cast_in_comparison(node, self.schema) })?;

That handles combining the transformed flag for you

alamb · 2025-12-01T17:14:37Z

datafusion/physical-expr/src/simplifier/mod.rs

+
        // Apply unwrap cast optimization
        #[cfg(test)]
        let original_type = node.data_type(self.schema).unwrap();


It would be nice to move the original type check before applying the not and the verification after the not

Somethg like

impl<'a> TreeNodeRewriter for PhysicalExprSimplifier<'a> { type Node = Arc<dyn PhysicalExpr>; fn f_up(&mut self, node: Self::Node) -> Result<Transformed<Self::Node>> { #[cfg(test)] let original_type = node.data_type(self.schema).unwrap(); // Apply NOT expression simplification first let rewritten = simplify_not_expr(&node, self.schema)?.transform_data(|node| { unwrap_cast::unwrap_cast_in_comparison(node, self.schema) })?; #[cfg(test)] assert_eq!( rewritten.data.data_type(self.schema).unwrap(), original_type, "Simplified expression should have the same data type as the original" ); Ok(rewritten) } }

alamb · 2025-12-01T17:15:57Z

datafusion/physical-expr/src/simplifier/not.rs

+        ])
+    }
+
+    #[test]


Since this function is not really run in isolation (it is always run as part of PhysicalExprSimplifier) I think it would be better if these tests were in datafusion/physical-expr/src/simplifier/mod.rs rather than here.

I don't think this change is required

alamb · 2025-12-01T17:16:30Z

datafusion/physical-expr/src/simplifier/not.rs

+}
+
+/// Returns the negated version of a comparison operator, if possible
+fn negate_operator(op: &Operator) -> Option<Operator> {


This looks the same as Operator::negate: https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.Operator.html#method.negate

I recommend using that code, or if there is a reason not to use Operator::negate add a comment explaining why it is not used

alamb · 2025-12-01T17:24:26Z

datafusion/physical-expr/src/simplifier/not.rs

+                    Arc::new(NotExpr::new(Arc::clone(binary_expr.right())));
+
+                // Recursively simplify the NOT expressions
+                let simplified_left = simplify_not_expr(&not_left, schema)?;


I think since simplify_not_expr is used in PhysicalExprSimplifier which is doing a walk up the plan (using f_up) there is no reason to also recurse explicitly in this rule.

You should be able to apply the rewrite rules only NOT(A OR B) --> NOT A AND NOT B rather than also changing the exprs

alamb · 2025-12-01T17:27:07Z

datafusion/physical-expr/src/simplifier/not.rs

+    let mut current_expr = Arc::clone(expr);
+    let mut overall_transformed = false;
+
+    loop {


I am also somewhat concerned about this loop -- it seems like it may be trying to handle a non complete recursion and it could potentially lead to infinite recursion if a rewrite flip/flopped

If the loop is needed I suggest:

Put it in the higher level PhysicalExprSimplifier

add a counter that breaks after some number of iterations (e.g. 5 or something)

alamb · 2025-12-01T17:34:38Z

datafusion/physical-expr/src/simplifier/not.rs

+
+        assert!(result.transformed);
+        // Should be simplified back to the original b > 5
+        assert_eq!(result.data.to_string(), inner_expr.to_string());


What is the reason for using to_string()?

I tried comparing the two directly and it seems to work

assert_eq!(&result.data, &inner_expr);

alamb · 2025-12-01T17:35:29Z

datafusion/physical-expr/src/simplifier/not.rs

+        assert!(result.transformed);
+        // Should be simplified back to the original b > 5
+        assert_eq!(result.data.to_string(), inner_expr.to_string());


It might also make these queries easier to read if you made a helper like

fn assert_transformed(expr, expected_expr);

and

fn assert_not_transformed(expr, expected_expr);

alamb · 2025-12-01T17:37:08Z

datafusion/physical-expr/src/simplifier/not.rs

+        if let Some(literal) = result.data.as_any().downcast_ref::<Literal>() {
+            assert_eq!(literal.value(), &ScalarValue::Boolean(Some(false)));
+        } else {
+            panic!("Expected literal result");
+        }


You could potentially make this a function to make the tests easier to read - something like

let literal = as_literal(&result);

And handle the panic internally in as_literal

That way the mechics of the test wouldn't obscure the test logic so much

I think you could do something similar with BinaryExpr

alamb · 2025-12-01T17:37:40Z

datafusion/physical-expr/src/simplifier/not.rs

+        let schema = test_schema();
+
+        // Create a deeply nested NOT expression: NOT(NOT(NOT(...NOT(b > 5)...)))
+        // This tests that we don't get stack overflow with many nested NOTs


Support simplify not for physical expr

c01c335

github-actions bot added physical-expr Changes to the physical-expr crates execution Related to the execution crate labels Nov 27, 2025

xudong963 mentioned this pull request Nov 27, 2025

Row group limit pruning #18868

Open

martin-g reviewed Nov 27, 2025

View reviewed changes

alamb mentioned this pull request Nov 30, 2025

Andrew Lamb Weekly-ish Open Source plan - 2025-11-24 #18888

Closed

37 tasks

xudong963 added 4 commits December 1, 2025 09:57

not(not a and not b)

a0116ce

support not list

0d075ba

replace recursive and add test

ceb85ff

Merge branch 'main' into simplify_not

d749da7

github-actions bot removed the execution Related to the execution crate label Dec 1, 2025

xudong963 requested a review from zhuqi-lucas December 1, 2025 07:31

martin-g approved these changes Dec 1, 2025

View reviewed changes

alamb mentioned this pull request Dec 1, 2025

Andrew Lamb Weekly-ish Open Source plan - 2025-12-01 #19016

Open

31 tasks

alamb approved these changes Dec 1, 2025

View reviewed changes

+                      ])
+                  }
+                  #[test]

Support simplify not for physical expr #18970

Are you sure you want to change the base?

Support simplify not for physical expr #18970

Uh oh!

Conversation

xudong963 commented Nov 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xudong963 commented Dec 1, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants