[SPARK-39328][SQL][TESTS] Fix flaky test SPARK-37753: Inhibit broadcast in left outer join when there are many empty partitions on outer/left side

[Last-remote11]

What changes were proposed in this pull request?

Improve test SPARK-37753: Inhibit broadcast in left outer join when there are many empty partitions on outer/left side of AdaptiveQueryExecSuite

Why are the changes needed?

This test appears to always succeed in the Apache GitHub Action runner environment, But some environments, test does not seem to proceed as intended.

On my environment:
4.18.0-553.8.1.el8_10.x86_64
Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
64G Mem
And ran test in master branch following the guide of official documentation

./build/sbt
testOnly org.apache.spark.sql.execution.adaptive.AdaptiveQueryExecSuite
...
- SPARK-37753: Inhibit broadcast in left outer join when there are many empty partitions on outer/left side *** FAILED ***
  The code passed to eventually never returned normally. Attempted 25 times over 15.040156205999999 seconds. Last failure message: 

even increasing the test's timeout to 1500 seconds results to failure after lots of retries.

SPARK-37753: Inhibit broadcast in left outer join when there are many empty partitions on outer/left side *** FAILED ***
  The code passed to failAfter did not complete within 20 minutes. (AdaptiveQueryExecSuite.scala:743)

---

The test says

    // if the right side is completed first and the left side is still being executed,
    // the right side does not know whether there are many empty partitions on the left side,
    // so there is no demote, and then the right side is broadcast in the planning stage.
    // so retry several times here to avoid unit test failure.
    eventually(timeout(15.seconds), interval(500.milliseconds)) {
...

It seems test failure occurs with very high probability by loading the ‘right side’ completes first.

While the reason is unclear, I believe it would be better to regulate the subquery loading speed in a predictable manner via applying simple udf rather than retrying until both sides load in the desired order.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Rerun the test.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

300 looks like a magic number, doesn't it? May I ask why you choose this value?

[Last-remote11]

Yes, I picked just a random magic number and there's no specific reason.