HIVE-29307: Incorrect split calculation causing less container to launch

[Aggarwal-Raghav]

What changes were proposed in this pull request?

Plesae check HIVE-29307

Why are the changes needed?

(int) waves is converting 1.7 (default) => 1 causing less number of container to launch.

Does this PR introduce any user-facing change?

NO

How was this patch tested?

In Prod scenario

[Aggarwal-Raghav]

CC @abstractdog , can you please provide your thoughts on this?

[deniskuzZ]

why not simply (int) (availableSlots * waves) as done in SplitGrouper?

[Aggarwal-Raghav]

yes it can be done and I'm also in favour of that. It just that, multiplyExact protects from overflow hence kept it. will update it after CI outcome.

[abstractdog]

This fix makes sense to me overall — split waves is indeed a float, so truncating it to an int is clearly a bug.

It’s worth noting that waves is buried deep inside the split generation logic, making it an expert-level setting. So it’s not surprising that this issue hasn’t been fixed until now (based on customer escalation history).

mm_all.q.out change also makes sense, it's the opposite of that was done in HIVE-19703

I wish we have a much cleaner unit test to reflect how these settings affect the number of splits, but it's not necessarily the scope of this PR

[Aggarwal-Raghav]

This fix makes sense to me overall — split waves is indeed a float, so truncating it to an int is clearly a bug.

It’s worth noting that waves is buried deep inside the split generation logic, making it an expert-level setting. So it’s not surprising that this issue hasn’t been fixed until now (based on customer escalation history).

mm_all.q.out change also makes sense, it's the opposite of that was done in HIVE-19703

I wish we have a much cleaner unit test to reflect how these settings affect the number of splits, but it's not necessarily the scope of this PR

Thanks for the reply @abstractdog . There is 1 UT failure mm_dp.q. I'm debugging that, seems relevant. I'll see if a junit test is possible for the test the number of containers launched logic.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud