[feat] Enable the hash join to accept a pre-built hash table for joining #150
Conversation
|
Ke Jia seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
|
@JkSelf Thx 4 the contribution! Can you add unit tests to cover the change |
|
@JkSelf Can you explain this benchmark in more detail? For example, what is the test dataset and what is the scale factor? Is it a gain for a single query or an overall gain? Also, please sign the Contributor License, thank you! |
| * -------------------------------------------------------------------------- | ||
| */ | ||
|
|
||
| #include <boost/sort/pdqsort/pdqsort.hpp> |
There was a problem hiding this comment.
why move boost headers to first?
There was a problem hiding this comment.
Why move boost headers to first?
It may be clang-format that sorted the header files.
|
|
||
| const bool nullAware_; | ||
|
|
||
| void* reusedHashTableAddress_; |
There was a problem hiding this comment.
This will made HashJoinNode cannot be serialized across process and machines. So better to add some comment or CHECK
| std::unique_ptr< | ||
| exec::BaseHashTable, | ||
| std::function<void(exec::BaseHashTable*)>> | ||
| hashTable(nullptr, [](exec::BaseHashTable* ptr) { /* Do nothing */ }); |
|
|
||
| // True if this is a build side of an anti or left semi project join and has | ||
| // at least one entry with null join keys. | ||
| bool joinHasNullKeys_{false}; |
There was a problem hiding this comment.
| bool joinHasNullKeys_{false}; | |
| std::atomic<bool> joinHasNullKeys_{false}; |
6 participants
What problem does this PR solve?
Issue Number: close #149
Type of Change
Description
In Spark, the hash table is constructed only once in the driver and then broadcasted to each executor. In Gluten, we replace the broadcast hash join with a hash join, which leads to performance issues when the broadcast threshold increases. This is because each task must build its own hash table when using hash join.
This PR enables HashBuild to accept pre-built HashTable, thereby bypassing the hash table construction process. Additionally, it modifies HashProbe to avoid clearing the shared hash table after use.
Performance Impact
No Impact: This change does not affect the critical path (e.g., build system, doc, error handling).
Positive Impact: I have run benchmarks.
Click to view Benchmark Results
Negative Impact: Explained below (e.g., trade-off for correctness).
Release Note
Please describe the changes in this PR
Release Note:
Checklist (For Author)
Breaking Changes
No
Yes (Description: ...)
Click to view Breaking Changes