Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(query): parallel k way merge sort #16340

Merged
merged 41 commits into from
Sep 21, 2024
Merged

feat(query): parallel k way merge sort #16340

merged 41 commits into from
Sep 21, 2024

Conversation

forsaken628
Copy link
Collaborator

@forsaken628 forsaken628 commented Aug 28, 2024 •
edited
Loading

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

#16217

The merge path is not used

  • It is more suitable for the outer sort algorithm, for which the entire column is accessible. But it is much more troublesome for stream sorting.
  • It is only applicable to cascading merge algorithms, we are k-way merging.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@forsaken628
merge_path
abc18c9 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
KWaySortPartition
112cf2b 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
KWayMergeWorkerProcessor
a895b48 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
stream
01a6179 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
BlockStream
64010ca 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
update
fdf66be 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
event
9a2853c 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
builder
48c1d20 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
settings
ec46a9e 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
clean
47be95f 
Signed-off-by: coldWater <forsaken628@gmail.com>
@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Aug 28, 2024
@forsaken628
fix
3e71938 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
bypass some task
07ba446 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
test
0575e48 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
refine
f62b29b 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
Merge remote-tracking branch 'up/main' into merge-path
c0c4f0a
@forsaken628
enable
3f96f80 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
list domain
4acca29 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
refine
81bf4ad 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
integrate
80a47de 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
Merge remote-tracking branch 'up/main' into merge-path
72d2af0
@forsaken628
fix
79e7eed 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
fix
58f244f 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
search per iter
d7af3f7 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
fix
0be800a 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628
Merge remote-tracking branch 'up/main' into merge-path
0e98460 
Signed-off-by: coldWater <forsaken628@gmail.com>
@forsaken628 forsaken628 marked this pull request as ready for review September 4, 2024 16:08
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. A-query Area: databend query labels Sep 4, 2024
@forsaken628
fix
60f749f 
Signed-off-by: coldWater <forsaken628@gmail.com>
@sundy-li sundy-li added the ci-benchmark Benchmark: run all test label Sep 18, 2024
@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-16340-32e9681-1726651758

note: this image tag is only available for internal use,
please check the internal doc for more details.

@github-actions GitHub Actions
Copy link
Contributor

ClickBench Report

forsaken628 and others added 2 commits September 19, 2024 12:03
@forsaken628
with_block_size_hit
ef22d33 
Signed-off-by: coldWater <forsaken628@gmail.com>
@sundy-li
Merge branch 'main' into merge-path
d8336c6
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Sep 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-16340-48103f3-1726721880

note: this image tag is only available for internal use,
please check the internal doc for more details.

@github-actions GitHub Actions
Copy link
Contributor

ClickBench Report

@sundy-li sundy-li removed the ci-benchmark Benchmark: run all test label Sep 19, 2024
@forsaken628
fix
d2ff843 
Signed-off-by: coldWater <forsaken628@gmail.com>
@sundy-li sundy-li added this pull request to the merge queue Sep 20, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 20, 2024
@forsaken628 forsaken628 added this pull request to the merge queue Sep 21, 2024
Merged via the queue into databendlabs:main with commit c5f9329 Sep 21, 2024
71 of 72 checks passed
@forsaken628 forsaken628 deleted the merge-path branch September 21, 2024 05:32
@andylokandy andylokandy mentioned this pull request Oct 15, 2024
Merged
11 tasks
@andylokandy andylokandy mentioned this pull request Oct 25, 2024
Merged
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhyass zhyass zhyass left review comments

@sundy-li sundy-li sundy-li approved these changes

Assignees
No one assigned
Labels
A-query Area: databend query lgtm This PR has been approved by a maintainer pr-feature this PR introduces a new feature to the codebase size:XXL This PR changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@forsaken628 @sundy-li @zhyass