Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Aggregator support batch serialize #9777

Open
wants to merge 16 commits into
base: master
Choose a base branch
from
Open

Aggregator support batch serialize #9777

wants to merge 16 commits into from

Conversation

guo-shaoge
Copy link
Contributor

@guo-shaoge guo-shaoge commented Jan 9, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #9692

Problem Summary:

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. labels Jan 9, 2025
@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jan 9, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from guo-shaoge, ensuring that each of them provides their approval before proceeding. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed do-not-merge/needs-linked-issue labels Jan 9, 2025
@guo-shaoge guo-shaoge force-pushed the hashagg_batch_serialize branch 4 times, most recently from ff696d3 to 8bbb062 Compare January 14, 2025 16:06
@ti-chi-bot ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jan 22, 2025
@guo-shaoge
tmp save
f6bb074 
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge
refine
51e24d8 
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge
plan 1 code
543f1b8 
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge
refine
370bf0a 
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge
remove changes in count
a13008f 
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge
refine
817854b 
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge
refine
d9aeae7 
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge
refine
eb01d9e 
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge
refine
62fbbed 
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge
refine
0ce2f53 
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge
refine
c4c0eb9 
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge
agg batch serialize
7b8c19f 
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge
refine
241537d 
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge guo-shaoge force-pushed the hashagg_batch_serialize branch from 30a5b1c to 241537d Compare January 29, 2025 03:57
@ti-chi-bot ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 29, 2025
@guo-shaoge
batch deserialize
e2bdc81 
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge guo-shaoge force-pushed the hashagg_batch_serialize branch 2 times, most recently from d4116ce to fc95526 Compare February 4, 2025 04:09
@guo-shaoge
case ok
3354b7f 
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge guo-shaoge force-pushed the hashagg_batch_serialize branch from fc95526 to 3354b7f Compare February 4, 2025 04:59
guo-shaoge
guo-shaoge commented Feb 4, 2025
View reviewed changes
dbms/src/Interpreters/Aggregator.cpp
std::vector<size_t> & hashvals,
std::vector<typename Method::State::Derived::KeyHolderType> & key_holders,
std::vector<KeyHolderType> & key_holders,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo const

guo-shaoge
guo-shaoge commented Feb 4, 2025
View reviewed changes
dbms/src/Common/ColumnsHashing.h Outdated
void santityCheck() const
{
// Make sure init() has been called.
assert(sort_key_containers.size() == batch_rows.size() && !sort_key_containers.empty());
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo assert cur_batch_size < sort_key_containers

@guo-shaoge
key_string batch
3bb7660 
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Feb 4, 2025

@guo-shaoge: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-integration-test 3bb7660 link true /test pull-integration-test
pull-unit-test 3bb7660 link true /test pull-unit-test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Aggregator support batch allocate memory for key_serialized
1 participant
@guo-shaoge