Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Separate hash aggregate finalization into its own operator #4913

Merged
merged 1 commit into from
Feb 15, 2025

Conversation

benjaminwinger
Copy link
Collaborator

@benjaminwinger benjaminwinger commented Feb 14, 2025

This fixes the data race I mentioned in #4906. It's a little difficult to rigorously test since I was having a hard time reproducing it, but moving the finalization into a separate operator should guarantee that all threads running HashAggregate operator tasks finish before finalization starts.

I think this may fix the issue in #4903, but I'm not positive that it's the same issue.

@benjaminwinger benjaminwinger force-pushed the aggregate-finalize-operator branch from 7badfe7 to 35c4b2d Compare February 14, 2025 22:43
Copy link

codecov bot commented Feb 14, 2025

Codecov Report

Attention: Patch coverage is 97.29730% with 1 line in your changes missing coverage. Please review.

Project coverage is 86.52%. Comparing base (cf860af) to head (35c4b2d).
Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
...lude/processor/operator/aggregate/hash_aggregate.h 94.44% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##           master    #4913   +/-   ##
=======================================
  Coverage   86.51%   86.52%           
=======================================
  Files        1403     1403           
  Lines       60665    60686   +21     
  Branches     7461     7466    +5     
=======================================
+ Hits        52484    52507   +23     
+ Misses       8011     8010    -1     
+ Partials      170      169    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

Benchmark Result

Master commit hash: cf860af7a6f8be5c0d9754c842946b32264d15cf
Branch commit hash: 837e0cc1b33205d50354cdab96b3217c2f57ac45

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 724.67 714.55 10.12 (1.42%)
aggregation q28 6398.28 6375.88 22.40 (0.35%)
filter q14 128.98 119.73 9.26 (7.73%)
filter q15 125.02 116.20 8.82 (7.59%)
filter q16 305.26 298.32 6.93 (2.32%)
filter q17 446.61 442.95 3.66 (0.83%)
filter q18 1901.98 1958.85 -56.87 (-2.90%)
filter zonemap-node 90.11 80.49 9.62 (11.95%)
filter zonemap-node-lhs-cast 90.55 80.42 10.14 (12.60%)
filter zonemap-node-null 91.52 80.19 11.33 (14.13%)
filter zonemap-rel 5635.69 5583.15 52.54 (0.94%)
fixed_size_expr_evaluator q07 570.13 563.21 6.91 (1.23%)
fixed_size_expr_evaluator q08 800.42 796.22 4.19 (0.53%)
fixed_size_expr_evaluator q09 801.35 793.83 7.52 (0.95%)
fixed_size_expr_evaluator q10 237.60 228.52 9.08 (3.97%)
fixed_size_expr_evaluator q11 229.16 220.85 8.30 (3.76%)
fixed_size_expr_evaluator q12 226.54 218.59 7.95 (3.64%)
fixed_size_expr_evaluator q13 1449.95 1455.68 -5.73 (-0.39%)
fixed_size_seq_scan q23 110.59 103.69 6.90 (6.65%)
join q29 758.73 708.82 49.91 (7.04%)
join q30 9933.00 10144.89 -211.89 (-2.09%)
join q31 7.38 7.93 -0.55 (-6.98%)
join SelectiveTwoHopJoin 54.69 58.89 -4.20 (-7.13%)
ldbc_snb_ic q35 2663.71 2682.28 -18.57 (-0.69%)
ldbc_snb_ic q36 468.50 448.07 20.43 (4.56%)
ldbc_snb_is q32 6.02 6.61 -0.58 (-8.83%)
ldbc_snb_is q33 15.44 14.27 1.16 (8.15%)
ldbc_snb_is q34 1.19 1.15 0.04 (3.09%)
multi-rel multi-rel-large-scan 1324.01 1321.16 2.85 (0.22%)
multi-rel multi-rel-lookup 45.31 30.54 14.78 (48.40%)
multi-rel multi-rel-small-scan 65.89 104.66 -38.77 (-37.05%)
order_by q25 132.41 120.56 11.86 (9.84%)
order_by q26 457.28 445.27 12.01 (2.70%)
order_by q27 1415.42 1406.15 9.28 (0.66%)
recursive_join recursive-join-bidirection 293.46 289.32 4.14 (1.43%)
recursive_join recursive-join-dense 7334.84 7345.79 -10.95 (-0.15%)
recursive_join recursive-join-path 24444.58 24144.04 300.54 (1.24%)
recursive_join recursive-join-sparse 1058.26 1051.52 6.74 (0.64%)
recursive_join recursive-join-trail 7340.25 7326.78 13.48 (0.18%)
scan_after_filter q01 172.45 165.35 7.10 (4.29%)
scan_after_filter q02 158.87 148.69 10.18 (6.85%)
shortest_path_ldbc100 q37 96.54 99.21 -2.68 (-2.70%)
shortest_path_ldbc100 q38 384.99 366.94 18.05 (4.92%)
shortest_path_ldbc100 q39 65.57 66.71 -1.14 (-1.71%)
shortest_path_ldbc100 q40 449.63 463.08 -13.46 (-2.91%)
var_size_expr_evaluator q03 2069.50 2133.67 -64.17 (-3.01%)
var_size_expr_evaluator q04 2247.14 2229.90 17.25 (0.77%)
var_size_expr_evaluator q05 2630.26 2612.23 18.03 (0.69%)
var_size_expr_evaluator q06 1343.83 1320.33 23.50 (1.78%)
var_size_seq_scan q19 1458.23 1439.94 18.29 (1.27%)
var_size_seq_scan q20 2484.80 2473.11 11.70 (0.47%)
var_size_seq_scan q21 2328.46 2296.57 31.89 (1.39%)
var_size_seq_scan q22 126.68 124.49 2.18 (1.75%)

Copy link
Contributor

@ray6080 ray6080 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thanks!

@ray6080 ray6080 merged commit 0e61d73 into master Feb 15, 2025
25 checks passed
@ray6080 ray6080 deleted the aggregate-finalize-operator branch February 15, 2025 03:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants