Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gc multiple indexes at once #5380

Merged
merged 5 commits into from
Sep 5, 2024
Merged

gc multiple indexes at once #5380

merged 5 commits into from
Sep 5, 2024

Conversation

trinity-1686a
Copy link
Contributor

Description

group deletion of splits by the gc. Before this, we'd do 2 list_splits per index every gc period (10m), now we do 2 list_splits in total (assuming pagination doesn't make us do more calls, in both cases). We can't group split deletion between index, so we still do n delete_splits, but n is the number of index with splits to delete, not the number of splits, so it ought to not be as bad.

2nd part of fix for #5353

How was this PR tested?

updated tests

@trinity-1686a trinity-1686a changed the title Trinity/collective gc gc multiple indexes at once Sep 3, 2024
Copy link

github-actions bot commented Sep 3, 2024

On SSD:

Average search latency is 1.01x that of the reference (lower is better).
Ref run id: 3248, ref commit: 80364dd
Link

On GCS:

Average search latency is 1.05x that of the reference (lower is better).
Ref run id: 3249, ref commit: 80364dd
Link

@trinity-1686a trinity-1686a enabled auto-merge (squash) September 5, 2024 09:37
@trinity-1686a trinity-1686a merged commit e905929 into main Sep 5, 2024
5 checks passed
@trinity-1686a trinity-1686a deleted the trinity/collective-gc branch September 5, 2024 09:38
This was referenced Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants