Skip to content

feat(pageserver): add automatic trigger for gc-compaction #10221

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from

Conversation

skyzh
Copy link
Member

@skyzh skyzh commented Dec 20, 2024

Problem

part of #9114

Summary of changes

  • Add gc-compaction related config items.
  • Add l2_lsn into index_part.
  • Add auto trigger but not enabled-by-default and untested (will add later).

@skyzh skyzh force-pushed the skyzh/auto-trigger branch from 9f6cd02 to c254500 Compare December 20, 2024 21:25
@skyzh skyzh mentioned this pull request Dec 20, 2024
33 tasks
Copy link

github-actions bot commented Dec 20, 2024

7480 tests run: 7088 passed, 10 failed, 382 skipped (full report)


Failures on Postgres 17

Failures on Postgres 16

Failures on Postgres 15

Failures on Postgres 14

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_pageserver_gc_compaction_trigger[release-pg14] or test_pageserver_gc_compaction_trigger[release-pg14] or test_pageserver_gc_compaction_trigger[release-pg15] or test_pageserver_gc_compaction_trigger[release-pg15] or test_pageserver_gc_compaction_trigger[release-pg16] or test_pageserver_gc_compaction_trigger[release-pg16] or test_pageserver_gc_compaction_trigger[release-pg17] or test_pageserver_gc_compaction_trigger[release-pg17] or test_pageserver_gc_compaction_trigger[release-pg17] or test_pageserver_gc_compaction_trigger[release-pg17]"

Test coverage report is not available

The comment gets automatically updated with the latest test results
f84b979 at 2025-02-12T21:57:21.441Z :recycle:

@skyzh skyzh requested review from problame and arpad-m January 6, 2025 21:58
@skyzh skyzh marked this pull request as ready for review January 6, 2025 21:58
@skyzh skyzh requested a review from a team as a code owner January 6, 2025 21:58
@skyzh skyzh force-pushed the skyzh/auto-trigger branch from 3378f55 to bd45ff8 Compare January 9, 2025 18:49
@skyzh
Copy link
Member Author

skyzh commented Jan 9, 2025

cherry-picked #10330 out of this patch -- 10330 is crucial for me to stably run compaction experiments over the staging environment.

@skyzh skyzh marked this pull request as draft January 9, 2025 18:51
@skyzh skyzh force-pushed the skyzh/auto-trigger branch from bd45ff8 to 121cb17 Compare January 13, 2025 20:45
github-merge-queue bot pushed a commit that referenced this pull request Jan 21, 2025
…0455)

## Problem

part of #9114

The automatic trigger is already implemented at
#10221 but I need to write some
tests and finish my experiments in staging before I can merge it with
confidence. Given that I have some other patches that will modify the
config items, I'd like to get the config items merged first to reduce
conflicts.

## Summary of changes

* add `l2_lsn` to index_part.json -- below that LSN, data have been
processed by gc-compaction
* add a set of gc-compaction auto trigger control items into the config

---------

Signed-off-by: Alex Chi Z <chi@neon.tech>
@skyzh skyzh self-assigned this Feb 10, 2025
@skyzh skyzh force-pushed the skyzh/auto-trigger branch 2 times, most recently from 5f13d5f to 0d92214 Compare February 12, 2025 19:16
Signed-off-by: Alex Chi Z <chi@neon.tech>
@skyzh skyzh force-pushed the skyzh/auto-trigger branch from 0d92214 to 8d02f7f Compare February 12, 2025 19:17
Signed-off-by: Alex Chi Z <chi@neon.tech>
@skyzh skyzh closed this Feb 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant