add logging on segment selection to upsert compaction task

[dang-stripe]

We set the invalidRecords threshold too low on one of our tables and noticed the same segments were getting compacted over and over since new upserts were coming in. This adds some logs around segment selection and task execution to make it easier to figure out what the right threshold should be.

We've deployed this internally and have these logs on our clusters:

[2024-11-15 20:40:00.441000] INFO [UpsertCompactionTaskGenerator] [DefaultQuartzScheduler_Worker-8:164] Segment test_segment1 contains 163 invalid records out of 14549 total records (count threshold: 1, percent threshold: 0.0), adding it to the compaction list

[2024-11-15 20:40:59.865849] INFO [UpsertCompactionTaskExecutor] [TaskStateModelFactory-task_thread-33:17] Finished task: UpsertCompactionTask with configs: {uploadURL=http://controller1:9000/segments, crc=1736789201, validDocIdsType=SNAPSHOT, authToken=null, downloadURL=s3://deep_store/test_segment1, segmentName=test_segment1, TASK_ID=Task_UpsertCompactionTask_31961e82-0ccb-445f-8bd9-f1d88c339ac0_1731703200437_24, tableName=test_Table1}. Total time: 25077ms. Total docs before compaction: 26624. Total docs after compaction: 25804. Valid doc IDs count: 25804

cc @Jackie-Jiang

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 58.33333% with 5 lines in your changes missing coverage. Please review.

Project coverage is 63.80%. Comparing base (59551e4) to head (89139dc).
Report is 1340 commits behind head on master.

Files with missing lines	Patch %	Lines
...upsertcompaction/UpsertCompactionTaskExecutor.java	0.00%	5 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #14469      +/-   ##
============================================
+ Coverage     61.75%   63.80%   +2.05%     
- Complexity      207     1566    +1359     
============================================
  Files          2436     2663     +227     
  Lines        133233   146227   +12994     
  Branches      20636    22403    +1767     
============================================
+ Hits          82274    93300   +11026     
- Misses        44911    46036    +1125     
- Partials       6048     6891     +843     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (+99.99%)	⬆️
integration	100.00% <ø> (+99.99%)	⬆️
integration1	100.00% <ø> (+99.99%)	⬆️
integration2	0.00% <ø> (ø)
java-11	63.77% <58.33%> (+2.06%)	⬆️
java-21	63.69% <58.33%> (+2.06%)	⬆️
skip-bytebuffers-false	63.79% <58.33%> (+2.05%)	⬆️
skip-bytebuffers-true	63.65% <58.33%> (+35.92%)	⬆️
temurin	63.80% <58.33%> (+2.05%)	⬆️
unittests	63.80% <58.33%> (+2.05%)	⬆️
unittests1	55.54% <ø> (+8.65%)	⬆️
unittests2	34.13% <58.33%> (+6.40%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.