Incremental `OPTIMIZE`

[hamersaw]

Currently, when calling OPTIMIZE on a table a background process starts which identifies all fragments that are undersized, computes a compaction plan where these are combined, and then submits separate compaction tasks to workers. In the case of a massive number of compactable fragments this process can be quite expensive and the OPTIMIZE can take a long time.

To update this process, we could bound the number of fragments that are rewritten within the OPTIMIZE command. This would allow for smaller jobs that complete faster, incrementally "checkpointing" the overall compaction.

[zhangyue19921010]

Hi @hamersaw, recently when we used Daft for Compaction on the client side, we also encountered similar issues: the Compaction plan for large tables takes too long to generate, and the high data volume leads to high costs for failure retries.

Recently, I have been building a new Compaction Planner based on Lance-Core that limits the overall data processing volume. Please refer to lance-format/lance#6039. One of the benefits is that it can be quickly reused for integration into different engines.

Perhaps, when implementing the OPTIMIZE optimization here, we can directly reuse this Planner through the Java API. Of course, I will complete the landing of the Bounded Compaction Planner as soon as possible.

[hamersaw]

Thanks, I wasn't aware of the work already done on this! It should be a pretty easy drop into Spark once completed. I would be happy to review when ready.