batch delete from gc

[trinity-1686a]

Description

follow up to #5380
if a cluster generate less than 1k split per 10 minutes, everything was fine, but in case it would generate more than that, we have a slow down for 2 reasons:

• we no longer run 10 tasks in parallel
• we may get a batch of 1k splits from many different indexes, and get a 2nd batch from the same set of indexes, meaning more than one delete per index, despite having few splits to delete for each indexes. This can in theory cause us to do up to n_index time more delete calls to the metastore than necessary (caped to 1k)

Changes made:

• we run up to 10 deletion from storage + metastore concurrently
• we retrieve splits sorted by index_uid from metastore, so we don't do many delete per index
• we query for 10k splits instead of 1k: before we would query for 10 * 1k, but as this is now batched, we can increase this number to keep a similar effect to before

How was this PR tested?

tested on a small cluster to not be utterly broken, for perf improvement, we'll need a way bigger cluster

[trinity-1686a]

note: i think this is a bad idea to do this long term, but i'd rather this fix be made without a migration for now, so we can easily revert clusters affected to previous versions if we find things are somehow still not good. If you disagree, we can either create the required index manually, or make a migration

[github-actions]

On SSD:

Average search latency is 1.01x that of the reference (lower is better).
Ref run id: 3358, ref commit: ec951aa
Link

On GCS:

Average search latency is 1.2x that of the reference (lower is better).
Ref run id: 3359, ref commit: ec951aa
Link