QQ: checkpointing frequency improvements

[kjnilsson]

The current approach takes too many checkpoints which affects performance negatively, especially with large backlogs.

This PR takes an approach more similar to what was done for release cursors in 3.13.x.

Also add a force_checkpoint aux command that the purge operation
emits - this can also be used to try to force a checkpoint

The checkpointing config can be changed by setting the the quorum_queue_checkpoint_config persistent term:

persistent_term:set(quorum_queue_checkpoint_config, {MinIntervalMs, MinIndexes, MaxIndexes}).

the current values are: {1000, 4096, 666667} which means it will take a checkpoint at most every 1s as long as at least 4096 indexes have been applied. The min indexes between each checkpoint will grow in line with the message backlog up to at most 666667.

[michaelklishin]

The forced push was a rebase.

[michaelklishin]

My PerfTest tests do not observe any anomalies. With an 88M (22M per queue) message backlog, there are 22-23 checkpoints per queue. When the queues are drained, the checkpoints go away roughly at the rate of consumption of 1M messages.

I assume that a checkpoint taken every ≈ 1M messages is a reasonable rate.

With a 50M message backlog across 4 queues, the node takes 18s to start on a mostly idle 10 core machine with a reasonably fast 3 year old SSD.

With a workload that simulates peak throughput with 4 queues, 3 publishers and 3 consumers,
and monitors queue directories using watch -n 1 (once a second), a checkpoint appears and disappears roughly every 1M messages published (≈ 5s).

[kjnilsson]

I assume that a checkpoint taken every ≈ 1M messages is a reasonable rate.

depending on how many messages there are in the backlog it will grow the number of indexes between checkpoints from 4096 to ~1M (max) so yes that tallys. Cheers.