Periodically pause copying ledger nodes during online_delete

[ximinez]

High Level Overview of Change

Mitigates disk write contention while old ledgers are being deleted, and specifically while the full ledger is being copied to the "new" node store.

Context of Change

PR #4503 (reverted by #4882) attempted to improve rippled performance by writing batches to NuDB asynchronously. However, it had an unintended side effect that when online_delete writes the entire ledger to disk, it tends to cause the buffer to fill up, which results in blocking new ledgers from being persisted.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• Performance (increase or change in throughput and/or latency)

Before / After

Cost: This change will cause online_delete to take significantly longer to copy the full ledger from the old node store to the new one, which is the last significant step in the process.
Benefit: rippled should be much less likely to desync, and put less load on the disk during online_delete.

online_delete is still a demanding process, so this won't be a panacea, but it should be a significant improvment. (Significant improvement to be measured.)

Performance details

1. This is an improvement to existing functionality, which could be considered a bug fix.
2. The change impacts node store writes. Specifically, it should reduce contention between online_delete and writing new ledgers.
3. The impact should be measured in a couple of different ways.
   1. rippled should struggle less during online_delete to stay synced, and other functions.
   2. rippled should put less strain/demand on the disk during online_delete.
4. This change affects concurrent processing, in the sense that multiple threads are writing to the node store, especially during online_delete.

Note that back_off_milliseconds is configurable, defaulting to 100. Node operators can de-prioritize online_delete operations more by increasing this value to whatever they are comfortable with.

[ximinez]

Internal tracker: https://ripplelabs.atlassian.net/browse/RPFC-107

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 40.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 77.9%. Comparing base (838978b) to head (7c79952).

Files with missing lines	Patch %	Lines
src/xrpld/app/misc/SHAMapStoreImp.cpp	25.0%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           develop   #4907     +/-   ##
=========================================
- Coverage     77.9%   77.9%   -0.0%     
=========================================
  Files          782     782             
  Lines        66621   66625      +4     
  Branches      8161    8140     -21     
=========================================
- Hits         51902   51893      -9     
- Misses       14719   14732     +13     

Files with missing lines	Coverage Δ
src/xrpld/app/misc/SHAMapStoreImp.h	96.6% <100.0%> (ø)
src/xrpld/app/misc/SHAMapStoreImp.cpp	76.6% <25.0%> (-0.5%)	⬇️

... and 5 files with indirect coverage changes

[mtrippled]

👍 LGTM

[scottschurr]

👍 LGTM!

[ximinez]

Even though this has two approvals, I'm changing it back to draft because early perf tests did not show any change, and there may be a better / different solution.