Non pre-emptible gVCF combiner jobs

[MattWellie]

By default this makes the Combiner host job non-preemptible. This alters the scheduling behaviour, and prevents the top level job being preempted/replaced. The quirk is this:

• We start a combiner job
• The combiner job starts an open QOB batch which carries out all the underlying tasks
• If the combiner job is preempted, it restarts, generating a new QOB batch
• Hail doesn't have a mechanic to abort the original QOB batch, so now we have two ongoing QOB batches doing the same work, writing to the same locations (unless the original one is manually cancelled)

worst case scenario here: https://batch.hail.populationgenomics.org.au/batches/588549/jobs/1 - multiple sporadic preemptions and a really muddy job history. The final job didn't appear to get close to a conclusion.

This change makes the single host/driver job non-spot to prevent preemption, which doesn't stop QOB jobs being preempted, but does stop multiple versions of the same batch being started. This job is relatively low-resource, so even a non-spot instance should represent a negligible cost relative to the gVCF combining.

---

Tagging @katiedelange and @Alexander-Stuckey as an FYI - this preemption/combiner duplication has been pretty frequent lately, and could easily happen in the PA pipeline

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 26.72%. Comparing base (7e06727) to head (438e430).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1096   +/-   ##
=======================================
  Coverage   26.72%   26.72%           
=======================================
  Files           9        9           
  Lines        1729     1729           
=======================================
  Hits          462      462           
  Misses       1267     1267           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.