Find jobs automatically from Hail Batch and abort them through cromwell

[EddieLF]

See https://centrepopgen.slack.com/archives/C082NA58R7S/p1748476503887639

Now takes a Hail batch ID via user input, and an auth token via config input.

Queries the batch and finds all watch jobs that haven't succeeded. Extracts the workflow ID for each unsuccessful watch job through the Hail batch log, and submits the workflow ID to cromwell abort workflow API.

Then, extracts any successful sub-workflow outputs and writes analyses for them.

[MattWellie]

This is super cool. Design wise I was curious about passing the auth token via config - is this something we can do with the Auth/key for the service account already in the driver job?

[EddieLF]

@MattWellie actually you're right, using the command line hailctl lets you use the auth from the batch service account.

E.g. https://batch.hail.populationgenomics.org.au/batches/615927/jobs/1

Do you think the right process for this script would be to use subprocess to run bash commands to query the different jobs / logs?

[MattWellie]

I was thinking that the generation of the auth token should be done within the batch (probably using subprocess) so that it isn't passed via config or plain text, either of which would mean the key being available in the batch config/logs