Draft Changelog

The following items should be added to the changelog for the next Toil release, at which point the list should be cleared. Add new items at the bottom:

Version 7.x Draft

Changelog

Highlighted Features Added

• toil debug-job now has --retrieveTaskDirectory <dir> which will set up a job's downloaded files under <dir> and try to stop the job after doing the downloads. Jobs can call self.files_downloaded_hook() to provide a stopping point for this mode. (#4815)
• toil debug-job can now reconstruct the inside-the-container environment for CWL and WDL tasks. (#4815)
• Added support for caching on Slurm and other HPC schedulers (#4775)
• Replace all instances of boto2 with boto3 for all Toil AWS code (#4718)
• Add support for Python 3.12 (#4718)
• Ceph input/output errors from file locking functions are now tolerated. (#4874)
• Toil now uses flock to enable directory locks to work properly (#4924)
• Added support to get Slurm partitions and automatically send jobs to GPUs on Slurm (#4833) (supports both CWL and WDL)
• New --symlinkJobStoreReads=False option lets you force local-node copies (possibly in the cache) even when reading directly from a FileJobStore is possible, potentially reducing shared filesystem IO. (#4673)

Breaking Changes

CWL

• Prevent simultaneous Singularity container pulls in toil-cwl-runner (#4990)
• Added support to import files on workers for toil-cwl-runner (#5025)
  • --runImportsOnWorkers to enable importing files on workers
  • --importWorkersDisk to control how much disk space the import worker will use

WDL

• Update WDL conformance tests on CI (#4875)
• Added support to run task only WDL files (#4960)
• Added support for the gpu field in WDL (#4949)
• Support passing inputs into toil-wdl-runner for task only WDLs (#4977)
• toil-wdl-runner will now carry through task exit codes (#4978)
• toil-wdl-runner will respect explicit null values for optional inputs (#4981)
• toil-wdl-runner will not immediately error on nonexistent coerced files until outputted (#4994)
  • File? type for string to file coercion is now supported (will be nullified)
• WDL output files will now live in directories named after their tasks instead of UUID directories (#5008)
• Fixed a bug with conditional statements inside a WDL scatter (#5055)
• toil-wdl-runner now correctly finds and returns outputs from tasks in scatters and conditionals when a WDL workflow lacks an output section. (#5094)
• toil-wdl-runner has a new --allCallOutputs option to allow including all calls' outputs in a workflow's output. (#5093)
• toil-wdl-runner can now detect and try not to delete the outputs of a workflow that is meant to use the Cromwell Output Organizer (croo). Note that croo still can't actually work on the output of toil-wdl-runner. (#5093)
• --allCallOutputs no longer discards WDL workflow outputs section outputs. (#5106)
• File virtualization in toil-wdl-runner now only happens at task boundaries (#5028)
  • File to String coercion should be supported
• Added support to import files on workers for toil-wdl-runner (#5103)

Kubernetes

Dependencies

Misc

• Makefile: use isolated builds, add dist target (sdist+wheel) and deprecate the sdist target. (#4820) (#4826)
• Toil will now wait --jobStoreTimeout seconds (default: 30) to see an update to/removal of a job that was run, and will not let the job succeed unless it is seen to make progress. (#3814)
• Toil job descriptions no longer have a command field, and we track the link to the job body and the command to invoke the Toil worker separately. (#4811)
• Several typos in the docs were fixed (#4889)
• Add a test to ensure batchsystem plugins are installable (#4879)
• Fix Toil utils to work without the AWS extra (#4953)
• Print commit hash with toil --version when installed from source. Before: 7.1.0a1. After: 7.1.0a1-ccf57e6071e32675daabdcbacb91988e871745a9 (#4954)
• Fixed a broken URL and an omitted variable in CI tests (#4974)
• Generate default config correctly (#5014)
• Use the latest setuptools when running cactus. (#5017)
• Toil will refuse to proceed if it detects that its coordination directory or a Singularity cache directory it needs to lock is on Ceph, to prevent hanging the Ceph MDS (#4972)
• Fix a NotImplementedError in the Grid Engine batchsystem (#5061)
  • Added basic Grid Engine CI tests
• Update Cactus on CI to 2.9.0 (#5062)
• Separate out create/delete iam role functions into lib.
• Remove deprecated pipes module (#5122)

Thank you to our contributors: