Code

Adding new choice to --on-error#1974

Code

Open

AlexTate wants to merge 30 commits intocommon-workflow-language:maincommon-workflow-language/cwltool:mainfrom AlexTate:on-error-abortAlexTate/cwltool:on-error-abortCopy head branch name to clipboard

+402-102

Commits

Commits on Nov 13, 2024

• Adding inputs for new "kill" mode of the on-error parameter

  

  AlexTate

  authored and

  mr-c

  committed
  572e865

• Adding kill switch trigger. If a job's processStatus != "success" and runtimeContext.on_error = "kill", then the switch is activated. WorkflowKillSwitch is raised so it can be handled at the workfl…

  Show description for d7b7712
  

  AlexTate

  authored and

  mr-c

  committed
  d7b7712

• Actively running jobs respond to the kill switch by checking the switch's status in the monitor function. The monitor function, up to this point, has been for gathering memory usage statistics via …

  Show description for 788a89d
  

  AlexTate

  authored and

  mr-c

  committed
  788a89d

• When the WorkflowKillSwitch exception reaches the TaskQueue, try to prevent pending tasks from starting by simply draining the queue. This is a very loose policy, but since kill switch response is …

  Show description for de73938
  

  AlexTate

  authored and

  mr-c

  committed
  de73938

• Propagation of the WorkflowKillSwitch exception stops once it reaches an executor. The workflow_eval_lock release had to be moved to the finally block in MultithreadedJobExecutor.run_jobs(). Otherw…

  Show description for a9a2ab6
  

  AlexTate

  authored and

  mr-c

  committed
  a9a2ab6

• Don't create runtime_context.kill_switch by default

  Show description for 0a00582
  mr-c

  committed
  0a00582

• This commit brings kill switch initialization and monitoring to the TaskQueue. This helps to better synchronize the kill switch event and avoid adding/executing tasks after the switch has been set.

  Show description for beaa3f4
  

  AlexTate

  authored and

  mr-c

  committed
  beaa3f4

• JobBase._execute() previously skipped some important post-job actions when exiting due to kill switch. Those actions have been placed under a finally block so that they are executed by both the "…

  Show description for b52333d
  

  AlexTate

  authored and

  mr-c

  committed
  b52333d

• Adding a unit test. Two outcomes are measured:

  Show description for a1ab910
  

  AlexTate

  authored and

  mr-c

  committed
  a1ab910

• Formatting corrections provided by make cleanup which are relevant to this issue. Other changes were offered by the tool, but they are outside the scope of this issue.

  

  AlexTate

  authored and

  mr-c

  committed
  5924cfe

• cleanups

  mr-c

  committed
  1bee55f

• DO NOT MERGE, let all tests run

  mr-c

  committed
  7eb2c0d

• Removing irrelevant debug line from test body

  

  AlexTate

  authored and

  mr-c

  committed
  03cde29

• Removing CPU resource constraints for the test. My intention is to have MultithreadedJobExecutor ignore allocated resources when deciding whether to run the next parallel job. The steps in this wor…

  Show description for 2b9d76a
  

  AlexTate

  authored and

  mr-c

  committed
  2b9d76a

• Using a different approach to remove the "one job per core" resource constraint. The current ResourceRequirement implementation doesn't allow {coresMin: 0}. However, this can still be achieved with…

  Show description for 36f12cf
  

  AlexTate

  authored and

  mr-c

  committed
  36f12cf

• Adding suggested changes to the --on-error help string per @mr-c. Formatting compliance updates in test_parallel.py

  

  AlexTate

  authored and

  mr-c

  committed
  791fe7b

• Missed running make mypy before pushing last commit. Opting to disable type checking on selectResources() because it's just an implementation detail for the test

  

  AlexTate

  authored and

  mr-c

  committed
  93948fb

• don't cancel jobs; typecheck the test just in case

  mr-c

  committed
  d938bf8

• test_on_error_kill: allow for some longer runtime due to busy systems

  mr-c

  committed
  5d92d2d

• Fixing ReceiveScatterOutput to allow outputs to be collected from successful steps when 1) these steps are upstream from a scattered subworkflow, 2) the workflow kill switch is activated by one of …

  Show description for fe53fb8
  

  AlexTate

  authored and

  mr-c

  committed
  fe53fb8

• Simplifying value checks for RuntimeContext.on_error in parallel_steps() and WorkflowJob.job().

  Show description for b639dfa
  

  AlexTate

  authored and

  mr-c

  committed
  b639dfa

• Adding __repr__ to WorkflowStep, WorkflowJob, and WorkflowJobStep classes. This can be VERY helpful while debugging, particularly when unraveling callback chains.

  

  AlexTate

  authored and

  mr-c

  committed
  8e0b6f2

• It is critical for WorkflowJob.processStatus to update in WorkflowJob.receive_output(). Otherwise, MultithreadedJobExecutor.run_jobs() will likely stop iterating over the topmost WorkflowJob.job() …

  Show description for 32b22d8
  

  AlexTate

  authored and

  mr-c

  committed
  32b22d8

• Simplifying the approach in on-error_kill.cwl. It now activates the kill switch via a ToolTimeLimit requirement. It also uses a much longer timeout which will hopefully be sufficient for the CI ser…

  Show description for 6485cff
  

  AlexTate

  authored and

  mr-c

  committed
  6485cff

• Reformatting and type hint updates

  

  AlexTate

  authored and

  mr-c

  committed
  beb6af7

• Formatting mistake fix

  

  AlexTate

  authored and

  mr-c

  committed
  50f2793

• Adding instance ID to repr strings, improving readability of WorkflowStep's repr string, and adding docstrings.

  Show description for 20fd703
  

  AlexTate

  authored and

  mr-c

  committed
  20fd703

• Changing TaskQueue constructor to accept the kill_switch threading.Event as an argument rather than an entire RuntimeContext, per @mr-c

  

  AlexTate

  authored and

  mr-c

  committed
  e8f9284

• codecov: don't fail CI if there is an error

  mr-c

  committed
  f17f678

Commits on Nov 20, 2024

• Merge branch 'main' into on-error-abort

  mr-c

  authored
  e139fae