Skip Mode #6039

wxtim · 2024-03-27T12:49:18Z

Closes #5641 (also fixes #5975, and fixes #5820 )

This branch includes (marked against skip mode proposal):

A new task run mode called "skip" to sit alongside the existing "live" and "simulation" and "dummy" modes.
Skip mode settings in `[runtime][<namespace>][skip].
Task-level control of the run mode - allowing run mode of tasks to override that of the workflow using [runtime][<namespace>]run mode.
a. Broadcast can change run mode for future task job submissions.
b. Cylc Validate and lint will warn about the setting not being live.
cylc set --out skip sets outputs from skip mode.
Tests to ensure that run mode = skip respects is_held flag.
Tests to ensure that force triggering a task will not override the run mode.

Extras
7. Run Mode is available as an task attribute in the UI
8. When tasks are run in skip mode, the prerequisites which correspond to the outputs they generate should be marked as satisfied by skip mode rather than satisfied naturally for provenance reasons. For the purpose of cylc remove logic, satisfied by skip mode should be treated the same as satisfied naturally.

There are two extensions, which I haven't dealt with yet, because I want to ensure that the basic functionality works, and move to the substantial documentation PR which need follow this.

Check List

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
Tests are included (or explain why tests are not needed).
CHANGES.md entry included if this is a change that can affect users
Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
Feature: PR against master

cylc/flow/cfgspec/workflow.py

cylc/flow/prerequisite.py

cylc/flow/scheduler_cli.py

cylc/flow/task_state.py

cylc/flow/scheduler.py

cylc/flow/task_pool.py

cylc/flow/network/schema.py

cylc/flow/task_state.py

cylc/flow/unicode_rules.py

cylc/flow/task_proxy.py

cylc/flow/task_pool.py

cylc/flow/run_modes/skip.py

cylc/flow/run_modes/nonlive.py

cylc/flow/cfgspec/workflow.py

Co-authored-by: Ronnie Dutta <61982285+MetRonnie@users.noreply.github.com>

MetRonnie · 2024-10-31T11:39:10Z

cylc/flow/task_outputs.py

+            exclude: Exclude one possible required message, allowing
+            specification of all required outputs if succeeded or failed.


Could this be explained better? I am not sure what this means (though judinging from the tests it is not as simple as just excluding the specified output from the return value, which was my first impression)

I've had a go.

cylc/flow/task_pool.py

cylc/flow/scheduler_cli.py

cylc/flow/run_modes/simulation.py

MetRonnie · 2024-11-11T17:00:47Z

cylc/flow/run_modes/skip.py

+        'disable task event handlers':
+            rtconfig['skip']['disable task event handlers'],


Actually, why add this to the platform dict instead of just accessing it on itask.tdef.rtconfig['simulation']['disable task event handlers']? (Or maybe setting it on itask.mode_settings for shorter access)

(disable task event handlers is not a settable platform configuration in global.cylc so this is slightly confusing)

This rtconfig object may include broadcasts. Broadcasts should apply to jobs submitted by a task, but should not modify the task proxy's definition. Hence this slight bodge.

cylc/flow/prerequisite.py

cylc/flow/config.py

cylc/flow/task_job_mgr.py

Co-authored-by: Ronnie Dutta <61982285+MetRonnie@users.noreply.github.com>

* upstream/master: Update cylc/flow/rundb.py GH Actions shortlog: check out head ref Add Paul Armstrong to CONTRIBUTING.md Report the error if write attempts fail to the DB Functional tests: allow `grep_fail` to accept options (cylc#6463) Bump dev version Prepare release 8.3.6 Bump pypa/gh-action-pypi-publish from 1.11.0 to 1.12.2 Simplify task state deltas Refactor task killing Bump pypa/gh-action-pypi-publish from 1.10.3 to 1.11.0 (cylc#6455) [skip ci] Spawn parentless sequential xtriggered task on set outputs (cylc#6448) Delete changes.d/6404.fix.md Pass pre-install plugins the value of workflow rundir as options.against source, allowing re-install and validate against source to access previous CLI opts as stored in rose-suite-cylc-install.conf

MetRonnie · 2024-11-14T13:45:06Z

cylc/flow/scheduler.py

        """
-        if self.get_run_mode() in {RunMode.SIMULATION, RunMode.DUMMY}:
+        if self.get_run_mode().value in WORKFLOW_ONLY_MODES:


It does not seem clear to me why you would not run event handlers in modes that apply to the workflow only, perhaps clearer to spell out the modes as before?

Suggested change

if self.get_run_mode().value in WORKFLOW_ONLY_MODES:

if self.get_run_mode() in {RunMode.SIMULATION, RunMode.DUMMY}:

MetRonnie · 2024-11-14T13:57:57Z

cylc/flow/task_job_mgr.py

+            if not submit_func:
+                # Return to nonlive.
+                nonlive_mode = False
+            else:
+                nonlive_mode = submit_func(
+                    self, itask, rtconfig, workflow, now)
+
+            if nonlive_mode:


Suggested change

if not submit_func:

# Return to nonlive.

nonlive_mode = False

else:

nonlive_mode = submit_func(

self, itask, rtconfig, workflow, now)

if nonlive_mode:

if submit_func and submit_func(

self, itask, rtconfig, workflow, now

):

wxtim marked this pull request as draft March 27, 2024 12:57

wxtim force-pushed the feature.skip_mode branch 2 times, most recently from b43d296 to d26c315 Compare March 28, 2024 09:11

oliver-sanders added this to the cylc-8.x milestone Apr 2, 2024

wxtim self-assigned this Apr 22, 2024

wxtim force-pushed the feature.skip_mode branch from d378e3a to e9c2e25 Compare April 22, 2024 13:57

wxtim requested review from oliver-sanders, markgrahamdawson and MetRonnie and removed request for markgrahamdawson April 23, 2024 10:00

wxtim marked this pull request as ready for review April 23, 2024 14:26

MetRonnie modified the milestones: 8.x, 8.4.0 Apr 23, 2024

MetRonnie reviewed Apr 23, 2024

View reviewed changes

cylc/flow/cfgspec/workflow.py Outdated Show resolved Hide resolved

cylc/flow/cfgspec/workflow.py Outdated Show resolved Hide resolved

cylc/flow/cfgspec/workflow.py Outdated Show resolved Hide resolved

cylc/flow/cfgspec/workflow.py Outdated Show resolved Hide resolved

MetRonnie added the config change Involves a change to global or workflow config label Apr 23, 2024

wxtim requested a review from MetRonnie April 24, 2024 15:10

wxtim force-pushed the feature.skip_mode branch 4 times, most recently from 3d002ee to dd0a9fc Compare April 25, 2024 12:56

wxtim force-pushed the feature.skip_mode branch from dd0a9fc to 19faa47 Compare May 2, 2024 12:46

MetRonnie reviewed May 8, 2024

View reviewed changes

wxtim force-pushed the feature.skip_mode branch 3 times, most recently from 0d42eab to c37ef72 Compare May 8, 2024 12:43

wxtim requested a review from MetRonnie May 8, 2024 13:47

wxtim force-pushed the feature.skip_mode branch 2 times, most recently from 3839a8a to d7ee653 Compare May 9, 2024 08:50

wxtim force-pushed the feature.skip_mode branch from 6677605 to 8de3073 Compare May 17, 2024 12:14

MetRonnie reviewed May 20, 2024

View reviewed changes

wxtim and others added 8 commits October 31, 2024 09:44

Type checking improved.

9333731

Update cylc/flow/task_outputs.py

b51ff06

fix outstanding broken test

d051fd1

undo some overzealous fixes to mypy.

509db85

sort test parameters: Prevents distribution clashes

31a753a

restore error message on restarting in a different mode

1f39b2c

use Enum correctly in a test

7c38690

Review suggestions

0c5e639

Co-authored-by: Ronnie Dutta <61982285+MetRonnie@users.noreply.github.com>

wxtim force-pushed the feature.skip_mode branch from a78b95d to 0c5e639 Compare October 31, 2024 09:45

remove un-needed check for itask.run_mode

2ec57e9

MetRonnie reviewed Oct 31, 2024

View reviewed changes

wxtim force-pushed the feature.skip_mode branch from aeca3e4 to 2aa3f1c Compare October 31, 2024 14:12

restore validate --run-mode

51f6585

wxtim force-pushed the feature.skip_mode branch from 2aa3f1c to 51f6585 Compare October 31, 2024 14:45

doc changes

40183c3

wxtim requested a review from MetRonnie October 31, 2024 15:01

MetRonnie mentioned this pull request Nov 4, 2024

Refactor task killing & simplify task state deltas #6457

Merged

8 tasks

Partially revert changes to Prerequisite (#64)

f9e8a83

MetRonnie reviewed Nov 11, 2024

View reviewed changes

wxtim and others added 6 commits November 12, 2024 10:33

Apply suggestions from code review

48c69fb

Co-authored-by: Ronnie Dutta <61982285+MetRonnie@users.noreply.github.com>

fix merge

0cc5b9e

remove now not needed taskdef.runmode infor

3f96ce7

f

7539686

f

8f992e1

wxtim requested a review from MetRonnie November 12, 2024 11:45

MetRonnie added the schema change Change to the Cylc GraphQL schema label Nov 12, 2024

MetRonnie mentioned this pull request Nov 12, 2024

Fix setting task run mode in Edit Runtime wxtim/cylc#65

Open

MetRonnie reviewed Nov 14, 2024

View reviewed changes

MetRonnie mentioned this pull request Nov 14, 2024

Clear up terminology wxtim/cylc#66

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Skip Mode #6039

Skip Mode #6039

wxtim commented Mar 27, 2024 •

edited

Loading

MetRonnie Oct 31, 2024

wxtim Oct 31, 2024

MetRonnie Nov 11, 2024

wxtim Nov 12, 2024

MetRonnie Nov 14, 2024

MetRonnie Nov 14, 2024

		exclude: Exclude one possible required message, allowing
		specification of all required outputs if succeeded or failed.

		'disable task event handlers':
		rtconfig['skip']['disable task event handlers'],

	if self.get_run_mode().value in WORKFLOW_ONLY_MODES:
	if self.get_run_mode() in {RunMode.SIMULATION, RunMode.DUMMY}:

Skip Mode #6039

Are you sure you want to change the base?

Skip Mode #6039

Conversation

wxtim commented Mar 27, 2024 • edited Loading

MetRonnie Oct 31, 2024

Choose a reason for hiding this comment

wxtim Oct 31, 2024

Choose a reason for hiding this comment

MetRonnie Nov 11, 2024

Choose a reason for hiding this comment

wxtim Nov 12, 2024

Choose a reason for hiding this comment

MetRonnie Nov 14, 2024

Choose a reason for hiding this comment

MetRonnie Nov 14, 2024

Choose a reason for hiding this comment

wxtim commented Mar 27, 2024 •

edited

Loading