Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add new 'RAW' variables for SHARE and WORK variables to ensure share/work dirs are created, not left with broken symlinks #5978

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add new 'RAW' variables for SHARE and WORK variables to ensure share/work dirs are created, not left with broken symlinks #5978

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions cylc/flow/etc/job.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,10 @@ cylc__job__main() {
export CYLC_WORKFLOW_LOG_DIR="${CYLC_WORKFLOW_RUN_DIR}/log/scheduler"
export CYLC_WORKFLOW_SHARE_DIR="${CYLC_WORKFLOW_RUN_DIR}/share"
export CYLC_WORKFLOW_WORK_DIR="${CYLC_WORKFLOW_RUN_DIR}/work"
# Make sure the 'raw' version of SHARE and WORK are defined.
# Default to the ones defined above.
export CYLC_WORKFLOW_SHARE_DIR_RAW="${CYLC_WORKFLOW_SHARE_DIR_RAW:-"${CYLC_WORKFLOW_SHARE_DIR}"}"
export CYLC_WORKFLOW_WORK_DIR_RAW="${CYLC_WORKFLOW_WORK_DIR_RAW:-"${CYLC_WORKFLOW_WORK_DIR}"}"
export CYLC_TASK_CYCLE_POINT="${CYLC_TASK_JOB%%/*}"
export CYLC_TASK_NAME="${CYLC_TASK_JOB#*/}"
CYLC_TASK_NAME="${CYLC_TASK_NAME%/*}"
Expand All @@ -95,6 +99,7 @@ cylc__job__main() {
CYLC_TASK_WORK_DIR_BASE="${CYLC_TASK_CYCLE_POINT}/${CYLC_TASK_NAME}"
fi
export CYLC_TASK_WORK_DIR="${CYLC_WORKFLOW_WORK_DIR}/${CYLC_TASK_WORK_DIR_BASE}"
export CYLC_TASK_WORK_DIR_RAW="${CYLC_WORKFLOW_WORK_DIR_RAW}/${CYLC_TASK_WORK_DIR_BASE}"
typeset contact="${CYLC_WORKFLOW_RUN_DIR}/.service/contact"
if [[ -f "${contact}" ]]; then
# (contact file not present for polled platforms)
Expand Down Expand Up @@ -134,9 +139,8 @@ cylc__job__main() {
export PATH="${CYLC_WORKFLOW_RUN_DIR}/share/bin:${CYLC_WORKFLOW_RUN_DIR}/bin:${PATH}"
export PYTHONPATH="${CYLC_WORKFLOW_RUN_DIR}/share/lib/python:${CYLC_WORKFLOW_RUN_DIR}/lib/python:${PYTHONPATH:-}"
# Create share and work directories
mkdir -p "${CYLC_WORKFLOW_SHARE_DIR}" || true
mkdir -p "$(dirname "${CYLC_TASK_WORK_DIR}")" || true
mkdir -p "${CYLC_TASK_WORK_DIR}"
mkdir -p "${CYLC_WORKFLOW_SHARE_DIR_RAW}" || true
mkdir -p "${CYLC_TASK_WORK_DIR_RAW}"
cd "${CYLC_TASK_WORK_DIR}"
# Env-Script, User Environment, Pre-Script, Script and Post-Script
# Run user scripts in subshell to protect cylc job script from interference.
Expand Down
8 changes: 8 additions & 0 deletions cylc/flow/job_file.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,14 @@
if var not in ('CYLC_DEBUG', 'CYLC_VERBOSE', 'CYLC_WORKFLOW_ID'):
handle.write('\n export %s="%s"' % (var, val))

for dir_ in ("share", "work"):
val = job_conf["symlink_dirs"].get(dir_, None)
if val is not None:
handle.write(

Check warning on line 211 in cylc/flow/job_file.py

View check run for this annotation

Codecov / codecov/patch

cylc/flow/job_file.py#L211 

Added line #L211 was not covered by tests
'\n '
f'export CYLC_WORKFLOW_{dir_.upper()}_DIR_RAW="{val}"'
)

if str(self.workflow_env.get('CYLC_UTC')) == 'True':
handle.write('\n export TZ="UTC"')

Expand Down
8 changes: 7 additions & 1 deletion cylc/flow/task_job_mgr.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,10 @@
pdeepcopy,
poverride
)
from cylc.flow.pathutil import get_remote_workflow_run_job_dir
from cylc.flow.pathutil import (
get_dirs_to_symlink,
get_remote_workflow_run_job_dir,
)
from cylc.flow.platforms import (
get_host_from_platform,
get_install_target_from_platform,
Expand Down Expand Up @@ -1345,6 +1348,9 @@ def get_job_conf(
'pre-script': rtconfig['pre-script'],
'script': rtconfig['script'],
'submit_num': itask.submit_num,
'symlink_dirs': get_dirs_to_symlink(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this were to go forward, I imagine making sure get_dirs_to_symlink was cached would be a reasonable thing to do for a minor speedup with limited extra memory load.

itask.platform['install target'], workflow
),
'flow_nums': itask.flow_nums,
'workflow_name': workflow,
'task_id': itask.identity,
Expand Down
Loading