Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Workflow error with HTTP files and --cachedir #1828

Open
bartns opened this issue Apr 19, 2023 · 0 comments · May be fixed by #1829
Open

Workflow error with HTTP files and --cachedir #1828

bartns opened this issue Apr 19, 2023 · 0 comments · May be fixed by #1829

Comments

@bartns
Copy link

bartns commented Apr 19, 2023

Having files HTTP files as input, defining a cache directory with the --cachedir option results in a Workflow error.
Runs successfully without.

Expected Behavior

Same behaviour as without --cachedir defined. (But utilizing the cache functionality)

Actual Behavior

Workflow Error

Workflow Code

cwltool --debug --cachedir ./CACHE concatenate.cwl --infiles http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz --outname cattest

concatenate.cwl

cwlVersion: v1.2
class: CommandLineTool
label: "Concatenate multiple files"

baseCommand: [cat]

stdout: $(inputs.outname)

hints:
  DockerRequirement:
    dockerPull: debian:buster
    
inputs:
  infiles:
    type: File[]
    inputBinding:
        position: 2
  outname:
    type: string

outputs:
  output:
    type: File
    outputBinding:
        glob: $(inputs.outname)

Full Traceback

cwltool --debug --cachedir ./CACHE concatenate.cwl --infiles http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz --outname cattest
INFO /home/bart/.local/bin//cwltool 3.1.20230325110543
INFO Resolved 'concatenate.cwl' to 'file:///home/bart/UNLOCK/git-dev/cwl/tools/bash/concatenate.cwl'
DEBUG Parsed job order from command line: {
    "id": "concatenate.cwl",
    "infiles": [
        {
            "class": "File",
            "location": "http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz"
        }
    ],
    "outname": "cattest"
}
ERROR Got workflow error
Traceback (most recent call last):
  File "/home/bart/.local/lib/python3.10/site-packages/cwltool/executors.py", line 217, in run_jobs
    for job in jobiter:
  File "/home/bart/.local/lib/python3.10/site-packages/cwltool/command_line_tool.py", line 835, in job
    visit_class([cachebuilder.files, cachebuilder.bindings], ("File"), _checksum)
  File "/home/bart/.local/lib/python3.10/site-packages/cwltool/utils.py", line 216, in visit_class
    visit_class(d, cls, op)
  File "/home/bart/.local/lib/python3.10/site-packages/cwltool/utils.py", line 216, in visit_class
    visit_class(d, cls, op)
  File "/home/bart/.local/lib/python3.10/site-packages/cwltool/utils.py", line 211, in visit_class
    op(rec)
  File "/home/bart/.local/lib/python3.10/site-packages/cwltool/process.py", line 1348, in compute_checksums
    with fs_access.open(location, "rb") as f:
  File "/home/bart/.local/lib/python3.10/site-packages/cwltool/stdfsaccess.py", line 38, in open
    return open(self._abs(fn), mode)
FileNotFoundError: [Errno 2] No such file or directory: 'http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz'
ERROR Workflow error:
[Errno 2] No such file or directory: 'http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz'
Traceback (most recent call last):
  File "/home/bart/.local/lib/python3.10/site-packages/cwltool/executors.py", line 217, in run_jobs
    for job in jobiter:
  File "/home/bart/.local/lib/python3.10/site-packages/cwltool/command_line_tool.py", line 835, in job
    visit_class([cachebuilder.files, cachebuilder.bindings], ("File"), _checksum)
  File "/home/bart/.local/lib/python3.10/site-packages/cwltool/utils.py", line 216, in visit_class
    visit_class(d, cls, op)
  File "/home/bart/.local/lib/python3.10/site-packages/cwltool/utils.py", line 216, in visit_class
    visit_class(d, cls, op)
  File "/home/bart/.local/lib/python3.10/site-packages/cwltool/utils.py", line 211, in visit_class
    op(rec)
  File "/home/bart/.local/lib/python3.10/site-packages/cwltool/process.py", line 1348, in compute_checksums
    with fs_access.open(location, "rb") as f:
  File "/home/bart/.local/lib/python3.10/site-packages/cwltool/stdfsaccess.py", line 38, in open
    return open(self._abs(fn), mode)
FileNotFoundError: [Errno 2] No such file or directory: 'http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/bart/.local/lib/python3.10/site-packages/cwltool/main.py", line 1295, in main
    (out, status) = real_executor(
  File "/home/bart/.local/lib/python3.10/site-packages/cwltool/executors.py", line 61, in __call__
    return self.execute(process, job_order_object, runtime_context, logger)
  File "/home/bart/.local/lib/python3.10/site-packages/cwltool/executors.py", line 144, in execute
    self.run_jobs(process, job_order_object, logger, runtime_context)
  File "/home/bart/.local/lib/python3.10/site-packages/cwltool/executors.py", line 251, in run_jobs
    raise WorkflowException(str(err)) from err
cwltool.errors.WorkflowException: [Errno 2] No such file or directory: 'http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz'

Your Environment

  • cwltool version: 3.1.20230325110543
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Pass a PathMapper to the function that computes checksums kinow/cwltool
1 participant
@bartns