test error with singularity: Error executing process > 'NFCORE_CUTANDRUN:CUTANDRUN:CUSTOM_DUMPSOFTWAREVERSIONS (1)' #243

mglgc · 2024-05-16T09:36:08Z

Description of the bug

Hi,
Runnig just the profile test with singularity (nextflow run nf-core/cutandrun -profile test,singularity), the job seems to start with no issues but ends with the following error:

Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_CUTANDRUN:CUTANDRUN:CUSTOM_DUMPSOFTWAREVERSIONS (1)'

Caused by:
Process NFCORE_CUTANDRUN:CUTANDRUN:CUSTOM_DUMPSOFTWAREVERSIONS (1) terminated with an error exit status (1)

Command used and terminal output

Command: nextflow run nf-core/cutandrun -profile test,singularity

Output:
Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_CUTANDRUN:CUTANDRUN:CUSTOM_DUMPSOFTWAREVERSIONS (1)'

Caused by:
  Process `NFCORE_CUTANDRUN:CUTANDRUN:CUSTOM_DUMPSOFTWAREVERSIONS (1)` terminated with an error exit status (1)

Command executed:

  #!/usr/bin/env python
  
  import yaml
  import platform
  from textwrap import dedent
  
  def _make_versions_html(versions):
      html = [
          dedent(
              '''\
              <style>
              #nf-core-versions tbody:nth-child(even) {
                  background-color: #f2f2f2;
              }
              </style>
              <table class="table" style="width:100%" id="nf-core-versions">
                  <thead>
                      <tr>
                          <th> Process Name </th>
                          <th> Software </th>
                          <th> Version  </th>
                      </tr>
                  </thead>
              '''
          )
      ]
      for process, tmp_versions in sorted(versions.items()):
          html.append("<tbody>")
          for i, (tool, version) in enumerate(sorted(tmp_versions.items())):
              html.append(
                  dedent(
                      f'''\
                      <tr>
                          <td><samp>{process if (i == 0) else ''}</samp></td>
                          <td><samp>{tool}</samp></td>
                          <td><samp>{version}</samp></td>
                      </tr>
                      '''
                  )
              )
          html.append("</tbody>")
      html.append("</table>")
      return "\n".join(html)
  
  def _make_versions_unique_html(versions):
      unique_versions = []
  
      for process, tmp_versions in sorted(versions.items()):
          for i, (tool, version) in enumerate(sorted(tmp_versions.items())):
              tool_version = tool + "=" + version
              if tool_version not in unique_versions:
                  unique_versions.append(tool_version)
  
      unique_versions.sort()
  
      html = [
          dedent(
              '''\
              <style>
              #nf-core-versions-unique tbody:nth-child(even) {
                  background-color: #f2f2f2;
              }
              </style>
              <table class="table" style="width:100%" id="nf-core-versions-unique">
                  <thead>
                      <tr>
                          <th> Software </th>
                          <th> Version  </th>
                      </tr>
                  </thead>
              '''
          )
      ]
  
      for tool_version in unique_versions:
          tool_version_split = tool_version.split('=')
          html.append("<tbody>")
          html.append(
              dedent(
                  f'''\
                  <tr>
                      <td><samp>{tool_version_split[0]}</samp></td>
                      <td><samp>{tool_version_split[1]}</samp></td>
                  </tr>
                  '''
              )
          )
          html.append("</tbody>")
      html.append("</table>")
      return "\n".join(html)
  
  module_versions = {}
  module_versions["NFCORE_CUTANDRUN:CUTANDRUN:CUSTOM_DUMPSOFTWAREVERSIONS"] = {
      'python': platform.python_version(),
      'yaml': yaml.__version__
  }
  
  with open("versions.yml") as f:
      workflow_versions = yaml.load(f, Loader=yaml.BaseLoader) | module_versions
  
  workflow_versions["Workflow"] = {
      "Nextflow": "23.04.2",
      "nf-core/cutandrun": "3.2.2"
  }
  
  versions_mqc = {
      'parent_id': 'software_versions',
      'parent_name': 'Software Versions',
      'parent_description': 'Details software versions used in the pipeline run',
      'id': 'software-versions-by-process',
      'section_name': 'nf-core/cutandrun software versions by process',
      'section_href': 'https://github.com/nf-core/cutandrun',
      'plot_type': 'html',
      'description': 'are collected at run time from the software output.',
      'data': _make_versions_html(workflow_versions)
  }
  
  versions_mqc_unique = {
      'parent_id': 'software_versions',
      'parent_name': 'Software Versions',
      'parent_description': 'Details software versions used in the pipeline run',
      'id': 'software-versions-unique',
      'section_name': 'nf-core/cutandrun Software Versions',
      'section_href': 'https://github.com/nf-core/cutandrun',
      'plot_type': 'html',
      'description': 'are collected at run time from the software output.',
      'data': _make_versions_unique_html(workflow_versions)
  }
  
  with open("software_versions.yml", 'w') as f:
      yaml.dump(workflow_versions, f, default_flow_style=False)
  
  with open("software_versions_mqc.yml", 'w') as f:
      yaml.dump(versions_mqc, f, default_flow_style=False)
  
  with open("software_versions_unique_mqc.yml", 'w') as f:
      yaml.dump(versions_mqc_unique, f, default_flow_style=False)
  
  with open('local_versions.yml', 'w') as f:
      yaml.dump(module_versions, f, default_flow_style=False)

Command exit status:
  1

Command output:
  (empty)

Command error:
  INFO:    Converting SIF file to temporary sandbox...
  Traceback (most recent call last):
    File ".command.sh", line 99, in <module>
      workflow_versions = yaml.load(f, Loader=yaml.BaseLoader) | module_versions
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/__init__.py", line 114, in load
      return loader.get_single_data()
             ^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/constructor.py", line 49, in get_single_data
      node = self.get_single_node()
             ^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 36, in get_single_node
      document = self.compose_document()
                 ^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 55, in compose_document
      node = self.compose_node(None, None)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node
      node = self.compose_mapping_node(anchor)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 133, in compose_mapping_node
      item_value = self.compose_node(node, item_key)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node
      node = self.compose_mapping_node(anchor)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 127, in compose_mapping_node
      while not self.check_event(MappingEndEvent):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/parser.py", line 98, in check_event
      self.current_event = self.state()
                           ^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/parser.py", line 438, in parse_block_mapping_key
      raise ParserError("while parsing a block mapping", self.marks[-1],
  yaml.parser.ParserError: while parsing a block mapping
    in "versions.yml", line 57, column 5
  expected <block end>, but found '['
    in "versions.yml", line 57, column 21
  INFO:    Cleaning up image...

Work dir:
  /mnt/lustre/scratch/nlsas/home/csic/sdb/mgl/kk/work/d5/c184de6e2d06f866cb452dcbee5a60

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details

Relevant files

No response

System information

No response

The text was updated successfully, but these errors were encountered:

mglgc · 2024-05-16T12:55:21Z

Furthering, the running was launched with 4 cores and 128GB RAM.

chris-cheshire · 2024-05-22T10:37:27Z

Is this reproducable? The pipeline is tested against singularity regulary in test mode

mglgc · 2024-05-22T11:58:23Z

Yes, it is. Additionally, under the docker profile the test works OK.

chris-cheshire · 2024-05-22T16:06:40Z

what version of singularity are you using?

mglgc · 2024-05-22T16:30:29Z

what version of singularity are you using?

4.0.0.
Thank you for your replies.

chris-cheshire · 2024-05-30T07:15:24Z

can you try using 3.6.4?

mglgc · 2024-06-24T09:43:56Z

Sorry for the delayed reply but I was abroad for some weeks.
I've just try using singularity 3.6.3 (I don't have the 3.6.4 version) with a successful result.
Please, update accordingly the specifications info for the singularity profile. Thank you very much for your support.

JWayne3 · 2024-07-01T07:05:34Z

Hi! I got a similar issue:
ERROR ~ Error executing process > 'NFCORE_CUTANDRUN:CUTANDRUN:CUSTOM_DUMPSOFTWAREVERSIONS (1)'

Caused by:
Process NFCORE_CUTANDRUN:CUTANDRUN:CUSTOM_DUMPSOFTWAREVERSIONS (1) terminated with an error exit status (1)

Command executed:

#!/usr/bin/env python

import yaml
import platform
from textwrap import dedent

def _make_versions_html(versions):
html = [
dedent(
'''
<style>
#nf-core-versions tbody:nth-child(even) {
background-color: #f2f2f2;
}
</style>

'''
)
]
for process, tmp_versions in sorted(versions.items()):
html.append("")
for i, (tool, version) in enumerate(sorted(tmp_versions.items())):
html.append(
dedent(
f'''

'''
)
)
html.append("")
html.append("

Process Name	Software	Version
`{process if (i == 0) else ''}`	`{tool}`	`{version}`

")
return "\n".join(html)

def _make_versions_unique_html(versions):
unique_versions = []

  for process, tmp_versions in sorted(versions.items()):
      for i, (tool, version) in enumerate(sorted(tmp_versions.items())):
          tool_version = tool + "=" + version
          if tool_version not in unique_versions:
              unique_versions.append(tool_version)

  unique_versions.sort()

  html = [
      dedent(
          '''\
          <style>
          #nf-core-versions-unique tbody:nth-child(even) {
              background-color: #f2f2f2;
          }
          </style>
          <table class="table" style="width:100%" id="nf-core-versions-unique">
              <thead>
                  <tr>
                      <th> Software </th>
                      <th> Version  </th>
                  </tr>
              </thead>
          '''
      )
  ]

  for tool_version in unique_versions:
      tool_version_split = tool_version.split('=')
      html.append("<tbody>")
      html.append(
          dedent(
              f'''\
              <tr>
                  <td><samp>{tool_version_split[0]}</samp></td>
                  <td><samp>{tool_version_split[1]}</samp></td>
              </tr>
              '''
          )
      )
      html.append("</tbody>")
  html.append("</table>")
  return "\n".join(html)

module_versions = {}
module_versions["NFCORE_CUTANDRUN:CUTANDRUN:CUSTOM_DUMPSOFTWAREVERSIONS"] = {
'python': platform.python_version(),
'yaml': yaml.version
}

with open("versions.yml") as f:
workflow_versions = yaml.load(f, Loader=yaml.BaseLoader) | module_versions

workflow_versions["Workflow"] = {
"Nextflow": "24.04.2",
"nf-core/cutandrun": "3.2.2"
}

versions_mqc = {
'parent_id': 'software_versions',
'parent_name': 'Software Versions',
'parent_description': 'Details software versions used in the pipeline run',
'id': 'software-versions-by-process',
'section_name': 'nf-core/cutandrun software versions by process',
'section_href': 'https://github.com/nf-core/cutandrun',
'plot_type': 'html',
'description': 'are collected at run time from the software output.',
'data': _make_versions_html(workflow_versions)
}

versions_mqc_unique = {
'parent_id': 'software_versions',
'parent_name': 'Software Versions',
'parent_description': 'Details software versions used in the pipeline run',
'id': 'software-versions-unique',
'section_name': 'nf-core/cutandrun Software Versions',
'section_href': 'https://github.com/nf-core/cutandrun',
'plot_type': 'html',
'description': 'are collected at run time from the software output.',
'data': _make_versions_unique_html(workflow_versions)
}

with open("software_versions.yml", 'w') as f:
yaml.dump(workflow_versions, f, default_flow_style=False)

with open("software_versions_mqc.yml", 'w') as f:
yaml.dump(versions_mqc, f, default_flow_style=False)

with open("software_versions_unique_mqc.yml", 'w') as f:
yaml.dump(versions_mqc_unique, f, default_flow_style=False)

with open('local_versions.yml', 'w') as f:
yaml.dump(module_versions, f, default_flow_style=False)

Command exit status:
1

Command output:
(empty)

Command error:
INFO: Convert SIF file to sandbox...
Traceback (most recent call last):
File ".command.sh", line 99, in
workflow_versions = yaml.load(f, Loader=yaml.BaseLoader) | module_versions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/init.py", line 114, in load
return loader.get_single_data()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/constructor.py", line 49, in get_single_data
node = self.get_single_node()
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 36, in get_single_node
document = self.compose_document()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 55, in compose_document
node = self.compose_node(None, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 133, in compose_mapping_node
item_value = self.compose_node(node, item_key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 127, in compose_mapping_node
while not self.check_event(MappingEndEvent):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/parser.py", line 98, in check_event
self.current_event = self.state()
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/parser.py", line 438, in parse_block_mapping_key
raise ParserError("while parsing a block mapping", self.marks[-1],
yaml.parser.ParserError: while parsing a block mapping
in "versions.yml", line 55, column 5
expected , but found '['
in "versions.yml", line 55, column 21
INFO: Cleaning up image...

and change the version of singularity didn't work for me ,and I have drawn on the discussion at https://github.com/nf-core/rnaseq/issues/1103 on GitHub, and upon checking my own versions.yml file, I also found that it contained log information that should not be there:

"NFCORE_CUTANDRUN:CUTANDRUN:FASTQC_TRIMGALORE:FASTQC":
fastqc: [0.002s][warning][os,container] Duplicate cpuset controllers detected. Picking /sys/fs/cgroup/cpuset, skipping /f/weiyongjian/anaconda3/envs/cutandrun/var/singularity/mnt/session/final/sys/fs/cgroup/cpuset.
0.12.1

I resolved my issue by modifying lines 40 to 52 in modules/nf-core/fastqc/main.nf from:

fastqc: $( fastqc --version | sed -e "s/FastQC v//g" )

to

fastqc: $( fastqc --version | tail -1 | sed -e "s/FastQC v//g" )

I hope this  can be helpful to others facing similar issues.

mglgc added the bug Something isn't working label May 16, 2024

mglgc closed this as completed Jun 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test error with singularity: Error executing process > 'NFCORE_CUTANDRUN:CUTANDRUN:CUSTOM_DUMPSOFTWAREVERSIONS (1)' #243

test error with singularity: Error executing process > 'NFCORE_CUTANDRUN:CUTANDRUN:CUSTOM_DUMPSOFTWAREVERSIONS (1)' #243

mglgc commented May 16, 2024

mglgc commented May 16, 2024

chris-cheshire commented May 22, 2024

mglgc commented May 22, 2024

chris-cheshire commented May 22, 2024

mglgc commented May 22, 2024

chris-cheshire commented May 30, 2024

mglgc commented Jun 24, 2024

JWayne3 commented Jul 1, 2024

test error with singularity: Error executing process > 'NFCORE_CUTANDRUN:CUTANDRUN:CUSTOM_DUMPSOFTWAREVERSIONS (1)' #243

test error with singularity: Error executing process > 'NFCORE_CUTANDRUN:CUTANDRUN:CUSTOM_DUMPSOFTWAREVERSIONS (1)' #243

Comments

mglgc commented May 16, 2024

Description of the bug

Command used and terminal output

Relevant files

System information

mglgc commented May 16, 2024

chris-cheshire commented May 22, 2024

mglgc commented May 22, 2024

chris-cheshire commented May 22, 2024

mglgc commented May 22, 2024

chris-cheshire commented May 30, 2024

mglgc commented Jun 24, 2024

JWayne3 commented Jul 1, 2024