[BUG]: fcs-gx Error: Process failed with retcode -13 and 1

[tyler-peirce]

fcs-gx does not run regularly giving process failed with retcode -13. This error will be given for some samples in a batch and then run on those that errored when submitted again. It may not work on anything for a whole day of testing and then work on everything the next day. eg. on Friday I could not get it working with a script that has worked before on multiple samples, tried a lot of things and nothing worked, then 2 samples in a batch worked. Today I re run it and everything worked. But then this afternoon it is not working again.

To Reproduce
Copy the full run_fcsadaptor.sh/run_gx.py command used.

python3 /app/bin/run_gx \
--fasta ${assembly} \
--tax-id $taxid \
--out-dir ./NCBI \
--gx-db /tmp/gxdb \
--debug

If you are having trouble with your genome, please ensure that you can run the pipeline with one of our test genomes first. If your installation works fine with the sample input, please tell us if you are willing and able to share your genome with us, if asked.

Software versions (please complete the following information):

• OS [e.g. CentOS 7, macOS , etc.].
• Cloud Platform VM [e.g. AWS , GCP], if applicable.
• Singularity version [`singularity-ce version 4.1.0].
• Singularity FCS image version [https://ftp.ncbi.nlm.nih.gov/genomes/TOOLS/FCS/releases/0.5.4/fcs-gx.sif].
• Python version if using FCS-GX [`Python 3.11.6].
• Nextflow/24.04.3

Log Files
log_1.txt
Warning: missing header '##[["GX hits",2,1]]'
####### Cleaning up process ['cat', 'OG2.ilmn.240716.v129mh.fasta']
Error: Process failed with retcode -13: ['cat', 'OG2.ilmn.240716.v129mh.fasta'])
####### Cleaning up process ['tee', '/dev/fd/6']
Error: Process failed with retcode -13: ['tee', '/dev/fd/6'])
####### Cleaning up process ['/app/bin/gx', 'split-fasta']
Error: Process failed with retcode -13: ['/app/bin/gx', 'split-fasta'])
####### Cleaning up process ['pv', '-Wbratpe', '--interval=0.5', '--size=4979093308', '--buffer-size=104857600']
####### Cleaning up process ['/busybox/time', '-v', 'nice', '-n19', '/app/bin/gx', 'align', '--gx-db=/tmp/gxdb/all.gxi', '--repeats-basis-fa=/dev/fd/3']
Error: Process failed with retcode 1: ['/busybox/time', '-v', 'nice', '-n19', '/app/bin/gx', 'align', '--gx-db=/tmp/gxdb/all.gxi', '--repeats-basis-fa=/dev/fd/3'])
####### Cleaning up process ['pv', '--quiet', '--buffer-size=104857600']
####### Cleaning up process ['/app/bin/gx', 'taxify', '--gx-db=/tmp/gxdb/all.gxi', '--output=./NCBI/OG2.ilmn.240716.v129mh.396785.taxonomy.rpt.tmp', '--asserted-div=anml:fishes', '--db-exclude-locs=/app/bin/db_exclude.locs.tsv']
####### Cleaning up process ['/app/bin/gx', 'get-fasta-stats']
####### Cleaning up process ['cat', 'OG2.ilmn.240716.v129mh.fasta']

Additional context
This is being run as part of a nextflow.
The fasta file i am using is zipped (I am using zipped because last time i was getting errors i zipped it and it worked, however it has not been working this time, zipped or unzipped)
This is also the same issue that Lauren Huet (my colleague) was reporting on behalf of both of us in this closed thread #80

[etvedte]

Thank you for the report.

We issued a potential fix for issue #80 and your colleague reported that GX was working again, but the problems appeared to be stemming from a python versioning issue which does not appear to be an issue from your log file. So it seems likely you were getting GX to work again transiently, but the fix didn't fully resolve the issue.

The main developer who had been looking this issue is on vacation, so we can dig into this again at the beginning of December. This issue is apparently OS or HPC dependent because we haven't been able to reproduce it on any of the compute environments we have available.

A couple of follow-up questions:

1. You're continuing runs on the Pawsey supercomputer using SLURM scheduler, correct?
2. What OS are you using?

[tyler-peirce]

Hi,

Yes we are continuing to use Pawsey supercomputer, Setonix, using SLURM scheduler.
The operating system is Cray the details are found here.

Here are a few more error reports from trying to run fcs-gx yesterday

error3.txt
error2.txt
error1.txt