importlib in the requirement.txt

[gq1]

Hi there. If I do make, I got the following error message. It seem importlib is included in the Python env.

Do you need it the requirement again?

Thanks.

Collecting importlib
  Using cached importlib-1.0.4.zip (7.1 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.

   ModuleNotFoundError: No module named 'importlib.abc'

[gq1]

Hi there.

Managed to install everything needed. When I run the test.
Do you know which line throws this error? I can modify it.

Any possible to publish your singularity image somewhere, then I can give a try?

Thanks.

./toga2.py test
Initializing TOGA 2
Creating working directory
[2025-10-22 15:48:13][toga_main.py] - WARNING: Output directory does exist; cleaning up the conflicting files from the previous runs
[2025-10-22 15:48:13][toga_main.py] - INFO: Working directory created
[2025-10-22 15:48:13][toga_main.py] - INFO: Writing project metadata

[2025-10-22 15:48:14][toga_main.py] - INFO: Chain indexing complete
[2025-10-22 15:48:14][toga_main.py] - INFO: Retrieving contig sizes for the reference genome
[2025-10-22 15:48:14][toga_main.py] - CRITICAL: Genome compression check failed:
/bin/sh: 1: set: Illegal option -o pipefail

[ReverendCasy]

Hi @gq1 ,
Thank you for opening the issue. The list of Python requirements is to be revised soon, but glad you managed to install it anyway.

The second problem concerns the Shell version you’re using, since ‘pipefail’ is bash’ism. I guess the Makefile has a directive for Shell check, otherwise I’ll have to have a look. Anyway, switching from sh to bash must fix the issue.

Best,
Yury

[gq1]

Hi Yury.

I do use bash. However, there are several /bin/sh being hardcoded in code.
https://github.com/search?q=repo%3Ahillerlab%2FTOGA2%20%2Fbin%2Fsh&type=code

Just not sure which one caused the error.

[gq1]

And also for nextflow, can I re-use the config files I created for TOGA 1.

Here are some modifications when I set up version 1.
hillerlab/TOGA@master...sanger-tol:TOGA:master

[ReverendCasy]

@gq1

I do use bash. However, there are several /bin/sh being hardcoded in code.
https://github.com/search?q=repo%3Ahillerlab%2FTOGA2%20%2Fbin%2Fsh&type=code

Just not sure which one caused the error.

Bash is now set as a default shell for all subprocesses. Please update src/python/modules/shared.py

And also for nextflow, can I re-use the config files I created for TOGA 1.

Here are some modifications when I set up version 1.
hillerlab/TOGA@master...sanger-tol:TOGA:master

Nextflow configs/execution scripts are not directly interchangeable between T1 and T2, but otherwise TOGA2 accepts user-defined configs. I will upload example configs to the repo ASAP; meanwhile, please check the toga2.py run help message for minimal Nextflow configuration tips.

Any possible to publish your singularity image somewhere, then I can give a try?

Apptainer definition file can be found at supply/apptainer.def. Please check the README message for building recipe, usage, and potential caveats.

[gq1]

Thanks for updating the bash in the script. The test went further, probably nextflow not being configured properly. We use LSF.

Here are some logs from the test:

2025-10-22 19:45:21][toga_main.py] - INFO: Classifying projections
INFO:numexpr.utils:Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
INFO:numexpr.utils:NumExpr defaulting to 16 threads.
[2025-10-22 19:45:23][shared.py] - INFO: Initializing ChainClassifier
[2025-10-22 19:45:23][shared.py] - INFO: Uploading feature dataset
[2025-10-22 19:45:23][shared.py] - INFO: Uploading classification models
[2025-10-22 19:45:23][toga_main.py] - INFO: No items were rejected at classification step
[2025-10-22 19:45:23][toga_main.py] - INFO: Projection successfully classified
[2025-10-22 19:45:23][toga_main.py] - INFO: Sanity check passed: 0 reference transcripts (0.0%) do not have any predicted orthologous projections
[2025-10-22 19:45:23][toga_main.py] - INFO: Recovering fragmented projections
[2025-10-22 19:45:23][toga_main.py] - INFO: Fragmented projections successfully recovered
[2025-10-22 19:45:23][toga_main.py] - INFO: Scheduling preprocessing jobs
[2025-10-22 19:45:23][shared.py] - INFO: Uploading transcript data
[2025-10-22 19:45:23][shared.py] - INFO: Transcript span data for all reference transcripts have been successfully uploaded
[2025-10-22 19:45:23][toga_main.py] - INFO: Preprocessing jobs scheduled
[2025-10-22 19:45:23][toga_main.py] - INFO: Running preprocessing jobs (parallel step)
[2025-10-22 19:45:23][parallel_jobs_manager.py] - WARNING: Parallel manager: pushing job nextflow *************************/src/TOGA2/sample_output/nextflow/execute_joblist.nf --joblist *************************/src/TOGA2/sample_output/tmp/cesar_preprocessing_jobs/joblist_preprocess
[2025-10-22 19:45:23][toga_main.py] - WARNING: Some of the monitored parallel processes at preprocessing step died
[2025-10-22 19:45:23][toga_main.py] - INFO: Preprocessing jobs finished
[2025-10-22 19:45:23][toga_main.py] - INFO: Aggregating preprocessing results
[2025-10-22 19:45:23][toga_main.py] - CRITICAL: Some of the monitored parallel processes at preprocessing step died

I did try to build the container myself using your recipe but need root permission.

[ReverendCasy]

@gq1
Sorry for the delay. Could you please check your nextflow/cesar_preprocess.../cesar_preprocess....log and logs/failed_batches_....tsv files as described here?

I did try to build the container myself using your recipe but need root permission.

If using root permission is not expected on your machine for container building, could you please provide the command used and the Apptainer error message?

Best,
Yury

[gq1]

Hi Yury.

I have a close look and the nextflow parts is actually fine.

Only one job in the joblist and being tried 3 times and all failed with exit code 1. All the log and error files for nextflow steps are empty. Nothing is helpful in main Nextflow log either.

Oct-24 12:00:45.621 [main] DEBUG nextflow.cli.Launcher - $> nextflow /software/treeoflife/src/TOGA2/sample_output/nextflow/execute_joblist.nf --joblist /software/treeoflife/src/TOGA2/sample_output/tmp/cesar_preprocessing_jobs/joblist_preprocess
Oct-24 12:00:45.802 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W  ~  version 25.10.0

Oct-24 12:00:50.468 [Task submitter] INFO  nextflow.Session - [19/1da6ff] Re-submitted process > execute_jobs (1)
Oct-24 12:00:51.129 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 4; name: execute_jobs (1); status: COMPLETED; exit: 1; error: -; workDir: /software/treeoflife/src/TOGA2/sample_output/nextflow/work/19/1da6ffcf46f848927f88198cd4e973]
Oct-24 12:00:51.132 [TaskFinalizer-4] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=execute_jobs (1); work-dir=/software/treeoflife/src/TOGA2/sample_output/nextflow/work/19/1da6ffcf46f848927f88198cd4e973
  error [nextflow.exception.ProcessFailedException]: Process `execute_jobs (1)` terminated with an error exit status (1)
Oct-24 12:00:51.151 [TaskFinalizer-4] ERROR nextflow.processor.TaskProcessor - Error executing process > 'execute_jobs (1)'

Caused by:
  Process `execute_jobs (1)` terminated with an error exit status (1)


Command executed:

  /software/treeoflife/src/TOGA2/sample_output/tmp/cesar_preprocessing_jobs/batch0.ex

Command exit status:
  1

If I run the actual command TOGA generated out of nextflow and toga, it returned immediately with exit code 1 and without any other error information. Here is the actual command:

$ /software/treeoflife/src/TOGA2/src/python/cesar_preprocess.py "ENST00000618101,ENST00000259400,ENST00000262455" 1169 /software/treeoflife/src/TOGA2/sample_output/tmp/input_data/reference_annotation_cds.bed /software/treeoflife/src/TOGA2/test_input/hg38.micro_sample.2bit /software/treeoflife/src/TOGA2/test_input/q2bit_micro_sample.2bit /software/treeoflife/src/TOGA2/sample_output/tmp/input_data/genome_alignment.bst /software/treeoflife/src/TOGA2/sample_output/tmp/input_data/reference_contig_sizes.tsv /software/treeoflife/src/TOGA2/sample_output/tmp/input_data/query_contig_sizes.tsv  --cesar_canon_u2_acceptor /software/treeoflife/src/TOGA2/supply/CESAR2.0/profiles/human/canon_U2_acceptor.tsv --cesar_canon_u2_donor /software/treeoflife/src/TOGA2/supply/CESAR2.0/profiles/human/canon_U2_donor.tsv --cesar_non_canon_u2_acceptor /software/treeoflife/src/TOGA2/supply/CESAR2.0/profiles/human/nonCanon_U2_acceptor.tsv --cesar_non_canon_u2_donor /software/treeoflife/src/TOGA2/supply/CESAR2.0/profiles/human/nonCanon_U2_donor.tsv --cesar_canon_u12_acceptor /software/treeoflife/src/TOGA2/supply/CESAR2.0/profiles/human/canon_U12_acceptor.tsv --cesar_canon_u12_donor /software/treeoflife/src/TOGA2/supply/CESAR2.0/profiles/human/canon_U12_donor.tsv --cesar_non_canon_u12_acceptor /software/treeoflife/src/TOGA2/supply/CESAR2.0/profiles/human/canon_U12_acceptor.tsv --cesar_non_canon_u12_donor /software/treeoflife/src/TOGA2/supply/CESAR2.0/profiles/human/non_canon_U12_acceptor.tsv --cesar_first_acceptor /software/treeoflife/src/TOGA2/supply/CESAR2.0/profiles/firstCodon_profile.tsv --cesar_last_donor /software/treeoflife/src/TOGA2/supply/CESAR2.0/profiles/lastCodon_profile.tsv --assembly_gap_size 10 --minimal_covered_fraction 0.0 --max_space_size 1000000 --extrapolation_modifier 1.2 --exon_locus_flank 100 --twobit2fa_binary /software/treeoflife/src/TOGA2/bin/twoBitToFa --bigwig2wig_binary /software/treeoflife/src/TOGA2/bin/bigWigToWig --memory_limit 15.0 --disable_spanning_chains --output /software/treeoflife/src/TOGA2/sample_output/tmp/cesar_preprocessing_res/batch0

$ echo $?
1

[ReverendCasy]

Hi @gq1 ,
Could you please try running the cesar_preprocess.py command above with -v/--verbose flag added and/or attach the batch execution log (tmp/cesar_preprocessing_res/batch0/log.txt) if one exists?

Best,
Yury

[gq1]

Thanks, that is helpful.

[shared.py] - CRITICAL: ERROR: bigWigToBed run failed with the following error:
bash: line 1: twoBitToFa: command not found

Once I added ./bin to the path, it went further.

[shared.py] - CRITICAL: ERROR: PRANK executable is missing from both PATH and user input

All went fine after prank being installed.

This is conda env file I ended up with:

name: toga2
channels:
  - conda-forge
  - bioconda
dependencies:
  - rust=1.90.0
  - python=3.13
  - nextflow=25.10.0
  - prank=170427

I also need to make sure both TOGA2/ and TOGA2/bin in my path.

Please let me know if you have any updates.

Thank you very much!

[ReverendCasy]

Hi @gq1 ,
Thank you for the update, and my apologies for you encountering the binary path bug. The default paths to external binaries have been added to the code recently, with the fix to be pushed soon.

Just to clarify:

1. Do you have prank downloaded and built at TOGA2/bin/prank? If so, we recommend using the version downloaded at TOGA installation (v.250331).
2. With the fixed paths, did you manage to successfully run the test command?

Thank you for your interest in TOGA2, and thank you for the suggested conda environment. We are currently working on improving TOGA2 distribution, so any feedback on user experiencing with installing and running the tool is very valuable to us.

Best,
Yury

[gq1]

Do you have prank downloaded and built at TOGA2/bin/prank?

No, I don't.

If so, we recommend using the version downloaded at TOGA installation (v.250331).

I just install the latest version using conda. Not sure why prank was not installed by your make and how to install the version you mentioned. https://github.com/ariloytynoja/prank-msa

With the fixed paths, did you manage to successfully run the test command?

Yes, the whole test done.

[gq1]

Hi Yury again.

I pulled your latest changes and re-built and got the following error:

make

.....

    Finished `release` profile [optimized] target(s) in 0.93s
echo "=============================="
==============================
cd bed2gtf && cargo build --release
error: could not find `Cargo.toml` in `/******/src/TOGA2/bed2gtf` or any parent directory
make: *** [Makefile:41: build_rust] Error 101

Can you have a look? Thanks.

UPDATE:
git submodules are missing in my checkout.

git submodule update --init --recursive

[ReverendCasy]

Hello @gq1
I updated the clone command in README, please try it. Installation and testing seem to work fine.

Regarding your previous question:

And also for nextflow, can I re-use the config files I created for TOGA 1.
Here are some modifications when I set up version 1.
hillerlab/TOGA@master...sanger-tol:TOGA:master

- I added sample config directory with a short README to the repo. Hope this helps.