failed to build index for overlap store

[JeanQ2020]

found this error in the log:

ABORT: Failed to open './canu_S0702411_soy17.ovlStore.BUILDING/0001-001' for reading: No such file or directory

However I did find ./canu_S0702411_soy17.ovlStore.BUILDING/'<0001-001>'

Look like that the names that it writes doesn't match the names that it expects to build index?

Any clue?

BTW, I'm using the latest version -- canu 2.3

Thanks,
Jean

[JeanQ2020]

The file name is '<0001-001>' but it looks for the file: 0001-001

[skoren]

I suspect your FS doesn't like the '<>' in the names. However, the filenames shouldn't be using <> in recent versions of canu, this was a compile-time option but has been removed for a while: 3f55aa6, since 2020.

Are you sure you're using v2.3 and that an older version isn't being picked up from the path? What does canu --version report? What is your exact full command? What is the full log of the run up to this point?

[JeanQ2020]

Thanks, skoren for your quick response and looking into my canu issue.

I was using version 2.0 initially but it failed at mhap step. I downloaded the version 2.3 and rerun the same command.

Here's the current version that I used

[qinx4@by2ad5] canu_out $ canu --version
canu 2.3

The command that I ran this sample:
canu -d assembly/canu_out/canu_S0702411_soy17 -p canu_S0702411_soy17 genomeSize=1g maxThreads=20 maxMemory=500G minThreads=10 useGrid="false" -nanopore reads/S0702411_soy17.fasta >& assembly/canu_out/canu_S0702411_soy17.log

Please find attachment for the full log

canu_S0702411_soy17.log

[JeanQ2020]

FYI, I downloaded canu binary. I can't compile the source codes as it needs some dependencies that requires root privilege to install it

[skoren]

That log starts with overlap store sorting, nothing from mhap/overlap bucketizing. I think your files w/the different names were created by version 2.0 and version 2.3 is not backwards compatible with output from v2.0, from the release notes: https://github.com/marbl/canu/releases/tag/v2.3
Canu v2.3 IS (expected to be) compatible with assemblies started with Canu v2.2, v2.1, and v2.1.1 but NOT with any earlier version. However, we DO NOT recommend mixing versions.
The safest thing to do would be to remove the folder completely and relaunch from scratch with v2.3.

However, the error is at the last cleanup step so likely your store is actually OK. Depending on how comfortable you are with running some steps manually, you could:

• create 0001-001 empty files (using touch) corresponding to any files <0001-001> so if there is a <0001-002> you'd also make a 0001-002 and so on in the store folder
• Rerun the failing command from inside the assembly/canu_out/canu_S0702411_soy17:

    /gpfs/rtp/data/prod/Projects/CropGenetics/code_base/canu-2.3/bin/ovStoreIndexer \
      -O  ./canu_S0702411_soy17.ovlStore.BUILDING \
      -S ../canu_S0702411_soy17.seqStore \
      -C  ./canu_S0702411_soy17.ovlStore.config \
      -delete

• if it finishes successfully, then you can do mv ./canu_S0702411_soy17.ovlStore.BUILDING ./canu_S0702411_soy17.ovlStore and continue the assembly with the same canu command you used before.

[JeanQ2020]

thank you so much for taking your time to examine the log and detailed instructions.

Just make sure if I understand you correctly. You suggested to touch 0001-001,... files? Would it work if I just rename '<0001-001>' to 0001-001?

[JeanQ2020]

FYI, another sample has been using canu 2.0 to run mhap but use canu3 to create files in the store but the files are still in the names of '<0001-001>'. I wonder if the early steps determines the file name in the store?

[skoren]

Yes moving would work but then these files will be removed by the store command. The touch I suggested will leave the original version just in case. I think the filename is determined during the configuration of the store before the files in it are generated.