bwa mem hangs after a few thousand reads

[chklee30]

I am trying to align a bunch of paired sample fastq files using bwa mem.

My original command was:

bwa mem -t 8 hg38.fa sample_read1.fq.gz sample_read2.fq.gz > sample_paired.sam

I am running this on a HPC cluster.

These files have approx. 25 million reads, so I initially anticipated that they might take a little bit of time; however I noticed that the programs seemed to hang - the sam file that I am redirecting the output to never became populated with any entries other than the header comments, even after hours of waiting.

Noticing this, I decided to first test whether the most bare bones bwa mem functionality was working, and so I took just the first 1000 lines off one of the fastq files and tested whether it was working with:

bwa mem hg38.fa test_1000lines.fq.gz > test_output.sam

So this works. This creates a proper sam file almost instantaneously. However, I kept testing this behavior by increasing the number of lines I took off the fastq file from 1000 to 2000,3000,etc., and eventually realized that any fastq files I input with more than approximately 8400-8500 lines will make the program hang.

If I run the following with a fastq file containing 8400 lines, the program will run and the sam file will be created in a few seconds:

bwa mem hg38.fa test_8400lines.fq.gz > test_output8400.sam

and this is the stdout output:

[M::bwa_idx_load_from_disk] read 0 ALT contigs

[M::process] read 2100 sequences (313733 bp)...

[M::mem_process_seqs] Processed 2100 reads in 1.051 CPU sec, 1.054 real sec

[main] Version: 0.7.17-r1188

[main] CMD: bwa mem supporting_files/hg38.fa test_8-4k.fq.gz

[main] Real time: 4.231 sec; CPU: 4.178 sec

However, if I run the following with just 100 more lines in the fastq file:

bwa mem hg38.fa test_8500lines.fq.gz > test_output8500.sam

The command will hang (at least, that is what it appears like its doing) and all I will see on the terminal is:

[M::bwa_idx_load_from_disk] read 0 ALT contigs

[M::process] read 2125 sequences (317468 bp)...

followed by a blinking cursor.

I've monitored the CPU and MEM usage with the "top" command while I was conducting these tests, and while the memory never goes above something like 15%, the CPU usage always goes to 100% when this hanging behavior happens. When the .sam files were created properly and the program didn't hang, the CPU usage was (for the very short amount of time it took for the program to run) like 60-70%.

Whether the CPU usage going to 100% is causing the hanging, or the hanging it causing the CPU usage to 100%, I'm not sure.

I'm completely out of ideas what the issue is here. Any ideas?