How does IQTREE decide on the number of Threads used with nt AUTO?

[veronicamb]

Hello all,

I'm working on a dataset with over 1000 taxa and 400 loci, and I'm having problems getting non-parametric bootstrap to work. I'm running bootstrap analyses as an array.

However, the amount of time that each replicate takes varies widely! I noticed that this seems to be directly correlated with the number of threads IQTREE chooses to use. For the most part, it chooses to use only 1 or 2 threads, despite having 60 threads available. Then these replicates take over 300 hours to complete, or worse, do not finish at all.
In some cases, however, IQTREE chooses to use 40 or 50 threads, and then those take about 20 hours to complete, which is much more reasonable.
I tried to force the number of threads used, but that worked for a few replicates only. For the most part, it just caused the analyses to stall.

I don't know what else to do! I'm using the PMSF model to run the bootstraps, as that is how the tree was inferred.

Any help will be immensely appreciated!

thanks!

Veronica

[roblanf]

Hi Veronica,

The way IQ-TREE chooses threads with nt AUTO is to run analyses at multiple different thread counts and choose the most efficient (roughly speaking...). One thing to note is that this can be quite slow, and perhaps particularly so with PMSF bootstraps.

There's a simpler way to approach this with bootstraps, which should be quicker overall. The idea is simple - run as many concurrent replicates as possible, and apportion threads appropriately. If you are doing B (e.g. 100) bootstraps and have K threads, just set -T to K/B rounded down to the nearest integer, with a minimum value (of course) of 1 (if B > K). This assumes you are not limited by RAM. If you are limited by RAM, then increase the number of threads to limit how many can run concurrently.

This works because with bootstraps we expect all of them to take roughly the same amount of time, and in that case the best threading option is to just run as many as possible concurrently.

Rob

P.S. I wasn't sure what you meant here: "I tried to force the number of threads used, but that worked for a few replicates only.", but that sound like a potential issue. For that, we'll need command lines and input files to help out.

[veronicamb]

Hi Rob,

Thanks for your reply!

Just to clarify what I meant by "forcing" the number of threads. What I did was simply to determine the number of threads instead of using AUTO. This seems to be basically what you suggested, correct?

I'll try following your suggestions and I'll report back on results. I'll be happy to share some scripts and log info if needed.

thanks again!

Cheers,

Veronica

[roblanf]

Yep. My suggestion is basically to set each job to use -T 1 and then to run as many concurrent jobs as you have threads. This is typically the most efficient way to do bootstraps. See how it goes! Interested to hear back.