Guidance on setting number of workers in crew_controller_local()

[jack-davison]

Hello!

I've been enjoying using {crew}, but one thing that has evaded me is the decision making needed behind setting a number of workers.

In other parallel processing packages I get that you define the number of cores you want to use, which is simple, but I can't see anything in the {crew} documentation that provides any guidance on picking a number of workers.

I imagine setting workers = 10000L is a bad idea in crew_controller_local(), but I don't exactly know why and what a "good number" would be instead!

Part of this is that I'm keen to use {crew} in a package for synchronous parallel processing, and I'd want to pass some rule of thumb onto its users.

Thanks very much

[wlandau]

Each local crew worker is an R process on your computer, just as with parallel::mclapply() or future::future() with future::plan(future::multisession). So whatever number for mc.cores or workers you would choose for other packages applies here too.

You will probably see most benefits from setting workers equal to the number of available cores on your machine. Anything more probably won't be much better, even with CPU hyperthreading. And of course, if you are running crew on a shared machine, e.g. the login node of a cluster, others will need cores too.

[jack-davison]

Thanks very much Will - very clear!