Control the number of cores used in production

[orelgueta]

On the grid it is important to know in advance the amount of cores used in each job. In particular we found it is significantly more efficient to submit jobs which use just one core (due to memory assignment).

In the current legacy framework there is a "sequential" mode which makes sure that CORSIKA and sim_telarray do not run simultaneously, but rather wait for each other to "pause". The same happens with multiple instances of sim_telarray.

The same feature or even more control should be included in simtools. Not sure if this feature is already requested (I think it does not exist). Apologies if I missed it.

[GernotMaier]

Where do we use multiple cores? The only place I know is in PR #1911 (RNDA with gradient descent) where ProcessPoolExecutor can we configured from the command line with n_workers. I talked already with @EshitaJoshi that the ProcessPoolExecutor should go into a generic function / module in simtools.job_execution to have a unified approach.

Do you have anything else in mind?

[orelgueta]

When running productions, CORSIKA and the multiple sim_telarray instances can run on separate cores. In the legacy tool, that is the default behaviour. In simtools I think we never implemented the sequential mode, forcing these processes to share a single core.

I admit I need to check further and also provide the example of how it is implemented in the current legacy tool. I will do that next week (if it is still necessary).

[GernotMaier]

This is not working?

simtools/src/simtools/runners/corsika_simtel_runner.py

Line 151 in b2da38e

f"bin/multipipe_corsika -c {multipipe_file} {self.sequential} "

[orelgueta]

It's disconcerting to see that I added this option myself (3bdd163) and that it wasn't even that long ago...

We should keep this functionality in mind when we replace the multipipe bash scripts in the future. This issue can anyway be closed I think.

[GernotMaier]

I would like to re-open it:

• make sure that 'sequential' is understood correctly and well documented
• ensure that the new features of Derive MRRA via gradient descent #1911 end up in a generic module with configurable workers.