Templates

[kkmann]

Hey,

just wondering about the rationale for skipping template files for e.g. the crew_controller_slurm. Imho, this makes it a bit harder to rationalize about the final job script being submitted and to implement more advanced features like containerization.

For instance, how would I ensure that all my jobs are executed in a specific singularity container using the current API?

Is there any chance of getting templates back (optionally)?

[wlandau]

I find template files inconvenient because of how it takes me out of R, and I think most R users find template files confusing. Not only are cluster job scripts written in a different language, but also the template text replacement mechanism is cryptic, and it is hard for new users to find clear answers to how it works.

So each time a crew.cluster launcher launches a worker, it generates the job script on demand using pedantically-documented user settings in R. The formal arguments don't cover everything, just the most common settings like memory and cores, but I think this gets most people most of the way there.

You can add arbitrary lines to the job script with the script_lines argument, and you can control logging with slurm_log_output and slurm_log_error. Because script_lines is arbitrarily flexible, I think you will probably be able to configure containerization that way.