Crew and batch system walltimes

[multimeric]

I've had a read through https://books.ropensci.org/targets/crew.html#configuration-and-auto-scaling, which seems to be the best explanation out there of crew's auto scaling. However I'm still a bit confused how this relates to something like slurm_time_minutes, which sets the wall time for the slurm job.

My expectation is that, if only slurm_time_minutes is set, then Slurm will just kill a job but crew won't be aware that this has happened. Does crew have a heartbeat mechanism to track workers dying? Will crew know to spin up a new worker when one hits its walltime? Finally, should we also set seconds_wall so that crew can try to gently kill the jobs before Slurm does?

[wlandau]

My expectation is that, if only slurm_time_minutes is set, then Slurm will just kill a job but crew won't be aware that this has happened.

Slurm will terminate the job, and crew will become aware automatically because of how its backend mirai works. mirai uses NNG to communicate with workers and detect disconnected ones. It's shockingly good, much better than heartbeating could ever be. I am still only beginning to learn how NNG works.

[wlandau]

Will crew know to spin up a new worker when one hits its walltime?

Yes, crew automatically makes decisions about (re-)launching workers based on the number of tasks that remain.

Finally, should we also set seconds_wall so that crew can try to gently kill the jobs before Slurm does?

seconds_wall or more of a soft wall time, and it behaves differently because it cannot interrupt a task (@shikokuchuo, please correct me if I am wrong about the walltime argument in mirai). If you want a guaranteed hard wall time that does not depend on the task length, then Slurm is the better choice. (Although I think most pipelines would want the reverse so tasks complete.)

[shikokuchuo]

Yes, walltime was implemented as a 'soft' walltime as that was the original ask.