slurm on AWS Parallel cluster

[kkmann]

Hi,

just tried crew_controller_slurm() on an AWS parallel cluster, but the controller does not seem to be able to push() correctly

It hangs at

controller <- crew_controller_slurm()
controller$start()
controller$push(name = "task", command = sqrt(4))
controller$wait()

and I cannot see any activity using squeue. Local execution wotks fine, so does clustermq.

I wonder whether this could be related to the networking host/port arguemnts and how to diagnose this. Any suggestions? Is there maybe a debug mode with more detailed logging?

[wlandau]

This is a great use case to test, so thanks!

From the discussion at #1, you might have to supply host = Sys.info()["nodename"] instead of the default getip::getip(type = "local"). For logging, you could set slurm_log_output and/or slurm_log_error to something other than the default /dev/null.

[wlandau]

Also, you may have to call push() and or pop() on the controller a few times to get workers to start. This is because auto-scaling uses throttling for efficiency and because auto-scaling happens on the client (user's R session) instead of the mirai dispatcher. Charlie Gao and I have an ongoing discussion to see whether/how crew can inject code into the mirai dispatcher process so auto-scaling can become fully asynchronous.