Spread the load acros two mirrored remote SLURM systems

[stemangiola]

Prework

• [ x] Read and agree to the Contributor Code of Conduct and contributing guidelines.
• [x ] If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
• [x ] New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a GitHub discussion.
• [ x] Format your code according to the tidyverse style guide.

Proposal

I am not sure if this is already possible, but let's say I have two remote SLURM clusters that have mirrored file systems (same directories and file locations). Would it be possible to spread the pipeline load across those two URLs?

Thanks a lot.

[wlandau]

You could set up a controller for each cluster, then combine them in a controller group: https://wlandau.github.io/crew/articles/groups.html. You will still need to manually decide which controller to send each task, but the interface is a bit more seamless. If you want to make a decision on the fly, you can use the saturated() method to check if a controller's workers are all busy: https://wlandau.github.io/crew/reference/crew_class_controller_group.html#method-crew_class_controller_group-saturated

[stemangiola]

Thanks @wlandau.

It would be great to generalise the concept of controller, so to scale pipelines, without having to engineer new targets for load splitting.

I read saturated() documentation but I am not sure if I really understood.

If you want to make a decision on the fly

It sounds like it does what I would need. But can you please make an example with this code?

melbourne = crew_controller_slurm(name = "melbourne", ..., host = "...")

sydney = crew_controller_slurm(name = "sydney", ..., host = "...")
		
list(
tar_target(
	out_target,
	in_target |> fx(),
	pattern = map(in_target),
	iteration = "group",
                       resources = tar_resources(
                         crew = tar_resources_crew(controller = crew_controller_group(melbourne, sydney)) # ???
                       ))
)

[wlandau]

Ah, so you are using crew with targets. In that case, the controller argument of tar_resources_crew() takes a character string with the name of the controller, e.g. tar_resources_crew(controller = "melbourne"). crew operates at a high level and relies on mirai for scheduling, so it is not feasible to dynamically distribute among controllers in a group based on load. You would need to manually choose which controller each target is assigned to.

[wlandau]

Converting this issue to a discussion.