SLURM job fails from controller running targets pipeline - Timed out

[colebrookson]

I'm trying to develop a working example/template for doing HPC using targets and crew.cluster to run targets pipelines on a SLURM cluster. I've been able to get it sort of working, but I'm running into a few problems I'm hoping I can ask for input on?

the code

First, here's the pipeline, that builds off of the simple example in the targets docs, but extends it to just use two different types of controllers in a way that I think is similar to how most people's pipelines might look:

library(targets)
library(here)
library(crew)
library(crew.cluster)

source(here("./src/R/00_functions.R"))

controller_small <- crew.cluster::crew_controller_slurm(
  name = "small_slurm",
  slurm_cpus_per_task = 1,
  workers = 1,
  slurm_time_minutes = 10,
  seconds_idle = 600,
  script_lines = c(
    "#SBATCH --mem-per-cpu=2G",
    "#SBATCH --mail-user=cole.brookson@gmail.com",
    "#SBATCH --mail-type=BEGIN",
    "#SBATCH --mail-type=END",
    "#SBATCH --mail-type=FAIL",
    "#SBATCH --mail-type=REQUEUE",
    "#SBATCH --nodes=1",
    "#SBATCH --ntasks-per-node=1",
    "module load StdEnv/2023 r/4.3.1"
  ),
  slurm_log_output = "/home/brookson/scratch/output.txt",
  slurm_log_error = "/home/brookson/scratch/error.txt",
)
controller_big <- crew.cluster::crew_controller_slurm(
  name = "bigger_slurm",
  workers = 4,
  slurm_cpus_per_task = 4,
  slurm_time_minutes = 10,
  seconds_idle = 600,
  script_lines = c(
    "#SBATCH --mem-per-cpu=2G",
    "#SBATCH --mail-user=cole.brookson@gmail.com",
    "#SBATCH --mail-type=BEGIN",
    "#SBATCH --mail-type=END",
    "#SBATCH --mail-type=FAIL",
    "#SBATCH --mail-type=REQUEUE",
    "#SBATCH --nodes=1",
    "#SBATCH --ntasks-per-node=4",
    "module load StdEnv/2023 r/4.3.1"
  ),
  slurm_log_output = "/home/brookson/scratch/big-output.txt",
  slurm_log_error = "/home/brookson/scratch/big-error.txt",
)
controller <- crew::crew_controller_group(controller_small,
                                           controller_big) 
tar_option_set(
  packages = c("readr", "dplyr", "ggplot2", "rstanarm", "qs"),
  controller = crew::crew_controller_group(controller_small,
                                           controller_big),
  resources = tar_resources(
    crew = tar_resources_crew(controller = "small_slurm")
  ),
  garbage_collection = TRUE,
  memory = "transient"
)

list(
  tar_target(data, get_data(here::here("./data/airquality.csv"))),
  tar_target(model, fit_model(data)),
  tar_target(plot, plot_model(model, data)),
  tar_target(big, big_model(data),
             resources = tar_resources(
               crew = tar_resources_crew(controller = "bigger_slurm",
                                         seconds_timeout = 60)
             )),
  tar_target(plot_big, big_plot(big))
)

The functions file is here:

get_data <- function(file) {
  read_csv(file, col_types = cols()) %>%
    filter(!is.na(Ozone))
}

fit_model <- function(data) {
  lm(Ozone ~ Temp, data) %>%
    coefficients()
}

plot_model <- function(model, data) {
  p <- ggplot(data) +
    geom_point(aes(x = Temp, y = Ozone)) +
    geom_abline(intercept = model[1], slope = model[2])
  ggplot2::ggsave(here::here("./figs/plot.png"), p)
}

big_model <- function(data) {
  rstanarm::stan_glmer(Ozone ~ Temp + Wind + Solar.R + (1|Month),
    data = data,
    iter = 10000,
    chains = 4,
    cores = 4,
    adapt_delta = 0.9999,
    control = list(max_treedepth = 50)
  )
}

big_plot <- function(big_model) {
  bayesplot::color_scheme_set("purple")
  p <- bayesplot::mcmc_areas(big_model, pars = c("Temp", "Wind"))

  ggplot2::ggsave(here::here("./figs/big_plot.png"), p)
}

problem

I've been able to get the controller to send the jobs to the SLURM launcher (the job actually launches), and the pipeline begins to run (I can see it saves the output from the plot target, but it always fails after what (I suspect) is the end of the targets that use the smaller worker. Here's the output from the R console I open in the project directory and launch the pipeline from (with tar_make()):

I can't seem to figure out where the target NA error: 'errorValue' int 5 | Timed out ? I have tried adding a command into the controllers for seconds_timeout = 60 but that doesn't seem to change anything.

The larger worker seems to actually launch, it does get queued by dispatched target big, but for some reason as soon as the smaller worker fails it gets cancelled, OR it launches but runs for like 30 seconds and fails with error 143 which corresponds to the Linux signal SIGTERM, which makes me think somehow the targets pipeline itself is killing it?

The output of the error files for both the workers aren't super interpret-able (unless I'm missing something?), here's the one from the smaller worker:

Saving 7 x 7 in image
/localscratch/spool/slurmd/job24221730/slurm_script
 line 17
 15932 Terminated              R -e 'crew

crew_worker(settings = list(url = "wss
//172.16.128.6
33903/c4b355fcfaa943f6540bd8cd", autoexit = 15L, cleanup = 1L, output = TRUE, maxtasks = Inf, idletime = 6e+05, walltime = Inf, timerstart = 0L, tls = c("-----BEGIN CERTIFICATE-----\nMIIFPzCCAyegAwIBAgIBATANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQDDAwxNzIu\nMTYuMTI4LjYxETAPBgNVBAoMCE5hbm9uZXh0MQswCQYDVQQGEwJKUDAeFw0wMTAx\nMDEwMDAwMDBaFw0zMDEyMzEyMzU5NTlaMDcxFTATBgNVBAMMDDE3Mi4xNi4xMjgu\nNjERMA8GA1UECgwITmFub25leHQxCzAJBgNVBAYTAkpQMIICIjANBgkqhkiG9w0B\nAQEFAAOCAg8AMIICCgKCAgEAmjeApjbyGqXvupiTwlFot7pvPWTXge1/IiViYw3S\nbaqHG+dMpdeV/0zfswSGgLOgNhbjmcwmHMmVw6I9ys0CQmZjxqTkBzqgFp3KdHGB\nMWUmIt3ry9IqgK/qOjJ7IH48Lq8BxM/tNCj/qVE6pHkMdeOxagylrGr+sNQM9XnZ\n0GIhVTSch5AZCtuBL5837G44IHVOhOErgpmR5m7WNwIyV/Vjjx8TLYvX4dNh6Fy7\nOiyMM1X77Uk2Bdjhzdi85CV45zEsQ8HAzQPxIr0P2G8Nji+TpZUygh0W3zg6W73Y\n0iTt+CBOkwgnvzr6Btj0JoUujXAD0uhK2/kdLw6mKEw5l27xLqWG3fuH8nThyv7d\n+GfpSiDkhmlalux+Ne9b3J6lCb03UM/cv3ZGTTq3YkRy9TQEVTXfOBR8ZTHMofG7\nT7RdZXO9pPx1+zTKNnCsLFr0ysN5Qxu79JBvtOYsV9oTNhXHJw3MkXsKtUi8Wsy9\nvf9Ve4qB3vHmGN48jXs+h6Jw/xaWYmQtFe1PWf/tZ94oQQQqxgrpGvZuOt7Y7i0Q\nY8dEcRbzGSd+YdX2VGXD4fv1WKdrtbkF+IvwqEAm/H0yFIEGE8w2Qg9mivWc9yzt\nZ6IEUrqdct8F8pjOOyEWfSsxoE8YgBC8b9i2+c9URIr/EbEVAIo+EhBWYAWtc+on\nS3sCAwEAAaNWMFQwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUeTA2yGU9\naiyUgMT6wN2l+EjxdbYwHwYDVR0jBBgwFoAUeTA2yGU9aiyUgMT6wN2l+EjxdbYw\nDQYJKoZIhvcNAQELBQADggIBACW2WvoCmKuSs9tSgKSyVD4IQkf+KCoIz6nUtORP\nxbZRV/5NE4wnbZXTPa3kfn7yOh6v8DWrxYW8dD5njQ8z1UrHLnZJW3NTupSlOQkS\nZwAsClKfXv8qQK0TLn+ZPSePBuK35BJuOc7SWjkgPBRgtso6DB7sjy+9Zt6AKofp\nG4mY+ikOFldHxbglezgl4YoH7xl54YAyDeHQVAikzSXV0PnwXm1rpd+ABSCg/Ixl\nAlljP9xvPqoeEEEwTzrDb3AAYi1+ukwhSOpFS3uDMx7U7PnnDG5yJWUNsPMXEpiu\nFcWGZr7x8N8vUf7nWKQqRZM+ewTSuZU0wjekHq4RKhNQh6bDCAshdKNC4+AzJ8ul\nZyci+wxuPEBFjK9x0OhhXKq8np8m3AaapGeKgK8ydfqOfqhJl46HGyqIhNvT8F6u\nSV2Y9jXXrn+E+JFMpXZgtAHbOKokc4p2hntm8i2MzOgwV/Al+dRLLErxKN9fZr3N\nz4PNh5Ux49MIEL7yMPcjVkouEUZ5ZZ+u8viCq2Cg1nfiSudWsjoFW6fuSSiHGMK/\n1hqavjQ6I7XL7f0jqWw4qsADGdRNHfH6RcIMT+/pEFsDulqUsuWR9bxxqjv5U8H2\n1mco57vK5spunK9DPStsansYzFX0wjx8jV9fQZNpGynRi5OV9svBsUFXO2AJagUi\nEWUO\n-----END CERTIFICATE-----\n", ""), rs = c(10407L, 1494541954L, 1274228763L, 1531569312L, -1083689951L, -986158226L, 923835671L)), launcher = "small_slurm", worker = 1L, instance = "c4b355fcfaa943f6540bd8cd")'

Anyways, I'm curious if there's anything obvious I'm doing here that's causing the small worker to fail even though it does successfully move through the pipeline as far as I can tell.

Any help very appreciated 🙏

[wlandau]

That timeout error can happen if the mirai dispatcher runs out of memory or becomes unresponsive when it is trying to upload/download large data, such as a Stan object with 10000 MCMC iterations per chain. You already have garbage_collection = TRUE and memory = "transient", which is good, but even more important for your case is storage = "worker" and retrieval = "worker"`.

You might also have better luck upgrading to the CRAN versions of mirai and nanonext, as well as the GitHub development versions of crew and `targets.

The larger worker seems to actually launch, it does get queued by dispatched target big, but for some reason as soon as the smaller worker fails it gets cancelled, OR it launches but runs for like 30 seconds and fails with error 143 which corresponds to the Linux signal SIGTERM, which makes me think somehow the targets pipeline itself is killing it?

If you are using the latest crew, nanonext, and mirai versions on CRAN, then workers should automatically send themselves SIGTERM when the dispatcher exits. At that point, the work is usually unrecoverable anyway and we don't want to waste resources on the cluster.

[colebrookson]

This is helpful thank you!!

So to try and narrow down the problem, I tried just a simpler targets script with only one controller that launches one job with lots of memory to eliminate that as a possibility:

library(targets)
library(here)
library(crew)
library(crew.cluster)

source(here("./src/R/00_functions.R"))

controller_small <- crew.cluster::crew_controller_slurm(
  name = "small_slurm",
  slurm_time_minutes = 10,
  seconds_idle = 600,
  script_lines = c(
    "#SBATCH --mem-per-cpu=5G",
    "#SBATCH --mail-user=cole.brookson@gmail.com",
    "#SBATCH --account=def-bat3man",
    "#SBATCH --mail-type=BEGIN",
    "#SBATCH --mail-type=END",
    "#SBATCH --mail-type=FAIL",
    "#SBATCH --mail-type=REQUEUE",
    "#SBATCH --cpus-per-task=1",
    "module load StdEnv/2023 r/4.3.1"
  ),
  slurm_log_output = "/home/brookson/scratch/output.txt",
  slurm_log_error = "/home/brookson/scratch/error.txt",
)

tar_option_set(
  packages = c("readr", "dplyr", "ggplot2", "rstanarm", "qs"),
  controller = controller_small,
  resources = tar_resources(
    crew = tar_resources_crew(controller = "small_slurm")
  ),
  garbage_collection = TRUE,
  memory = "transient",
  storage = "worker",
  retrieval = "worker"
)

list(
  tar_target(data, get_data(here::here("./data/airquality.csv"))),
  tar_target(model, fit_model(data)),
  tar_target(plot, plot_model(model, data))
)

with your suggestions on the storage = "worker" and retrieval = "worker", and this seems to run the pipeline at least as far as targets/R are concerned? I can check by when the plot target was saved that it's running, and I'm getting no errors on the R end:

but my struggle is still that the SLURM job is registering as failing? The pipeline is running with no errors, but somehow as it's exiting the workflow it's failing the job itself:

The error log is similar to last time:

I did make sure I was running the most up-to-date versions of crew, nanonext, and mirai, and I've got the latest github versions of both targets and crew.cluster. Since it's the same error with the SIGTERM, I'm suspicious that something about the workers ending still going to SIGTERM even though there's no error been thrown.

Curious if you have any intuition as to why that would be?

[wlandau]

The screenshot with green text suggests that either the cluster or the operating system terminated the crew worker. Given your other comments, the worker may be running out of memory. You could increase the amount of memory allotted using the slurm_memory_gigabytes_per_cpu of crew.cluster::crew_controller_slurm() (if my interpretation of --mem-per-cpu is correct and this is required memory and not a memory cap).

[wlandau]

The SLURM job log files were extremely helpful to have in this case (for onlookers: slurm_log_output and slurm_log_error arguments for crew.cluster::crew_controller_slurm(), equivalent arguments for other controllers). So thank you for turning them on.

[colebrookson]

Thanks for your suggestions!! I'm really not sure what an alternative explanation is, but I've tried messing with the memory and I can't really find a solution along those lines. I've messed around with it quite a bit, but even if I simplify down the script to just the following:

library(targets)
library(here)
library(crew)
library(crew.cluster)

controller_small <- crew.cluster::crew_controller_slurm(
  name = "small_slurm",
  slurm_time_minutes = 5,
  seconds_idle = 10,
  tasks_max = Inf,
  verbose = TRUE,
  script_lines = c(
    "#SBATCH --mem-per-cpu=8G",
    "#SBATCH --mail-user=cole.brookson@gmail.com",
    "#SBATCH --account=def-bat3man",
    "#SBATCH --mail-type=BEGIN",
    "#SBATCH --mail-type=END",
    "#SBATCH --mail-type=FAIL",
    "#SBATCH --mail-type=REQUEUE",
    "#SBATCH --cpus-per-task=1",
    "module load StdEnv/2023 r/4.3.1"
  ),
  slurm_log_output = "/home/brookson/scratch/output.txt",
  slurm_log_error = "/home/brookson/scratch/error.txt",
  script_directory = "/home/brookson/scratch/script/"
)

tar_option_set(
  packages = c("readr", "dplyr", "ggplot2", "rstanarm", "qs"),
  controller = controller_small,
  resources = tar_resources(
    crew = tar_resources_crew(controller = "small_slurm")
  ),
  garbage_collection = TRUE,
  memory = "transient",
  storage = "worker",
  retrieval = "worker"
)

get_data <- function(file) {
  read_csv(file, col_types = cols()) %>%
    filter(!is.na(Ozone))
}

list(
  tar_target(data, get_data(here::here("./data/airquality.csv")))
)

I get the same error - 143 to the SIGTERM, but I've allocated 8GB of RAM (I did check that --mem-per-cpu is in fact the minimum not the cap), and it still throws the same error, and here I've just asked to run the data target which reads in the file and does one filtering operation, which on my local or on the cluster without this framework needs less than 500MG.

My only thought is there's some communication that's going targets -> crew.cluster -> mirai that I'm not understanding, because the pipeline shows to be completing fine once the job launches

The error.txt:

Perhaps this is beyond the scope of this Q&A function, but I can't really find the specifics of what targets informs to crew.cluster which informs mirai which is the end-point of interacting with the actual HPC if I understand correctly. It looks like other folks have successfully run targets plus crew.cluster on SLURM systems before so I'm sure I'm missing something.

@stemangiola I see you've done some discussion here and ostensibly have used targets with crew.cluster, have you ever had any similar challenges?

[wlandau]

I adapted your example at #37 (reply in thread) to my company's SGE cluster and ran it there, and the pipeline completed successfully. So I think it's something to do with your computing environment.

If it's a communication error among packages like you described, it's almost always due to incompatible packages. I highly recommend upgrading your packages and making sure that the compute nodes share the same package libraries and package versions as the local R session. If you are running and renv environment, how you do this might be different than if you are using the system package library or a personal package library. These are the current CRAN versions you'll need: nanonext 0.13.0, mirai 0.12.1, crew 0.9.0, and crew.cluster 0.3.0. targets 1.5.0 is unfortunately waiting for CRAN to accept it, but you can install the development verison with remotes::install_github("ropensci/targets").

Other than that, you can try peeling back layers: crew without targets, and then mirai without crew. Please try the stuff at #36 (comment).