Emeli's steps for using workflows

[ejanderson1]

Overview:

• Adrien's workflow code can be used to send batch jobs to the HPC
• Can specify multiple scenarios to test in one script
• Replaces using separate sim.R, runSim.sh, and mainSim.sh files and submitting your job using bash
• Note: there may be a better way to do some of the steps below - this is just how I have been using it (@AdrienLeGuillou maybe you can comment if anything is incorrect)

Before you begin

• Add workflows/ to your gitignore
• This ensures that your workflow folder won't be pushed to GitHub, which can mess things up on the HPC (I believe)

Step 1 - create workflow script

• The first step is to write your workflow script. Here is an example from the HIV template repo for calibration:
  https://github.com/EpiModel/EpiModelHIV-Template/blob/100k/R/workflow-model_calibration.R
• You can modify this script to run your full scenarios post-calibration (I'll add more notes on that later)

Step 2 - run workflow script

• Running the script saves a workflows folder in your project
• Inside the workflows folder is another folder with the name you give your workflow when you create it (more on this later)

Step 3 - create wf directory on HPC

• Create a workflows directory in your project on the HPC. You can do this by logging into the HPC, going to your project directory, and using the mkdir command or you can run the following from terminal when not logged into the HPC:
  ssh <user>@clogin01.sph.emory.edu "mkdir -p ~/projects/BigNets/workflows" where you insert the location that you want your workflows folder in place of the BigNets code on the right.

Step 4 - copy wf files to HPC

• The next step is to copy what is in your local workflows folder to the workflows folder on the HPC:
  scp -r workflows/[name_of_wf] <user>@clogin01.sph.emory.edu:/projects/epimodel/[user name]/[your project name]/workflows/
• Again this should be done in terminal when not logged into the HPC

Step 5 - check that your copying worked

• Now you can login to the HPC and move to your project directory
• Check that the workflows folder is there and that your workflow is inside it:
  ls -lh workflows
• If you pushed any changes to GitHub recently, you can run git pull now as well

Step 6 - submit batch job

• Once you confirm your workflow is on the HPC you can run it
• This involves 2 pieces of code:

1. Run:
   chmod +x workflows/[name_of_wf]/start_workflow.sh

2. Then:
   ./workflows/[name_of_wf]/start_workflow.sh

• At this point you should receive a batch number confirming that your code was submitted. If you did not, then something went wrong.

[ejanderson1]

Notes on re-running a workflow

(i.e., if you want to make changes to workflow script and resubmit it)

• delete the workflows folder in your project locally
• make sure the workflows directory the HPC is empty (but still there or you'll have to remake the directory)
• delete any sims in your output directory on the HPC so that when your new batch job output doesn't get mixed with old output (make sure they are copied to your laptop if you need them before deleting)
• Once that is done you can follow the same steps as above

[ejanderson1]

Creating a workflow script

• Below I go through each step of my own script for submitting several scenarios to the HPC. I'll point out what you'll need to change for your script.

Step 1 - setup:

# Setup ------------------------------------------------------------------------
library(slurmworkflow)
library(EpiModelHPC)
library(data.table)
library(EpiModelHIV)

hpc_configs <- swf_configs_rsph(
  partition = "preemptable",
  mail_user = "[email_address]@emory.edu"
)

sim_dir <- "data/intermediate/sims"
max_cores <- 5

• For hpc-configs the partition can be preemptable or epimodel
• The sim_dir is the location that your sims will go on the HPCn
• Max cores are the total number of cores you wish to allot to one set of sims. It replaces the ncores argument in control_msm . If you want to run 32 sims per batch, then you need to set this to 32.

Step 2 - create workflow

wf <- create_workflow(
  wf_name = "test",
  default_sbatch_opts = hpc_configs$default_sbatch_opts
)
 

• This step is the one that creates your workflows folder
• You can name the workflow whatever you like using the wf_name argument. Here mine is called test. If I look at the newly created workflows folder in my project there should be another folder within it called test

Step 3 - add a wf step to update renv on the HPC

• This step saves you the trouble of having to run renv::restore() on the HPC

wf <- add_workflow_step(
  wf_summary = wf,
  step_tmpl = step_tmpl_renv_restore(
    git_branch = "main",
    setup_lines = hpc_configs$r_loader
  ),
  sbatch_opts = hpc_configs$renv_sbatch_opts
)

• The only thing you might need to change here is the branch you are pulling from (git_branch)

Step 4 - specify your sims

• This code looks just like your standard sim.R code
• Load netstats, netest, and epistats files:

epistats <- readRDS("data/intermediate/estimates/epistats.rds")
netstats <- readRDS("data/intermediate/estimates/netstats.rds")
orig     <- readRDS("data/intermediate/estimates/restart.rds")

Note that in replace of my epistats file I have a restart file from calibration

• Next run param.net, for example:

param <- param.net(
  data.frame.params = readr::read_csv("data/input/params-100000.csv"),
  netstats          = netstats,
  epistats          = epistats,
  sti.cond.eff.rr = 0.9,
  riskh.start = prep_start - 52,
  prep.start = prep_start,

• Then, init_msm:

init <- init_msm(
  prev.ugc = 0.1,
  prev.rct = 0.1,
  prev.rgc = 0.1,
  prev.uct = 0.1
)

• Finally, control_msm but here you will find some differences with the standard sim code:

control <- control_msm(
  start               = calib_end + 1,
  nsteps              = end_sim,
  nsims               = 1,
  ncores              = 1,
  initialize.FUN      = reinit_msm,
  cumulative.edgelist = TRUE,
  truncate.el.cuml    = 0,
  verbose             = FALSE,
)

• nsims and ncores should be set to 1. These will specified in a later step
• If you are using calibrated data to restart your sims, then make sure the initialize function is reinit_msm

Step 5 - specify your scenarios

• For this step you will create a dataframe of each scenario you want to run on the HPC.

scenarios.df <- dplyr::tibble(
  .scenario.id = c("base", "scen_1", "scen_2", "scen_3"),
  .at = int_start,
  part.index.prob= c(0, 0.333, 0.666, 0.999))

scenarios.list <- EpiModel::create_scenario_list(scenarios.df)

• The above code creates 4 scenarios, each using a different value for part.index.prob
• You can look at the dataframe to make sure it lists each scenario correctly

Step 6 - netsim code

wf <- add_workflow_step(
  wf_summary = wf,
  step_tmpl = step_tmpl_netsim_scenarios(
    orig, param, init, control,
    scenarios_list = scenarios.list,
    output_dir = sim_dir,
    libraries = "EpiModelHIV",
    n_rep = 32,
    n_cores = max_cores,
    max_array_size = 500,
    setup_lines = hpc_configs$r_loader
  ),
  sbatch_opts = list(
    "mail-type" = "END",
    "cpus-per-task" = max_cores,
    "time" = "04:00:00",
    "mem" = "0" # special: all mem on node
  )
)

• This step is the one where netsim is run
• you give it your scenario df, est, param, init, and control objects, the output directory you created above, plus some additional inofrmation
• n_rep is the same as nsims in the control function; for example, you might want to do 64 sims
• n_cores is set to the number of cores you want allot to each batch; for example, if you do 64 sims and want to 2 batches, you would set this to 32
• Other parts you can leave the same