Skip to content

Running hecatomb on a cluster

shandley edited this page Jun 16, 2021 · 1 revision

Using hecatomb on a cluster

You can run hecatomb on a single machine but it excels when you run it across a cluster. We recommend using slurm or sge

Tip: SGE and slurm are two similar cluster queuing systems, and your cluster likely has one or the other. If you routinely use qsub to submit jobs, your cluster is probably using SGE. If you routinely use sbatch to submit jobs, your cluster is probably using slurm. Not sure? Ask your systems administrator for advice!

Running on SGE

If you are using SGE or a variant, you can run hecatomb using the snakemake cluster command:

mkdir sge_out sge_err
snakemake -s snakemake/workflow/Snakefile --configfile config/sample_config.yaml --cluster 'qsub -cwd -o sge_out -e sge_err' --local-cores 6 --cores 600 --latency-wait 60  --default-resources cpus=1 mem_mb=2000 --use-conda --conda-frontend mamba

Note that here we have added a couple of additional options that you should vary depending on your cluster configuration:

  • -cwd means using the current working directory
  • -o sge_out and -e sge_err write the output and error log files to sge_out and sge_err respectively. Because we make these as directories individual output files are written.
  • --local-cores is how many cores there are on the master node of the cluster
  • --cores is how many compute cores there are on the cluster
  • --latency-wait allows time for the files to be transfered once the job has finished computing.
  • --default-resources specifies the default memory request and cpu request if it is not overridden by a job
  • --use-conda is to ensure we use conda for all software installation
  • --conda-frontend mamba uses the mamba replacement for conda. This is optional, but recommended.

Running on Slurm

You can run on slurm with a similar command, just changing the --cluster part:

mkdir logs_slurm
snakemake -s snakemake/workflow/Snakefile --configfile snakemake/config/sample_config.yaml --cluster 'sbatch  --mem={resources.mem_mb} -c {resources.cpus} -o logs_slurm/{rule}_{jobid} _{jobid}.out -e logs_slurm/{rule}_{jobid}.err' --local-cores 32 --cores 600 --latency-wait 60 --default-resources cpus=1 mem_mb=2000 --use-conda --conda-frontend mamba

This puts the output and error files in the directory logs_slurm. On my cluster, nothing runs if I forget to make the logs_slurm output directories, though!

Advanced. Set up your snakemake environment

We recommend using profiles (see these great and great blogs for more information).

You can make a profile for a computer with no cluster, sge, or slurm.

In any case, we are going to make a directory and file in your home directory. That way, whenever you run snakemake it can access it.

SGE profile

For SGE we will make a file called config.yaml in the sge directory:

mkdir -p ~/.config/snakemake/sge/
vi ~/.config/snakemake/sge/config.yaml

In that file put these contents:

# non-slurm settings
jobs: 600
use-conda: True
conda-frontend: mamba
default-resources: [cpus=1, mem_mb=2000]
keep-going: True

# sge settings
cluster: "qsub -cwd -o sge_out -e sge_err -pe smp {resources.cpus} -V "
latency-wait: 60
local-cores: 6

Tip: Your cluster may have slightly different parallel environments. To figure out what is avaialble, try the command qconf -spl or ask your systems administrator. You may need to change the smp part in this file.

Tip: Adjust the local-cores and cores to reflect your cluster.

Once you have set this up, you can run the same snakemake command as before a lot simpler:

mkdir sge_out sge_err
snakemake -s snakemake/workflow/Snakefile --configfile config/sample_config.yaml --profile sge

Slurm profile

For slurm we will make a file called config.yaml in the slurm directory:

mkdir -p ~/.config/snakemake/slurm/
vi ~/.config/snakemake/slurm/config.yaml

Put this in that file:

# non-slurm settings

jobs: 10
use-conda: True
conda-frontend: mamba
default-resources: [cpus=1, mem_mb=2000]
keep-going: True


# slurm settings

cluster: "sbatch -t {resources.time_min} --mem={resources.mem_mb} -c {resources.cpus} -o logs_slurm/{rule}_{jobid}.out -e logs_slurm/{rule}_{jobid}.err "
latency-wait: 60
local-cores: 32

Once you have set this up, you can run the same snakemake command as before a lot simpler:

mkdir logs_slurm
snakemake -s snakemake/workflow/Snakefile --configfile config/sample_config.yaml --profile slurm

The config file

You should make a copy of the config file. We typically make a copy of that file into each directory where we are working. Then if you make any changes to that file they reside with the data. There are example config files in both JSON and YAML, and of course snakemake can use either. (If you are not sure, YAML is probably easier to start with than JSON).

The key things in the config file are:

  1. The database file location. You can set that in the config file and then create the database as described below
  2. The directory name where your raw reads (fastq files) reside.

You can adjust almost everything else as needed, and the directories will be created when the data is generated.

Setting up snakemake:

We use snakemake profiles (see these great and great blogs). Our default profile is

# non-slurm settings

use-conda: True
conda-frontend: mamba


# slurm settings

jobs: 10
cluster: "sbatch -t {resources.time_min} --mem={resources.mem_mb} -c {resources.cpus} -o logs_slurm/{rule}_{jobid}.out -e logs_slurm/{rule}_{jobid}.err "
default-resources: [cpus=1, mem_mb=2000, time_min=60]
latency-wait: 60
local-cores: 32
latency-wait: 60
keep-going: True

With that profile in ~/.config/snakemake/slurm/config.yaml, and $HECATOMB set to the base directory you checked out from GitHub, you should be able to run this with:

snakemake --profile slurm --configfile config.yaml -s $HECATOMB/snakemake/hecatomb_alt.snakefile