-
Notifications
You must be signed in to change notification settings - Fork 12
Running hecatomb on a cluster
You can run hecatomb
on a single machine but it excels when you run it across a cluster. We recommend using slurm
or sge
Tip:
SGE
andslurm
are two similar cluster queuing systems, and your cluster likely has one or the other. If you routinely useqsub
to submit jobs, your cluster is probably usingSGE
. If you routinely usesbatch
to submit jobs, your cluster is probably usingslurm
. Not sure? Ask your systems administrator for advice!
If you are using SGE
or a variant, you can run hecatomb
using the snakemake cluster command:
mkdir sge_out sge_err
snakemake -s snakemake/workflow/Snakefile --configfile config/sample_config.yaml --cluster 'qsub -cwd -o sge_out -e sge_err' --local-cores 6 --cores 600 --latency-wait 60 --default-resources cpus=1 mem_mb=2000 --use-conda --conda-frontend mamba
Note that here we have added a couple of additional options that you should vary depending on your cluster configuration:
-
-cwd
means using the current working directory -
-o sge_out
and-e sge_err
write the output and error log files tosge_out
andsge_err
respectively. Because we make these as directories individual output files are written. -
--local-cores
is how many cores there are on the master node of the cluster -
--cores
is how many compute cores there are on the cluster -
--latency-wait
allows time for the files to be transfered once the job has finished computing. -
--default-resources
specifies the default memory request and cpu request if it is not overridden by a job -
--use-conda
is to ensure we use conda for all software installation -
--conda-frontend mamba
uses the mamba replacement for conda. This is optional, but recommended.
You can run on slurm with a similar command, just changing the --cluster
part:
mkdir logs_slurm
snakemake -s snakemake/workflow/Snakefile --configfile snakemake/config/sample_config.yaml --cluster 'sbatch --mem={resources.mem_mb} -c {resources.cpus} -o logs_slurm/{rule}_{jobid} _{jobid}.out -e logs_slurm/{rule}_{jobid}.err' --local-cores 32 --cores 600 --latency-wait 60 --default-resources cpus=1 mem_mb=2000 --use-conda --conda-frontend mamba
This puts the output and error files in the directory logs_slurm
. On my cluster, nothing runs if I forget to make the logs_slurm output directories, though!
We recommend using profiles (see these great and great blogs for more information).
You can make a profile for a computer with no cluster, sge, or slurm.
In any case, we are going to make a directory and file in your home directory. That way, whenever you run snakemake
it can access it.
For SGE
we will make a file called config.yaml
in the sge directory:
mkdir -p ~/.config/snakemake/sge/
vi ~/.config/snakemake/sge/config.yaml
In that file put these contents:
# non-slurm settings
jobs: 600
use-conda: True
conda-frontend: mamba
default-resources: [cpus=1, mem_mb=2000]
keep-going: True
# sge settings
cluster: "qsub -cwd -o sge_out -e sge_err -pe smp {resources.cpus} -V "
latency-wait: 60
local-cores: 6
Tip: Your cluster may have slightly different parallel environments. To figure out what is avaialble, try the command
qconf -spl
or ask your systems administrator. You may need to change thesmp
part in this file.
Tip: Adjust the
local-cores
andcores
to reflect your cluster.
Once you have set this up, you can run the same snakemake command as before a lot simpler:
mkdir sge_out sge_err
snakemake -s snakemake/workflow/Snakefile --configfile config/sample_config.yaml --profile sge
For slurm
we will make a file called config.yaml
in the slurm directory:
mkdir -p ~/.config/snakemake/slurm/
vi ~/.config/snakemake/slurm/config.yaml
Put this in that file:
# non-slurm settings
jobs: 10
use-conda: True
conda-frontend: mamba
default-resources: [cpus=1, mem_mb=2000]
keep-going: True
# slurm settings
cluster: "sbatch -t {resources.time_min} --mem={resources.mem_mb} -c {resources.cpus} -o logs_slurm/{rule}_{jobid}.out -e logs_slurm/{rule}_{jobid}.err "
latency-wait: 60
local-cores: 32
Once you have set this up, you can run the same snakemake command as before a lot simpler:
mkdir logs_slurm
snakemake -s snakemake/workflow/Snakefile --configfile config/sample_config.yaml --profile slurm
You should make a copy of the config file. We typically make a copy of that file into each directory where we are working. Then if you make any changes to that file they reside with the data. There are example config files in both JSON and YAML, and of course snakemake can use either. (If you are not sure, YAML is probably easier to start with than JSON).
The key things in the config file are:
- The database file location. You can set that in the config file and then create the database as described below
- The directory name where your raw reads (
fastq files
) reside.
You can adjust almost everything else as needed, and the directories will be created when the data is generated.
Setting up snakemake:
We use snakemake profiles (see these great and great blogs). Our default profile is
# non-slurm settings
use-conda: True
conda-frontend: mamba
# slurm settings
jobs: 10
cluster: "sbatch -t {resources.time_min} --mem={resources.mem_mb} -c {resources.cpus} -o logs_slurm/{rule}_{jobid}.out -e logs_slurm/{rule}_{jobid}.err "
default-resources: [cpus=1, mem_mb=2000, time_min=60]
latency-wait: 60
local-cores: 32
latency-wait: 60
keep-going: True
With that profile in ~/.config/snakemake/slurm/config.yaml
, and $HECATOMB set to the base directory you checked out from GitHub, you should be able to run this with:
snakemake --profile slurm --configfile config.yaml -s $HECATOMB/snakemake/hecatomb_alt.snakefile