Slurm Job Example

Step 1: Make a subjects file listing the subject numbers (e.g., subjects.txt), one per line, as in:

1131276 
1131277 
1131327 
1131365 ...

Step 2: Make your job file template. This file calls your bash script, but does it through slurm. A python script replaces the subject number in this template file, saves a new subject-specific job file, and submits the job. The example below submits a lausanne atlas job (Note: save in scratch).

#!/bin/bash
##---Number of core
#SBATCH -n 4
#
##---Job's name in slurm system
#SBATCH -J lausanne_subject
#
##---Error file
#SBATCH -e /home/data/dcnlab/AHEAD/error/lausanne_err_subject
#
##---Output file
#SBATCH -o /home/data/dcnlab/AHEAD/out/lausanne_out_subject
#
##---Queue name
#SBATCH -q pq_dcn
#
##---Access
#
#SBATCH -A acc_dcn

##---Memory Allocation
#SBATCH --mem-per-cpu 4000
#
##---Partition
#SBATCH -p investor
#
##########################################################
# Set up environmental variables.
##########################################################
##########################################################

export NPROCS=`echo $LSB_HOSTS | wc -w`
export OMP_NUM_THREADS=$NPROCS

MODULESHOME=/home/share/Modules/3.2.10
. $MODULESHOME/../global/profile.modules

module add miniconda/2.7
module load slicer/4.8.1
module load dtiprep/1.2.4
module load tortoise/3.1.1
module load afni/17.3.06
module load dsistudio/1.0
module load fsl/6.0.1

TORTDIR=/home/applications/tortoise/3.1.1/DIFFPREPV311/bin/bin
DRBUDDIDIR=/home/applications/tortoise/3.1.1/DRBUDDIV311/bin
SLICERDIR='/home/applications/slicer/4.8.1/lib/Slicer-4.8/cli-modules'
DIFFCALCDIR=/home/applications/tortoise/3.1.1/DIFFCALC/DIFFCALCV311

##########################################################
##########################################################
sub=subject

bash /home/data/dcnlab/AHEAD/dset/scripts/lausanne_AHEAD.sh $sub /home/data/dcnlab/AHEAD

Step 3: Write your bash script for the job. The example below is to apply the lausanne atlas transformation. The text below would be in a .sh file named lausanne_AHEAD.sh, which is called by the slurm job script.

#!/bin/bash
##make sure to run module load from submit jobs script before running this line by line
module add miniconda/2.7
module load afni/17.3.06
module load freesurfer/dcnlab
#sample command:
#bash lausanne_submitjobs.sh 4150318 /home/data/dcnlab/AHEAD
sub=$1
#sub=4150262
echo $sub
workdir=$2
#workdir=/home/data/dcnlab/AHEAD
echo $workdir
fsurfdir=/home/applications/freesurfer/6.0/freesurfer/bin
echo $fsurfdir
cd $workdir/freesurfer_v6/$sub/
export SUBJECTS_DIR=$workdir/freesurfer_v6

###This section is freesurfer commands###
#left hemi
$fsurfdir/mris_ca_label $sub lh lh.sphere.reg $workdir/lausanne_atlas/my_atlas_gcs/myatlas_250_lh.gcs $workdir/freesurfer_v6/$sub/label/lh.lausanne2008_aparc.annot
#right hemi
$fsurfdir/mris_ca_label $sub rh rh.sphere.reg $workdir/lausanne_atlas/my_atlas_gcs/myatlas_250_rh.gcs $workdir/freesurfer_v6/$sub/label/rh.lausanne2008_aparc.annot
#left hemi
$fsurfdir/mri_annotation2label --annotation lausanne2008_aparc --hemi lh --subject $sub --outdir ./lausanne2008_label
#right hemi
$fsurfdir/mri_annotation2label --annotation lausanne2008_aparc --hemi rh --subject $sub --outdir ./lausanne2008_label
#left hemi
$fsurfdir/mris_label2annot --ctab $workdir/lausanne_atlas/my_atlas_gcs/lh.lausanne2008_LUT.txt --subject $sub --ldir lausanne2008_label --no-unknown --annot lausanne2008_aparc_label --hemi lh
#right hemi
$fsurfdir/mris_label2annot --ctab $workdir/lausanne_atlas/my_atlas_gcs/rh.lausanne2008_LUT.txt --subject $sub --ldir lausanne2008_label --no-unknown --annot lausanne2008_aparc_label --hemi rh

$fsurfdir/mris_volmask --label_left_white 2 --label_left_ribbon 3 --label_right_white 41 --label_right_ribbon 42 --save_ribbon --save_distance $sub

$fsurfdir/mri_aparc2aseg --s $sub --annot lausanne2008_aparc

###This section is AFNI commands###

##Go into FS/surf folder
cd $workdir/freesurfer_v6/$sub/surf
echo "Working directory is $workdir"
$workdir/lausanne_atlas/@SUMA_Make_Spec_FS_lausanne.sh -use_mgz -sid $sub
cd $workdir/freesurfer_v6/$sub/surf/SUMA

@Atlasize                                \
    -space ORIG                           \
    -dset lausanne2008_aparc+aseg.nii             \
    -lab_file $workdir/lausanne_atlas/lausanne2008_aparc+aseg.txt 1 0               \
    -lab_file_delim ' '                  \
    -atlas_type S                        \
    -atlas_name lausanne2008                     \
    -atlas_description 'lausanne2008'

cp $workdir/dset/sub-$sub/ses-S1/anat/sub-"$sub"_ses-S1_T1w.nii.gz ./

@SUMA_AlignToExperiment -exp_anat sub-"$sub"_ses-S1_T1w.nii.gz -surf_anat "$sub"_SurfVol+orig -atlas_followers
3dAFNItoNIFTI -prefix lausanne2008_aparc+aseg_rank_Alnd_Exp+orig lausanne2008_aparc+aseg_rank_Alnd_Exp+orig
3dAFNItoNIFTI -prefix lausanne2008_aparc+aseg_Alnd_Exp+orig lausanne2008_aparc+aseg_Alnd_Exp+orig
3dAFNItoNIFTI -prefix "$sub"_SurfVol_Alnd_Exp+orig "$sub"_SurfVol_Alnd_Exp+orig

echo "The atlas script has completed"

Step 5: Write your python script. The text below goes in a file called lausanne_AHEAD.py.

import os
import os.path as op
import shutil
import subprocess
from glob import glob

with open("/home/data/dcnlab/AHEAD/scripts/subject.txt", "r") as f:
    data = f.read().splitlines()
    for subject in data:
        print(subject)
        print("Start script!")
        template_job = "/scratch/adick/lausanne_jobs/lausanne_slurm_jobs.sub"
        subj_job = '/scratch/adick/lausanne_jobs/{0}_lausanne_AHEAD.sub'.format(subject)
        print(subj_job)
        with open(template_job, 'r') as fo:
            text = fo.read()
            text = text.replace('subject', subject)
        with open(subj_job, 'w') as fo:
            fo.write(text)
            os.chmod(subj_job, 0o775)
        subprocess.call('sbatch {0}'.format(subj_job), shell=True)

Step 6: Call the python file: python lausanne_AHEAD.py