From aaaa1c416f1b981bcdf4682f5a4980920aabdb14 Mon Sep 17 00:00:00 2001 From: Nick Scholand Date: Sat, 23 Nov 2024 17:24:18 +0100 Subject: [PATCH] singularity: Update doc and add openMPI and CuDNN --- doc/singularity/README.md | 99 ++++++++++++++++++++++++++++++--- doc/singularity/bart_debian.def | 5 +- doc/singularity/bart_ubuntu.def | 12 +++- 3 files changed, 107 insertions(+), 9 deletions(-) diff --git a/doc/singularity/README.md b/doc/singularity/README.md index 129dbc7e0..7d022b2b0 100644 --- a/doc/singularity/README.md +++ b/doc/singularity/README.md @@ -1,8 +1,9 @@ # Singularity Container for BART Running BART on a high performance computing (HPC) cluster can be challenging due to missing libraries. -One solution exploits containers. They encapsulating the software with all required dependencies. +One solution exploits containers. They encapsulate software with all their dependencies. Here, we provide some basic information about how to run BART in a container using [Singularity](https://sylabs.io/singularity/). +The definition files have been tested with Singularity-CE in version 3.8.0. A blueprint to create a singularity container for BART can be found in the definition files * [`bart_ubuntu.def`](bart_ubuntu.def): for an Ubuntu 22.04 operating system. @@ -15,18 +16,102 @@ sudo singularity build container.sif bart.def Both containers download and compile BART with all libraries including GPU support using CUDA. Make sure to select the CUDA version your local HPC host provides. +The definition files above also represent simple guides of how to install BART on the individual operating systems. -You can start an interactive shell with +### Run Scripts in Singularity Containers -```code +To run a container load the required `singularity` and potentially `cuda` module on your HPC: +```bash +module load singularity +module load cuda +``` +After transferring the container to BigPurple a script `recon.sh` can be executed within the container using `bash`: +```bash +singularity exec --nv container.sif bash 'recon.sh' +``` +`--nv` allows the container to access the Nvidia drivers on the host system and is mandatory for running BART with GPU support. + +### Run BART from the CLI inside a Singularity Container +Using the singularity image file, an interactive session can be started to run BART's command line tools: +```bash singularity shell --nv container.sif ``` -and the `--nv` adds access to the installed Nvidia drivers on the host system. -A BASH script can be executed inside the container with +## Mounting Directories +It is convenient to mount directories in the singularity container to access executables or files from within the container. +This can be done by adding `--bind`. To mount the directory `/tmp` from the host to the location `/tmp` within the container. +```bash +singularity shell --nv --bind /tmp:/tmp container.sif +``` -```code -singularity exec --nv container.sif bash 'recon.sh' +## Example +In this short example, we use a container `container.sif` on our HPC and compile and run a local BART version. + +##### 1. Download BART +On the host, we download the newest master branch version of BART into the `/tmp` directory: +```bash +cd /tmp +git clone https://github.com/mrirecon/bart.git +``` +and enter it +```bash +cd bart +``` + +##### 2. Create local Makefile for GPU Compilation +To compile BART with GPU support, we have to set the environment variable `CUDA=1`. +While this could be set during compilation +```bash +CUDA=1 make +``` +the recommended solution is to just create a local Makefile +```bash +touch Makefiles/Makefile.local +printf "PARALLEL=1\nCUDA=1\nCUDA_BASE=/usr/local/cuda\nCUDA_LIB=lib64\n" > Makefiles/Makefile.local +``` +With `CUDA_BASE=/usr/local/cuda` and `CUDA_LIB=lib64` you can define the local CUDA paths and with `PARALLEL=1` the compilation is performed in parallel. + +##### 3. Create Interactive Session Inside Container +Next, the `singularity` and `cuda` modules are loaded in the host session on the HPC +```bash +module load singularity +module load cuda +``` +and an interactive session within the container can be started +```bash +singularity shell --nv --bind /tmp:/tmp /container.sif +``` +Per default the session is started in the directory from which the `singularity` call has been executed. +Please make sure that you are already in the `/tmp/bart` folder +```bash +pwd +``` + +##### 4. Compile BART +To compile BART execute +```bash +make +``` + +##### 5. Run BART with GPU Support +The singularity container contains a default BART version. To make sure the newly compiled version is used in your scripts save the path of the executable and update the `$PATH` variable +```bash +BART="$(pwd)"/bart +export PATH=$BART:$PATH +``` + +Finally, BART can be started in the interactive container session +```bash +bart +``` +and a short test verifies that the GPU support is provided: +```bash +# Generate k-space-based phantom data +bart traj t +bart phantom -k -t t p + +# Perform an nuFFT on the GPU +bart nufft -g t p reco ``` ### Note diff --git a/doc/singularity/bart_debian.def b/doc/singularity/bart_debian.def index c3a686acc..8587e9fac 100644 --- a/doc/singularity/bart_debian.def +++ b/doc/singularity/bart_debian.def @@ -29,6 +29,9 @@ From: debian:12 # Install useful tools for the installation process and workflows apt-get install -y make gcc git wget vim dpkg unzip screen time bc + # Install openMPI + apt-get install -y libopenmpi-dev libopenmpi3 openmpi-bin openmpi-common openmpi-doc + # Install CUDA CUDA_VERSION=12-3 wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb @@ -44,7 +47,7 @@ From: debian:12 rm v${BART_VERSION}.zip cd bart touch Makefiles/Makefile.local - printf "PARALLEL=1\nCUDA=1\nCUDA_BASE=/usr/local/cuda\nCUDA_LIB=lib64\n" > Makefiles/Makefile.local + printf "PARALLEL=1\nCUDA=1\nCUDA_BASE=/usr/local/cuda\nCUDA_LIB=lib64\nMPI=1\n" > Makefiles/Makefile.local make cd .. diff --git a/doc/singularity/bart_ubuntu.def b/doc/singularity/bart_ubuntu.def index 4f0a98d3f..a60ba0d76 100644 --- a/doc/singularity/bart_ubuntu.def +++ b/doc/singularity/bart_ubuntu.def @@ -27,6 +27,9 @@ From: ubuntu:22.04 # Install useful tools for the installation process and workflows apt-get install -y make gcc git wget vim dpkg unzip screen time bc + # Install openMPI + apt-get install -y libopenmpi-dev libopenmpi3 openmpi-bin openmpi-common openmpi-doc + # Install CUDA CUDA_VERSION=12-0 wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb @@ -34,6 +37,13 @@ From: ubuntu:22.04 apt-get update DEBIAN_FRONTEND=noninteractive apt-get -y install cuda-${CUDA_VERSION} # `DEBIAN_FRONTEND=noninteractive` avoids keyboard pop-up + # Install CuDNN + wget https://developer.download.nvidia.com/compute/cudnn/9.2.0/local_installers/cudnn-local-repo-ubuntu2204-9.2.0_1.0-1_amd64.deb + dpkg -i cudnn-local-repo-ubuntu2204-9.2.0_1.0-1_amd64.deb + cp /var/cudnn-local-repo-ubuntu2204-9.2.0/cudnn-*-keyring.gpg /usr/share/keyrings/ + apt-get update + apt-get -y install cudnn-cuda-12 + # Install BART and compile it with GPU support BART_VERSION=0.9.00 wget https://github.com/mrirecon/bart/archive/v${BART_VERSION}.zip @@ -42,7 +52,7 @@ From: ubuntu:22.04 rm v${BART_VERSION}.zip cd bart touch Makefiles/Makefile.local - printf "PARALLEL=1\nCUDA=1\nCUDA_BASE=/usr/local/cuda\nCUDA_LIB=lib64\n" > Makefiles/Makefile.local + printf "PARALLEL=1\nCUDA=1\nCUDA_BASE=/usr/local/cuda\nCUDA_LIB=lib64\nCUDNN=1\nMPI=1\n" > Makefiles/Makefile.local make cd ..