singularity: Update doc and add openMPI and CuDNN

doc/singularity/README.md

@@ -1,8 +1,9 @@
 # Singularity Container for BART
 
 Running BART on a high performance computing (HPC) cluster can be challenging due to missing libraries.
-One solution exploits containers. They encapsulating the software with all required dependencies.
+One solution exploits containers. They encapsulate software with all their dependencies.
 Here, we provide some basic information about how to run BART in a container using [Singularity](https://sylabs.io/singularity/).
+The definition files have been tested with Singularity-CE in version 3.8.0.
 
 A blueprint to create a singularity container for BART can be found in the definition files
 * [`bart_ubuntu.def`](bart_ubuntu.def): for an Ubuntu 22.04 operating system. 
@@ -15,18 +16,102 @@ sudo singularity build container.sif bart.def
 
 Both containers download and compile BART with all libraries including GPU support using CUDA.
 Make sure to select the CUDA version your local HPC host provides.
+The definition files above also represent simple guides of how to install BART on the individual operating systems.
 
-You can start an interactive shell with
+### Run Scripts in Singularity Containers
 
-```code
+To run a container load the required `singularity` and potentially `cuda` module on your HPC:
+```bash
+module load singularity
+module load cuda
+```
+After transferring the container to BigPurple a script `recon.sh` can be executed within the container using `bash`:
+```bash
+singularity exec --nv container.sif bash 'recon.sh'
+```
+`--nv` allows the container to access the Nvidia drivers on the host system and is mandatory for running BART with GPU support.
+
+### Run BART from the CLI inside a Singularity Container
+Using the singularity image file, an interactive session can be started to run BART's command line tools:
+```bash
 singularity shell --nv container.sif
 ```
-and the `--nv` adds access to the installed Nvidia drivers on the host system.
 
-A BASH script can be executed inside the container with
+## Mounting Directories
+It is convenient to mount directories in the singularity container to access executables or files from within the container.
+This can be done by adding `--bind`. To mount the directory `/tmp` from the host to the location `/tmp` within the container.
+```bash
+singularity shell --nv --bind /tmp:/tmp container.sif
+```
 
-```code
-singularity exec --nv container.sif bash 'recon.sh'
+## Example
+In this short example, we use a container `container.sif` on our HPC and compile and run a local BART version.
+
+##### 1. Download BART
+On the host, we download the newest master branch version of BART into the `/tmp` directory:
+```bash
+cd /tmp
+git clone https://github.com/mrirecon/bart.git
+```
+and enter it
+```bash
+cd bart
+```
+
+##### 2. Create local Makefile for GPU Compilation
+To compile BART with GPU support, we have to set the environment variable `CUDA=1`.
+While this could be set during compilation
+```bash
+CUDA=1 make
+```
+the recommended solution is to just create a local Makefile
+```bash
+touch Makefiles/Makefile.local
+printf "PARALLEL=1\nCUDA=1\nCUDA_BASE=/usr/local/cuda\nCUDA_LIB=lib64\n" > Makefiles/Makefile.local
+```
+With `CUDA_BASE=/usr/local/cuda` and `CUDA_LIB=lib64` you can define the local CUDA paths and with `PARALLEL=1` the compilation is performed in parallel.
+
+##### 3. Create Interactive Session Inside Container
+Next, the `singularity` and `cuda` modules are loaded in the host session on the HPC
+```bash
+module load singularity
+module load cuda
+```
+and an interactive session within the container can be started
+```bash
+singularity shell --nv --bind /tmp:/tmp <path-to-container>/container.sif
+```
+Per default the session is started in the directory from which the `singularity` call has been executed.
+Please make sure that you are already in the `/tmp/bart` folder
+```bash
+pwd
+```
+
+##### 4. Compile BART
+To compile BART execute
+```bash
+make
+```
+
+##### 5. Run BART with GPU Support
+The singularity container contains a default BART version. To make sure the newly compiled version is used in your scripts save the path of the executable and update the `$PATH` variable
+```bash
+BART="$(pwd)"/bart
+export PATH=$BART:$PATH
+```
+
+Finally, BART can be started in the interactive container session
+```bash
+bart
+```
+and a short test verifies that the GPU support is provided:
+```bash
+# Generate k-space-based phantom data
+bart traj t
+bart phantom -k -t t p
+
+# Perform an nuFFT on the GPU
+bart nufft -g t p reco
 ```
 
 ### Note

doc/singularity/bart_debian.def

@@ -29,6 +29,9 @@ From: debian:12
 	# Install useful tools for the installation process and workflows
 	apt-get install -y make gcc git wget vim dpkg unzip screen time bc
 
+	# Install openMPI
+	apt-get install -y libopenmpi-dev libopenmpi3 openmpi-bin openmpi-common openmpi-doc
+
 	# Install CUDA
 	CUDA_VERSION=12-3
 	wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb
@@ -44,7 +47,7 @@ From: debian:12
 	rm v${BART_VERSION}.zip
 	cd bart
 	touch Makefiles/Makefile.local
-	printf "PARALLEL=1\nCUDA=1\nCUDA_BASE=/usr/local/cuda\nCUDA_LIB=lib64\n" > Makefiles/Makefile.local
+	printf "PARALLEL=1\nCUDA=1\nCUDA_BASE=/usr/local/cuda\nCUDA_LIB=lib64\nMPI=1\n" > Makefiles/Makefile.local
 	make
 	cd ..
 

doc/singularity/bart_ubuntu.def

@@ -27,13 +27,23 @@ From: ubuntu:22.04
 	# Install useful tools for the installation process and workflows
 	apt-get install -y make gcc git wget vim dpkg unzip screen time bc
 
+	# Install openMPI
+	apt-get install -y libopenmpi-dev libopenmpi3 openmpi-bin openmpi-common openmpi-doc
+
 	# Install CUDA
 	CUDA_VERSION=12-0
 	wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
 	dpkg -i cuda-keyring_1.1-1_all.deb
 	apt-get update
 	DEBIAN_FRONTEND=noninteractive apt-get -y install cuda-${CUDA_VERSION} # `DEBIAN_FRONTEND=noninteractive` avoids keyboard pop-up
 
+	# Install CuDNN
+	wget https://developer.download.nvidia.com/compute/cudnn/9.2.0/local_installers/cudnn-local-repo-ubuntu2204-9.2.0_1.0-1_amd64.deb
+	dpkg -i cudnn-local-repo-ubuntu2204-9.2.0_1.0-1_amd64.deb
+	cp /var/cudnn-local-repo-ubuntu2204-9.2.0/cudnn-*-keyring.gpg /usr/share/keyrings/
+	apt-get update
+	apt-get -y install cudnn-cuda-12
+
 	# Install BART and compile it with GPU support
 	BART_VERSION=0.9.00
 	wget https://github.com/mrirecon/bart/archive/v${BART_VERSION}.zip
@@ -42,7 +52,7 @@ From: ubuntu:22.04
 	rm v${BART_VERSION}.zip
 	cd bart
 	touch Makefiles/Makefile.local
-	printf "PARALLEL=1\nCUDA=1\nCUDA_BASE=/usr/local/cuda\nCUDA_LIB=lib64\n" > Makefiles/Makefile.local
+	printf "PARALLEL=1\nCUDA=1\nCUDA_BASE=/usr/local/cuda\nCUDA_LIB=lib64\nCUDNN=1\nMPI=1\n" > Makefiles/Makefile.local
 	make
 	cd ..