From 7da8cdcb03dcab8af97aaceeba8ffb7442827148 Mon Sep 17 00:00:00 2001
From: Caspar Jungbacker <caspar.jungbacker@outlook.com>
Date: Thu, 2 Jan 2025 11:33:41 +0100
Subject: [PATCH 1/2] Update compilation guide

---
 book/running/compilation.ipynb | 29 ++++++++++++-----------------
 1 file changed, 12 insertions(+), 17 deletions(-)

diff --git a/book/running/compilation.ipynb b/book/running/compilation.ipynb
index e33da3f..9724516 100644
--- a/book/running/compilation.ipynb
+++ b/book/running/compilation.ipynb
@@ -22,28 +22,23 @@
     "```{code} shell\n",
     "make \n",
     "```\n",
-    "After successfull compilation, the executable `dales4.4` (@team: number to change) is located in the subdirectory `src/`. Compilation of the code may be sped up using the `-j <nprocs>` specifier, with `nprocs` being the amount of parallel processes.\n",
+    "After successfull compilation, the executable `dales` is located in the subdirectory `bin/`. Compilation of the code may be sped up using the `-j <nprocs>` specifier, with `nprocs` being the amount of parallel processes.\n",
     "\n",
     "## Compilation options\n",
-    "It is possible to specify optional features at the compilation stage of the model. These optional features can be activated by adding them as specifyers to the cmake command, for example, \n",
+    "It is possible to specify optional features at the compilation stage of the model. These optional features can be activated by adding them as specifyers to the cmake command. CMake options are specified as `-D<option>=<value>`. For example:\n",
     "```{code} shell\n",
     "cmake ../dales -DCMAKE_BUILD_TYPE=Debug\n",
     "```\n",
-    "will produce a debug build. The debug build is much slower than the release build but contains more error checks. A list of other compilation options is shown below\n",
-    "- Choose between single and double precision for the main prognostic fields and for the Poisson solver (from v4.4, default `64`)\n",
-    "    ```{code} shell\n",
-    "    -DFIELD_PRECISION=32\n",
-    "    ```\n",
-    "    ```{code} shell\n",
-    "    -DPOIS_PRECISION=32\n",
-    "    ```\n",
-    "- Optional alternative Poisson solvers (since v4.3, default `False`)\n",
-    "    ```{code} shell\n",
-    "    -DUSE_FFTW=True\n",
-    "    ```         \n",
-    "    ```{code} shell\n",
-    "    -DUSE_HYPRE=True\n",
-    "    ```         \n",
+    "will produce a debug build. The debug build is much slower than the release build but contains more error checks. A list of commonly used options is given below.\n",
+    "\n",
+    "| Option | Description | Allowed values | Default |\n",
+    "| ------ | ----------- | -------------- | ------- |\n",
+    "| `-DENABLE_FFTW` | Build with FFTW | On/Off | On |\n",
+    "| `-DENABLE_HYPRE` | Build with HYPRE | On/Off | Off |\n",
+    "| `-DENABLE_FP32_FIELDS` | Use single precision floating-point numbers for prognostic fields (momentum, temperature, et cetera) | On/Off | Off |\n",
+    "| `-DENABLE_FP32_POIS` | Use single precision floating-point numbers for the Poisson solver | On/Off | Off |\n",
+    "| `-DENABLE_ACC` | Build with GPU support through OpenACC | On/Off | Off |\n",
+    "\n",
     "```{caution}\n",
     "To use HYPRE or FFTW, the library needs to both be enabled at compilation and selected at runtime by setting the &SOLVER section of the namoptions input file. `FIX LINK: See Alternative Poisson solvers`.\n",
     "```\n",

From 956b9300703965897d007690ad88f13de338f2ab Mon Sep 17 00:00:00 2001
From: Caspar Jungbacker <caspar.jungbacker@outlook.com>
Date: Fri, 3 Jan 2025 10:13:21 +0100
Subject: [PATCH 2/2] Add GPU information

---
 book/running/gpu.ipynb | 239 -----------------------------------------
 book/running/gpu.md    |  76 +++++++++++++
 2 files changed, 76 insertions(+), 239 deletions(-)
 delete mode 100644 book/running/gpu.ipynb
 create mode 100644 book/running/gpu.md

diff --git a/book/running/gpu.ipynb b/book/running/gpu.ipynb
deleted file mode 100644
index c46ec44..0000000
--- a/book/running/gpu.ipynb
+++ /dev/null
@@ -1,239 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Setting up your system\n",
-    "\n",
-    "DALES can handle an arbitrary number of tracers. Examples of tracers are aerosols, hydrometeors, chemical species, et cetera. We distinguish two types of tracers:\n",
-    "\n",
-    "- Internal tracers. These are tracers defined internally, by the model itself. For example, the the bulkmicrophysics schemes define various tracers that represent quantities like rain and cloud droplets. The user does not have to provide any input for these tracers.\n",
-    "- User-defined tracers. These tracers need to be provided by the user in a NetCDF file. \n",
-    "\n",
-    "## Preparing the NetCDF file"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "First, we have to define a new tracer NetCDF file. This file has to be called `tracers.<iexpnr>.nc`, where `<iexpnr>` corresponds to what you have set as `iexpnr` in the `&RUN` section of your namelist."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import netCDF4 as nc\n",
-    "\n",
-    "tracers_nc = nc.Dataset(\"tracers.001.nc\", \"w\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Next, we'll create a dimension corresponding to the vertical levels. Tracers are defined at the full levels (cell centers), so the dimension we have to provide is `zt`. We don't have to provide the actual values of the levels, since those have already been read from `init.001.nc`. This example is based on the benchmark RICO case, which uses 126 vertical levels."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "<class 'netCDF4._netCDF4.Dataset'>\n",
-      "root group (NETCDF4 data model, file format HDF5):\n",
-      "    dimensions(sizes): zt(126)\n",
-      "    variables(dimensions): \n",
-      "    groups: \n"
-     ]
-    }
-   ],
-   "source": [
-    "height_dim = tracers_nc.createDimension(\"zt\", size=126)\n",
-    "print(tracers_nc)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Adding a tracer\n",
-    "Now we can define our tracer. For this example, we make a tracer that represents carbon dioxide (CO2): "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "<class 'netCDF4._netCDF4.Dataset'>\n",
-      "root group (NETCDF4 data model, file format HDF5):\n",
-      "    dimensions(sizes): zt(126)\n",
-      "    variables(dimensions): float64 co2(zt)\n",
-      "    groups: \n"
-     ]
-    }
-   ],
-   "source": [
-    "co2 = tracers_nc.createVariable(\"co2\", \"f8\", (\"zt\",))\n",
-    "print(tracers_nc)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "And we provide some (fake) initial profile:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import numpy as np\n",
-    "\n",
-    "co2[:] = np.ones(126) * 1e-6"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Tracer attributes\n",
-    "We can provide a number of attributes for our new tracer. These attributes either provide extra information for output, or enable extra schemes for the tracer.\n",
-    "\n",
-    "| Attribute    | Type    | Description |\n",
-    "| ------------ | ------- | ----------- |\n",
-    "| `long_name`  | String  | Full name of the tracer |\n",
-    "| `unit`       | String  | Unit of the tracer |\n",
-    "| `molar_mass` | Float   | Molar mass of the tracer |\n",
-    "| `lemis`      | Integer | Tracer is emitted (1=True, 0=False)|\n",
-    "| `lreact`     | Integer | Tracer is involved in chemistry |\n",
-    "| `ldep`       | Integer | Tracer is deposited |\n",
-    "| `lags`       | Integer | Tracer is involved in photosynthesis |\n",
-    "\n",
-    "Attributes can be set like this:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "metadata": {},
-   "outputs": [
-    {
-     "ename": "RuntimeError",
-     "evalue": "NetCDF: Not a valid ID",
-     "output_type": "error",
-     "traceback": [
-      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
-      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
-      "Cell \u001b[0;32mIn[13], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mco2\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mlong_name\u001b[49m \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mcarbon dioxide\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m      2\u001b[0m co2\u001b[38;5;241m.\u001b[39munit \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mkg/kg\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m      3\u001b[0m co2\u001b[38;5;241m.\u001b[39mmolar_mass \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m44.009\u001b[39m\u001b[38;5;124m\"\u001b[39m\n",
-      "File \u001b[0;32msrc/netCDF4/_netCDF4.pyx:4907\u001b[0m, in \u001b[0;36mnetCDF4._netCDF4.Variable.__setattr__\u001b[0;34m()\u001b[0m\n",
-      "File \u001b[0;32msrc/netCDF4/_netCDF4.pyx:4613\u001b[0m, in \u001b[0;36mnetCDF4._netCDF4.Variable.setncattr\u001b[0;34m()\u001b[0m\n",
-      "File \u001b[0;32msrc/netCDF4/_netCDF4.pyx:1775\u001b[0m, in \u001b[0;36mnetCDF4._netCDF4._set_att\u001b[0;34m()\u001b[0m\n",
-      "File \u001b[0;32msrc/netCDF4/_netCDF4.pyx:1700\u001b[0m, in \u001b[0;36mnetCDF4._netCDF4._get_format\u001b[0;34m()\u001b[0m\n",
-      "File \u001b[0;32msrc/netCDF4/_netCDF4.pyx:2113\u001b[0m, in \u001b[0;36mnetCDF4._netCDF4._ensure_nc_success\u001b[0;34m()\u001b[0m\n",
-      "\u001b[0;31mRuntimeError\u001b[0m: NetCDF: Not a valid ID"
-     ]
-    }
-   ],
-   "source": [
-    "co2.long_name = \"carbon dioxide\"\n",
-    "co2.unit = \"kg/kg\"\n",
-    "co2.molar_mass = \"44.009\"\n",
-    "co2.lemis = 1\n",
-    "co2.lags = 1\n",
-    "\n",
-    "print(co2)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Don't forget to close the NetCDF file:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "tracers_nc.close()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "We can check the end result with `ncdump`:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "netcdf tracers.001 {\n",
-      "dimensions:\n",
-      "\tzt = 126 ;\n",
-      "variables:\n",
-      "\tdouble co2(zt) ;\n",
-      "data:\n",
-      "\n",
-      " co2 = _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, \n",
-      "    _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, \n",
-      "    _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, \n",
-      "    _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, \n",
-      "    _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, \n",
-      "    _, _, _, _, _, _, _ ;\n",
-      "}\n"
-     ]
-    }
-   ],
-   "source": [
-    "!ncdump tracers.001.nc"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "dalesdocs",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.13.0"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/book/running/gpu.md b/book/running/gpu.md
new file mode 100644
index 0000000..8cbaa0a
--- /dev/null
+++ b/book/running/gpu.md
@@ -0,0 +1,76 @@
+# Using the GPU
+DALES now has the option to use graphics processing units (GPU's) to accelerate computations. 
+
+## Prerequisites
+
+- One or more NVIDIA GPU's
+- An OpenACC compatible compiler.
+- cuFFT
+- A GPU-aware MPI library
+
+The easiest and recommended way to get started on the GPU is by downloading the NVIDIA HPC SDK. This SDK contains the nvfortran compiler (which supports OpenACC), cuFFT, and a version of the OpenMPI library that is GPU-compatible. You can download it [here](https://developer.nvidia.com/hpc-sdk-downloads).
+
+## Compiling
+
+To compile DALES for the GPU, OpenACC has to be enabled when configuring the build with CMake. Here, it's important to make sure that CMake uses the correct compiler such that the compilation flags are setup correctly. Using the `which` command, check that the `mpif90` command points to the MPI Fortran compiler wrapper that is bundled with the HPC SDK:
+
+```{code} bash
+$ which mpif90
+/opt/nvidia/hpc_sdk/Linux_x86_64/2023/comm_libs/mpi/bin/mpif90
+```
+
+If the output looks similar, you're all set. The next step is to configure the GPU build using CMake:
+
+```{code} bash
+cd dales
+mkdir build
+cd build
+export FC=mpif90
+cmake -DENABLE_ACC=On ..
+make -j
+```
+
+## Problems with NetCDF-Fortran
+
+As mentioned before, the NVIDIA compiler might complain about your installation of NetCDF-Fortran. If this is the case, you will see an error mentioning something about "Old or corrupt module files" during the compilation phase. The solution is to compile the NetCDF-Fortran bindings yourself, using the NVIDIA compiler. In this section, we will briefly explain how to do this.
+
+You can download the NetCDF-Fortran source code from GitHub using `wget`:
+
+```{code} bash
+wget https://github.com/Unidata/netcdf-fortran/archive/refs/tags/v4.6.1.tar.gz
+tar xvf v4.6.1.tar.gz
+```
+
+Next, we need to export some variables such that NetCDF-Fortran compiler correctly. You can use `nc-config`, which comes bundled with NetCDF-C to automatically figure out some compiler flags:
+```{code} bash
+cd netcdf-fortran-4.6.1
+export CC=nvcc
+export FC=nvfortran
+export CPPFLAGS=$(nc-config --cflags)
+export LDFLAGS=$(nc-config --libs)
+```
+
+Finally, we can compile and install NetCDF-Fortran. Here, we use `~/software/netcdf-fortran` as the installation location:
+```{code} bash
+mkdir -p $HOME/software/netcdf-fortran
+./configure --prefix=$HOME/software/netcdf-fortran --disable-shared
+make install
+```
+
+To then compile DALES with this new library, we need to point CMake to its location:
+
+```{code} bash
+export FC=mpif90
+cmake -DENABLE_ACC=On -DNetCDF_Fortran_ROOT=$HOME/software/netcdf-fortran ..
+make -j
+```
+
+## Running
+
+If you have compiled DALES succesfully with OpenACC enabled, running it is not very different from the CPU version. DALES can be launched using the `mpirun` command:
+
+```{code} bash
+mpirun -np N <path-to-DALES> <path-to-namoptions>
+```
+
+where N should match the number of GPU's you want to use. 
\ No newline at end of file