diff --git a/docs/source/index.rst b/docs/source/index.rst index 898155b71..6609cc105 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -17,7 +17,7 @@ This is the next generation of the `IMPACT-Z `__ and get in touch with the community. +If you are starting using ImpactX, or if you have a user question, please pop in our `discussions page `__ and get in touch with the community. The `ImpactX GitHub repo `__ is the main communication platform. Have a look at the action icons on the top right of the web page: feel free to watch the repo if you want to receive updates, or to star the repo to support the project. diff --git a/docs/source/install/hpc.rst b/docs/source/install/hpc.rst index b397a0d9d..5a90fa202 100644 --- a/docs/source/install/hpc.rst +++ b/docs/source/install/hpc.rst @@ -24,17 +24,11 @@ HPC Systems .. toctree:: :maxdepth: 1 - hpc/cori hpc/perlmutter -.. hpc/summit -.. hpc/spock -.. hpc/juwels -.. hpc/lassen -.. hpc/quartz -.. hpc/lawrencium -.. hpc/ookami .. tip:: Your HPC system is not in the list? - `Open an issue `__ and together we can document it! + Our instructions are nearly identical to `installing WarpX, documented here `__. + + Also, please do not hesitate to `open an issue `__ and together we can document how to run on your preferred system! diff --git a/docs/source/install/hpc/cori.rst b/docs/source/install/hpc/cori.rst deleted file mode 100644 index 11813fa7d..000000000 --- a/docs/source/install/hpc/cori.rst +++ /dev/null @@ -1,315 +0,0 @@ -.. _building-cori: - -Cori (NERSC) -============ - -The `Cori cluster `_ is located at NERSC. - -If you are new to this system, please see the following resources: - -* `GPU nodes `__ - -* `Cori user guide `__ -* Batch system: `Slurm `__ -* `Jupyter service `__ -* `Production directories `__: - - * ``$SCRATCH``: per-user production directory (20TB) - * ``/global/cscratch1/sd/m3239``: shared production directory for users in the project ``m3239`` (50TB) - * ``/global/cfs/cdirs/m3239/``: community file system for users in the project ``m3239`` (100TB) - -Installation ------------- - -Use the following commands to download the ImpactX source code and switch to the correct branch: - -.. code-block:: bash - - git clone https://github.com/ECP-WarpX/impactx.git $HOME/src/impactx - -KNL -^^^ - -We use the following modules and environments on the system (``$HOME/knl_impactx.profile``). - -.. code-block:: bash - - module swap craype-haswell craype-mic-knl - module swap PrgEnv-intel PrgEnv-gnu - module load cmake/3.22.1 - module load cray-hdf5-parallel/1.10.5.2 - module load cray-fftw/3.3.8.10 - module load cray-python/3.9.7.1 - - export PKG_CONFIG_PATH=$FFTW_DIR/pkgconfig:$PKG_CONFIG_PATH - export CMAKE_PREFIX_PATH=$HOME/sw/knl/adios2-2.7.1-install:$CMAKE_PREFIX_PATH - - if [ -d "$HOME/sw/knl/venvs/impactx" ] - then - source $HOME/sw/knl/venvs/impactx/bin/activate - fi - - export CXXFLAGS="-march=knl" - export CFLAGS="-march=knl" - -For PICMI and Python workflows, also install a virtual environment: - -.. code-block:: bash - - # establish Python dependencies - python3 -m pip install --user --upgrade pip - python3 -m pip install --user virtualenv - - python3 -m venv $HOME/sw/knl/venvs/impactx - source $HOME/sw/knl/venvs/impactx/bin/activate - - python3 -m pip install --upgrade pip - MPICC="cc -shared" python3 -m pip install --upgrade --no-cache-dir -v mpi4py - python3 -m pip install --upgrade pytest - python3 -m pip install --upgrade -r $HOME/src/impactx/requirements.txt - python3 -m pip install --upgrade -r $HOME/src/impactx/examples/requirements.txt - -Haswell -^^^^^^^ - -We use the following modules and environments on the system (``$HOME/haswell_impactx.profile``). - -.. code-block:: bash - - module swap PrgEnv-intel PrgEnv-gnu - module load cmake/3.22.1 - module load cray-hdf5-parallel/1.10.5.2 - module load cray-fftw/3.3.8.10 - module load cray-python/3.9.7.1 - - export PKG_CONFIG_PATH=$FFTW_DIR/pkgconfig:$PKG_CONFIG_PATH - export CMAKE_PREFIX_PATH=$HOME/sw/haswell/adios2-2.7.1-install:$CMAKE_PREFIX_PATH - - if [ -d "$HOME/sw/haswell/venvs/impactx" ] - then - source $HOME/sw/haswell/venvs/impactx/bin/activate - fi - -For PICMI and Python workflows, also install a virtual environment: - -.. code-block:: bash - - # establish Python dependencies - python3 -m pip install --user --upgrade pip - python3 -m pip install --user virtualenv - - python3 -m venv $HOME/sw/haswell/venvs/impactx - source $HOME/sw/haswell/venvs/impactx/bin/activate - - python3 -m pip install --upgrade pip - MPICC="cc -shared" python3 -m pip install --upgrade --no-cache-dir -v mpi4py - python3 -m pip install --upgrade -r $HOME/src/impactx/requirements.txt - -GPU (V100) -^^^^^^^^^^ - -Cori provides a partition with `18 nodes that include V100 (16 GB) GPUs `__. -We use the following modules and environments on the system (``$HOME/gpu_impactx.profile``). - -.. code-block:: bash - - export proj="m1759" - - module purge - module load modules - module load cgpu - module load esslurm - module load gcc/8.3.0 cuda/11.4.0 cmake/3.22.1 - module load openmpi - - export CMAKE_PREFIX_PATH=$HOME/sw/cori_gpu/adios2-2.7.1-install:$CMAKE_PREFIX_PATH - - if [ -d "$HOME/sw/cori_gpu/venvs/impactx" ] - then - source $HOME/sw/cori_gpu/venvs/impactx/bin/activate - fi - - # compiler environment hints - export CC=$(which gcc) - export CXX=$(which g++) - export FC=$(which gfortran) - export CUDACXX=$(which nvcc) - export CUDAHOSTCXX=$(which g++) - - # optimize CUDA compilation for V100 - export AMREX_CUDA_ARCH=7.0 - - # allocate a GPU, e.g. to compile on - # 10 logical cores (5 physical), 1 GPU - function getNode() { - salloc -C gpu -N 1 -t 30 -c 10 --gres=gpu:1 -A $proj - } - -For PICMI and Python workflows, also install a virtual environment: - -.. code-block:: bash - - # establish Python dependencies - python3 -m pip install --user --upgrade pip - python3 -m pip install --user virtualenv - - python3 -m venv $HOME/sw/cori_gpu/venvs/impactx - source $HOME/sw/cori_gpu/venvs/impactx/bin/activate - - python3 -m pip install --upgrade pip - python3 -m pip install --upgrade --no-cache-dir -v mpi4py - python3 -m pip install --upgrade -r $HOME/src/impactx/requirements.txt - -Building ImpactX ----------------- - -We recommend to store the above lines in individual ``impactx.profile`` files, as suggested above. -If you want to run on either of the three partitions of Cori, open a new terminal, log into Cori and *source* the environment you want to work with: - -.. code-block:: bash - - # KNL: - source $HOME/knl_impactx.profile - - # Haswell: - #source $HOME/haswell_impactx.profile - - # GPU: - #source $HOME/gpu_impactx.profile - -.. warning:: - - Consider that all three Cori partitions are *incompatible*. - - Do not *source* multiple ``...impactx.profile`` files in the same terminal session. - Open a new terminal and log into Cori again, if you want to switch the targeted Cori partition. - - If you re-submit an already compiled simulation that you ran on another day or in another session, *make sure to source* the corresponding ``...impactx.profile`` again after login! - -Then, ``cd`` into the directory ``$HOME/src/impactx`` and use the following commands to compile: - -.. code-block:: bash - - cd $HOME/src/impactx - rm -rf build - - # append if you target GPUs: -DImpactX_COMPUTE=CUDA - cmake -S . -B build -DImpactX_OPENPMD=ON -DImpactX_DIMS=3 - cmake --build build -j 16 - - -.. _building-cori-tests: - -Testing -------- - -To run all tests (here on KNL), do: - -.. code-block:: bash - - srun -C knl -N 1 -t 30 -q debug ctest --test-dir build --output-on-failure - - -.. _running-cpp-cori: - -Running -------- - -Navigate (i.e. ``cd``) into one of the production directories (e.g. ``$SCRATCH``) before executing the instructions below. - -KNL -^^^ - -The batch script below can be used to run a ImpactX simulation on 2 KNL nodes on -the supercomputer Cori at NERSC. Replace descriptions between chevrons ``<>`` -by relevant values, for instance ```` could be ``laserWakefield``. - -Do not forget to first ``source $HOME/knl_impactx.profile`` if you have not done so already for this terminal session. - -For PICMI Python runs, the ```` has to read ``python3`` and the ```` is the path to your PICMI input script. - -.. literalinclude:: ../../../../etc/impactx/cori-nersc/batch_cori.sh - :language: bash - -To run a simulation, copy the lines above to a file ``batch_cori.sh`` and run - -.. code-block:: bash - - sbatch batch_cori.sh - -to submit the job. - -For a 3D simulation with a few (1-4) particles per cell using FDTD Maxwell -solver on Cori KNL for a well load-balanced problem (in our case laser -wakefield acceleration simulation in a boosted frame in the quasi-linear -regime), the following set of parameters provided good performance: - -* ``amr.max_grid_size=64`` and ``amr.blocking_factor=64`` so that the size of - each grid is fixed to ``64**3`` (we are not using load-balancing here). - -* **8 MPI ranks per KNL node**, with ``OMP_NUM_THREADS=8`` (that is 64 threads - per KNL node, i.e. 1 thread per physical core, and 4 cores left to the - system). - -* **2 grids per MPI**, *i.e.*, 16 grids per KNL node. - -Haswell -^^^^^^^ - -The batch script below can be used to run a ImpactX simulation on 1 `Haswell node `_ on the supercomputer Cori at NERSC. - -Do not forget to first ``source $HOME/haswell_impactx.profile`` if you have not done so already for this terminal session. - -.. literalinclude:: ../../../../etc/impactx/cori-nersc/batch_cori_haswell.sh - :language: bash - -To run a simulation, copy the lines above to a file ``batch_cori_haswell.sh`` and -run - -.. code-block:: bash - - sbatch batch_cori_haswell.sh - -to submit the job. - -For a 3D simulation with a few (1-4) particles per cell using FDTD Maxwell -solver on Cori Haswell for a well load-balanced problem (in our case laser -wakefield acceleration simulation in a boosted frame in the quasi-linear -regime), the following set of parameters provided good performance: - -* **4 MPI ranks per Haswell node** (2 MPI ranks per `Intel Xeon E5-2698 v3 `_), with ``OMP_NUM_THREADS=16`` (which uses `2x hyperthreading `_) - -GPU (V100) -^^^^^^^^^^ - -Do not forget to first ``source $HOME/gpu_impactx.profile`` if you have not done so already for this terminal session. - -Due to the limited amount of GPU development nodes, just request a single node with the above defined ``getNode`` function. -For single-node runs, try to run one grid per GPU. - -A multi-node batch script template can be found below: - -.. literalinclude:: ../../../../etc/impactx/cori-nersc/batch_cori_gpu.sh - :language: bash - - -.. _post-processing-cori: - -Post-Processing ---------------- - -For post-processing, most users use Python via NERSC's `Jupyter service `__ (`Docs `__). - -As a one-time preparatory setup, `create your own Conda environment as described in NERSC docs `__. -In this manual, we often use this ``conda create`` line over the officially documented one: - -.. code-block:: bash - - conda create -n myenv -c conda-forge python mamba ipykernel ipympl matplotlib numpy pandas yt openpmd-viewer openpmd-api h5py fast-histogram - -We then follow the `Customizing Kernels with a Helper Shell Script `__ section to finalize the setup of using this conda-environment as a custom Jupyter kernel. - -When opening a Jupyter notebook, just select the name you picked for your custom kernel on the top right of the notebook. - -Additional software can be installed later on, e.g., in a Jupyter cell using ``!mamba install -c conda-forge ...``. -Software that is not available via conda can be installed via ``!python -m pip install ...``. diff --git a/docs/source/install/hpc/perlmutter-nersc/install_cpu_dependencies.sh b/docs/source/install/hpc/perlmutter-nersc/install_cpu_dependencies.sh new file mode 100755 index 000000000..9ad833be8 --- /dev/null +++ b/docs/source/install/hpc/perlmutter-nersc/install_cpu_dependencies.sh @@ -0,0 +1,133 @@ +#!/bin/bash +# +# Copyright 2023 The ImpactX Community +# +# This file is part of ImpactX. +# +# Author: Axel Huebl +# License: BSD-3-Clause-LBNL + +# Exit on first error encountered ############################################# +# +set -eu -o pipefail + + +# Check: ###################################################################### +# +# Was perlmutter_cpu_impactx.profile sourced and configured correctly? +if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your perlmutter_cpu_impactx.profile file! Please edit its line 2 to continue!"; exit 1; fi + + +# Check $proj variable is correct and has a corresponding CFS directory ####### +# +if [ ! -d "${CFS}/${proj}/" ] +then + echo "WARNING: The directory ${CFS}/${proj}/ does not exist!" + echo "Is the \$proj environment variable of value \"$proj\" correctly set? " + echo "Please edit line 2 of your perlmutter_cpu_impactx.profile file to continue!" + exit +fi + + +# Remove old dependencies ##################################################### +# +SW_DIR="${CFS}/${proj}/${USER}/sw/perlmutter/cpu" +rm -rf ${SW_DIR} +mkdir -p ${SW_DIR} + +# remove common user mistakes in python, located in .local instead of a venv +python3 -m pip uninstall -qq -y impactx +python3 -m pip uninstall -qqq -y mpi4py 2>/dev/null || true + + +# General extra dependencies ################################################## +# + +# c-blosc (I/O compression) +if [ -d $HOME/src/c-blosc ] +then + cd $HOME/src/c-blosc + git fetch --prune + git checkout v1.21.1 + cd - +else + git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git $HOME/src/c-blosc +fi +rm -rf $HOME/src/c-blosc-pm-cpu-build +cmake -S $HOME/src/c-blosc -B $HOME/src/c-blosc-pm-cpu-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/c-blosc-1.21.1 +cmake --build $HOME/src/c-blosc-pm-cpu-build --target install --parallel 16 +rm -rf $HOME/src/c-blosc-pm-cpu-build + +# ADIOS2 +if [ -d $HOME/src/adios2 ] +then + cd $HOME/src/adios2 + git fetch --prune + git checkout v2.8.3 + cd - +else + git clone -b v2.8.3 https://github.com/ornladios/ADIOS2.git $HOME/src/adios2 +fi +rm -rf $HOME/src/adios2-pm-cpu-build +cmake -S $HOME/src/adios2 -B $HOME/src/adios2-pm-cpu-build -DADIOS2_USE_Blosc=ON -DADIOS2_USE_CUDA=OFF -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/adios2-2.8.3 +cmake --build $HOME/src/adios2-pm-cpu-build --target install -j 16 +rm -rf $HOME/src/adios2-pm-cpu-build + +# BLAS++ (for PSATD+RZ) +if [ -d $HOME/src/blaspp ] +then + cd $HOME/src/blaspp + git fetch --prune + git checkout master + git pull + cd - +else + git clone https://github.com/icl-utk-edu/blaspp.git $HOME/src/blaspp +fi +rm -rf $HOME/src/blaspp-pm-cpu-build +CXX=$(which CC) cmake -S $HOME/src/blaspp -B $HOME/src/blaspp-pm-cpu-build -Duse_openmp=ON -Dgpu_backend=OFF -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=${SW_DIR}/blaspp-master +cmake --build $HOME/src/blaspp-pm-cpu-build --target install --parallel 16 +rm -rf $HOME/src/blaspp-pm-cpu-build + +# LAPACK++ (for PSATD+RZ) +if [ -d $HOME/src/lapackpp ] +then + cd $HOME/src/lapackpp + git fetch --prune + git checkout master + git pull + cd - +else + git clone https://github.com/icl-utk-edu/lapackpp.git $HOME/src/lapackpp +fi +rm -rf $HOME/src/lapackpp-pm-cpu-build +CXX=$(which CC) CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S $HOME/src/lapackpp -B $HOME/src/lapackpp-pm-cpu-build -DCMAKE_CXX_STANDARD=17 -Dbuild_tests=OFF -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/lapackpp-master +cmake --build $HOME/src/lapackpp-pm-cpu-build --target install --parallel 16 +rm -rf $HOME/src/lapackpp-pm-cpu-build + + +# Python ###################################################################### +# +python3 -m pip install --upgrade pip +python3 -m pip install --upgrade virtualenv +python3 -m pip cache purge +rm -rf ${SW_DIR}/venvs/impactx +python3 -m venv ${SW_DIR}/venvs/impactx +source ${SW_DIR}/venvs/impactx/bin/activate +python3 -m pip install --upgrade pip +python3 -m pip install --upgrade build +python3 -m pip install --upgrade packaging +python3 -m pip install --upgrade wheel +python3 -m pip install --upgrade setuptools +python3 -m pip install --upgrade cython +python3 -m pip install --upgrade numpy +python3 -m pip install --upgrade pandas +python3 -m pip install --upgrade scipy +MPICC="cc -shared" python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py +python3 -m pip install --upgrade openpmd-api +python3 -m pip install --upgrade matplotlib +python3 -m pip install --upgrade yt +# install or update impactx dependencies such as picmistandard +python3 -m pip install --upgrade -r $HOME/src/impactx/requirements.txt +# optional: for libEnsemble +python3 -m pip install -r $HOME/src/impactx/Tools/LibEnsemble/requirements.txt diff --git a/docs/source/install/hpc/perlmutter-nersc/install_gpu_dependencies.sh b/docs/source/install/hpc/perlmutter-nersc/install_gpu_dependencies.sh new file mode 100755 index 000000000..95a2a801e --- /dev/null +++ b/docs/source/install/hpc/perlmutter-nersc/install_gpu_dependencies.sh @@ -0,0 +1,137 @@ +#!/bin/bash +# +# Copyright 2023 The ImpactX Community +# +# This file is part of ImpactX. +# +# Author: Axel Huebl +# License: BSD-3-Clause-LBNL + +# Exit on first error encountered ############################################# +# +set -eu -o pipefail + + +# Check: ###################################################################### +# +# Was perlmutter_gpu_impactx.profile sourced and configured correctly? +if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your perlmutter_gpu_impactx.profile file! Please edit its line 2 to continue!"; exit 1; fi + + +# Check $proj variable is correct and has a corresponding CFS directory ####### +# +if [ ! -d "${CFS}/${proj%_g}/" ] +then + echo "WARNING: The directory ${CFS}/${proj%_g}/ does not exist!" + echo "Is the \$proj environment variable of value \"$proj\" correctly set? " + echo "Please edit line 2 of your perlmutter_gpu_impactx.profile file to continue!" + exit +fi + + +# Remove old dependencies ##################################################### +# +SW_DIR="${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu" +rm -rf ${SW_DIR} +mkdir -p ${SW_DIR} + +# remove common user mistakes in python, located in .local instead of a venv +python3 -m pip uninstall -qq -y impactx +python3 -m pip uninstall -qqq -y mpi4py 2>/dev/null || true + + +# General extra dependencies ################################################## +# + +# c-blosc (I/O compression) +if [ -d $HOME/src/c-blosc ] +then + cd $HOME/src/c-blosc + git fetch --prune + git checkout v1.21.1 + cd - +else + git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git $HOME/src/c-blosc +fi +rm -rf $HOME/src/c-blosc-pm-gpu-build +cmake -S $HOME/src/c-blosc -B $HOME/src/c-blosc-pm-gpu-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/c-blosc-1.21.1 +cmake --build $HOME/src/c-blosc-pm-gpu-build --target install --parallel 16 +rm -rf $HOME/src/c-blosc-pm-gpu-build + +# ADIOS2 +if [ -d $HOME/src/adios2 ] +then + cd $HOME/src/adios2 + git fetch --prune + git checkout v2.8.3 + cd - +else + git clone -b v2.8.3 https://github.com/ornladios/ADIOS2.git $HOME/src/adios2 +fi +rm -rf $HOME/src/adios2-pm-gpu-build +cmake -S $HOME/src/adios2 -B $HOME/src/adios2-pm-gpu-build -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/adios2-2.8.3 +cmake --build $HOME/src/adios2-pm-gpu-build --target install -j 16 +rm -rf $HOME/src/adios2-pm-gpu-build + +# BLAS++ (for PSATD+RZ) +if [ -d $HOME/src/blaspp ] +then + cd $HOME/src/blaspp + git fetch --prune + git checkout master + git pull + cd - +else + git clone https://github.com/icl-utk-edu/blaspp.git $HOME/src/blaspp +fi +rm -rf $HOME/src/blaspp-pm-gpu-build +CXX=$(which CC) cmake -S $HOME/src/blaspp -B $HOME/src/blaspp-pm-gpu-build -Duse_openmp=OFF -Dgpu_backend=cuda -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=${SW_DIR}/blaspp-master +cmake --build $HOME/src/blaspp-pm-gpu-build --target install --parallel 16 +rm -rf $HOME/src/blaspp-pm-gpu-build + +# LAPACK++ (for PSATD+RZ) +if [ -d $HOME/src/lapackpp ] +then + cd $HOME/src/lapackpp + git fetch --prune + git checkout master + git pull + cd - +else + git clone https://github.com/icl-utk-edu/lapackpp.git $HOME/src/lapackpp +fi +rm -rf $HOME/src/lapackpp-pm-gpu-build +CXX=$(which CC) CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S $HOME/src/lapackpp -B $HOME/src/lapackpp-pm-gpu-build -DCMAKE_CXX_STANDARD=17 -Dbuild_tests=OFF -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/lapackpp-master +cmake --build $HOME/src/lapackpp-pm-gpu-build --target install --parallel 16 +rm -rf $HOME/src/lapackpp-pm-gpu-build + + +# Python ###################################################################### +# +python3 -m pip install --upgrade pip +python3 -m pip install --upgrade virtualenv +python3 -m pip cache purge +rm -rf ${SW_DIR}/venvs/impactx +python3 -m venv ${SW_DIR}/venvs/impactx +source ${SW_DIR}/venvs/impactx/bin/activate +python3 -m pip install --upgrade pip +python3 -m pip install --upgrade build +python3 -m pip install --upgrade packaging +python3 -m pip install --upgrade wheel +python3 -m pip install --upgrade setuptools +python3 -m pip install --upgrade cython +python3 -m pip install --upgrade numpy +python3 -m pip install --upgrade pandas +python3 -m pip install --upgrade scipy +MPICC="cc -target-accel=nvidia80 -shared" python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py +python3 -m pip install --upgrade openpmd-api +python3 -m pip install --upgrade matplotlib +python3 -m pip install --upgrade yt +# install or update impactx dependencies such as picmistandard +python3 -m pip install --upgrade -r $HOME/src/impactx/requirements.txt +python3 -m pip install cupy-cuda11x # CUDA 11.7 compatible wheel +# optional: for libEnsemble +python3 -m pip install -r $HOME/src/impactx/Tools/LibEnsemble/requirements.txt +# optional: for optimas (based on libEnsemble & ax->botorch->gpytorch->pytorch) +python3 -m pip install --upgrade torch # CUDA 11.7 compatible wheel +python3 -m pip install -r $HOME/src/impactx/Tools/optimas/requirements.txt diff --git a/docs/source/install/hpc/perlmutter-nersc/perlmutter_cpu.sbatch b/docs/source/install/hpc/perlmutter-nersc/perlmutter_cpu.sbatch new file mode 100644 index 000000000..1be806ce4 --- /dev/null +++ b/docs/source/install/hpc/perlmutter-nersc/perlmutter_cpu.sbatch @@ -0,0 +1,39 @@ +#!/bin/bash -l + +# Copyright 2021-2023 ImpactX +# +# This file is part of ImpactX. +# +# Authors: Axel Huebl +# License: BSD-3-Clause-LBNL + +#SBATCH -t 00:10:00 +#SBATCH -N 2 +#SBATCH -J ImpactX +#SBATCH -A +#SBATCH -q regular +#SBATCH -C cpu +#SBATCH --ntasks-per-node=16 +#SBATCH --exclusive +#SBATCH -o ImpactX.o%j +#SBATCH -e ImpactX.e%j + +# executable & inputs file or python interpreter & PICMI script here +EXE=./impactx +INPUTS=inputs_small + +# each CPU node on Perlmutter (NERSC) has 64 hardware cores with +# 2x Hyperthreading/SMP +# https://en.wikichip.org/wiki/amd/epyc/7763 +# https://www.amd.com/en/products/cpu/amd-epyc-7763 +# Each CPU is made up of 8 chiplets, each sharing 32MB L3 cache. +# This will be our MPI rank assignment (2x8 is 16 ranks/node). + +# threads for OpenMP and threaded compressors per MPI rank +export SRUN_CPUS_PER_TASK=16 # 8 cores per chiplet, 2x SMP +export OMP_PLACES=threads +export OMP_PROC_BIND=spread + +srun --cpu-bind=cores \ + ${EXE} ${INPUTS} \ + > output.txt diff --git a/docs/source/install/hpc/perlmutter-nersc/perlmutter_cpu_impactx.profile.example b/docs/source/install/hpc/perlmutter-nersc/perlmutter_cpu_impactx.profile.example new file mode 100644 index 000000000..a4aa62f82 --- /dev/null +++ b/docs/source/install/hpc/perlmutter-nersc/perlmutter_cpu_impactx.profile.example @@ -0,0 +1,56 @@ +# please set your project account +export proj="" # change me! + +# remembers the location of this script +export MY_PROFILE=$(cd $(dirname $BASH_SOURCE) && pwd)"/"$(basename $BASH_SOURCE) +if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your $MY_PROFILE file! Please edit its line 2 to continue!"; return; fi + +# required dependencies +module load cpu +module load cmake/3.22.0 +module load cray-fftw/3.3.10.3 + +# optional: for QED support with detailed tables +export BOOST_ROOT=/global/common/software/spackecp/perlmutter/e4s-23.05/default/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/boost-1.82.0-ow5r5qrgslcwu33grygouajmuluzuzv3 + +# optional: for openPMD and PSATD+RZ support +module load cray-hdf5-parallel/1.12.2.7 +export CMAKE_PREFIX_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/c-blosc-1.21.1:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/adios2-2.8.3:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/blaspp-master:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/lapackpp-master:$CMAKE_PREFIX_PATH + +export LD_LIBRARY_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/c-blosc-1.21.1/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/adios2-2.8.3/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/blaspp-master/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/lapackpp-master/lib64:$LD_LIBRARY_PATH + +export PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/adios2-2.8.3/bin:${PATH} + +# optional: CCache +export PATH=/global/common/software/spackecp/perlmutter/e4s-23.05/default/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/ccache-4.8-eqk2d3bipbpkgwxq7ujlp6mckwal4dwz/bin:$PATH + +# optional: for Python bindings or libEnsemble +module load cray-python/3.9.13.1 + +if [ -d "${CFS}/${proj}/${USER}/sw/perlmutter/cpu/venvs/impactx" ] +then + source ${CFS}/${proj}/${USER}/sw/perlmutter/cpu/venvs/impactx/bin/activate +fi + +# an alias to request an interactive batch node for one hour +# for parallel execution, start on the batch node: srun +alias getNode="salloc --nodes 1 --qos interactive --time 01:00:00 --constraint cpu --account=$proj" +# an alias to run a command on a batch node for up to 30min +# usage: runNode +alias runNode="srun --nodes 1 --qos interactive --time 01:00:00 --constraint cpu $proj" + +# optimize CPU microarchitecture for AMD EPYC 3rd Gen (Milan/Zen3) +# note: the cc/CC/ftn wrappers below add those +export CXXFLAGS="-march=znver3" +export CFLAGS="-march=znver3" + +# compiler environment hints +export CC=cc +export CXX=CC +export FC=ftn diff --git a/docs/source/install/hpc/perlmutter-nersc/perlmutter_gpu.sbatch b/docs/source/install/hpc/perlmutter-nersc/perlmutter_gpu.sbatch new file mode 100644 index 000000000..a767d4908 --- /dev/null +++ b/docs/source/install/hpc/perlmutter-nersc/perlmutter_gpu.sbatch @@ -0,0 +1,46 @@ +#!/bin/bash -l + +# Copyright 2021-2023 Axel Huebl, Kevin Gott +# +# This file is part of ImpactX. +# +# License: BSD-3-Clause-LBNL + +#SBATCH -t 00:10:00 +#SBATCH -N 2 +#SBATCH -J ImpactX +# note: must end on _g +#SBATCH -A +#SBATCH -q regular +# A100 40GB (most nodes) +#SBATCH -C gpu +# A100 80GB (256 nodes) +#S BATCH -C gpu&hbm80g +#SBATCH --exclusive +# ideally single:1, but NERSC cgroups issue +#SBATCH --gpu-bind=none +#SBATCH --ntasks-per-node=4 +#SBATCH --gpus-per-node=4 +#SBATCH -o ImpactX.o%j +#SBATCH -e ImpactX.e%j + +# executable & inputs file or python interpreter & PICMI script here +EXE=./impactx +INPUTS=inputs + +# pin to closest NIC to GPU +export MPICH_OFI_NIC_POLICY=GPU + +# threads for OpenMP and threaded compressors per MPI rank +# note: 16 avoids hyperthreading (32 virtual cores, 16 physical) +export SRUN_CPUS_PER_TASK=16 + +# GPU-aware MPI optimizations +GPU_AWARE_MPI="amrex.use_gpu_aware_mpi=1" + +# CUDA visible devices are ordered inverse to local task IDs +# Reference: nvidia-smi topo -m +srun --cpu-bind=cores bash -c " + export CUDA_VISIBLE_DEVICES=\$((3-SLURM_LOCALID)); + ${EXE} ${INPUTS} ${GPU_AWARE_MPI}" \ + > output.txt diff --git a/docs/source/install/hpc/perlmutter-nersc/perlmutter_gpu_impactx.profile.example b/docs/source/install/hpc/perlmutter-nersc/perlmutter_gpu_impactx.profile.example new file mode 100644 index 000000000..14d241b34 --- /dev/null +++ b/docs/source/install/hpc/perlmutter-nersc/perlmutter_gpu_impactx.profile.example @@ -0,0 +1,62 @@ +# please set your project account +export proj="" # change me! GPU projects must end in "..._g" + +# remembers the location of this script +export MY_PROFILE=$(cd $(dirname $BASH_SOURCE) && pwd)"/"$(basename $BASH_SOURCE) +if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your $MY_PROFILE file! Please edit its line 2 to continue!"; return; fi + +# required dependencies +module load cmake/3.22.0 + +# optional: for QED support with detailed tables +export BOOST_ROOT=/global/common/software/spackecp/perlmutter/e4s-23.05/default/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/boost-1.82.0-ow5r5qrgslcwu33grygouajmuluzuzv3 + +# optional: for openPMD and PSATD+RZ support +module load cray-hdf5-parallel/1.12.2.7 +export CMAKE_PREFIX_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/c-blosc-1.21.1:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/adios2-2.8.3:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/blaspp-master:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/lapackpp-master:$CMAKE_PREFIX_PATH + +export LD_LIBRARY_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/c-blosc-1.21.1/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/adios2-2.8.3/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/blaspp-master/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/lapackpp-master/lib64:$LD_LIBRARY_PATH + +export PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/adios2-2.8.3/bin:${PATH} + +# optional: CCache +export PATH=/global/common/software/spackecp/perlmutter/e4s-23.05/default/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/ccache-4.8-eqk2d3bipbpkgwxq7ujlp6mckwal4dwz/bin:$PATH + +# optional: for Python bindings or libEnsemble +module load cray-python/3.9.13.1 + +if [ -d "${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/venvs/impactx" ] +then + source ${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/venvs/impactx/bin/activate +fi + +# an alias to request an interactive batch node for one hour +# for parallel execution, start on the batch node: srun +alias getNode="salloc -N 1 --ntasks-per-node=4 -t 1:00:00 -q interactive -C gpu --gpu-bind=single:1 -c 32 -G 4 -A $proj" +# an alias to run a command on a batch node for up to 30min +# usage: runNode +alias runNode="srun -N 1 --ntasks-per-node=4 -t 0:30:00 -q interactive -C gpu --gpu-bind=single:1 -c 32 -G 4 -A $proj" + +# necessary to use CUDA-Aware MPI and run a job +export CRAY_ACCEL_TARGET=nvidia80 + +# optimize CUDA compilation for A100 +export AMREX_CUDA_ARCH=8.0 + +# optimize CPU microarchitecture for AMD EPYC 3rd Gen (Milan/Zen3) +# note: the cc/CC/ftn wrappers below add those +export CXXFLAGS="-march=znver3" +export CFLAGS="-march=znver3" + +# compiler environment hints +export CC=cc +export CXX=CC +export FC=ftn +export CUDACXX=$(which nvcc) +export CUDAHOSTCXX=CC diff --git a/docs/source/install/hpc/perlmutter.rst b/docs/source/install/hpc/perlmutter.rst index bf0b51f64..5f61935cf 100644 --- a/docs/source/install/hpc/perlmutter.rst +++ b/docs/source/install/hpc/perlmutter.rst @@ -3,126 +3,224 @@ Perlmutter (NERSC) ================== -.. warning:: +The `Perlmutter cluster `_ is located at NERSC. - Perlmutter is still in acceptance testing. - This page documents our internal testing workflow only. -The `Perlmutter cluster `_ is located at NERSC. +Introduction +------------ -If you are new to this system, please see the following resources: +If you are new to this system, **please see the following resources**: * `NERSC user guide `__ * Batch system: `Slurm `__ -* `Jupyter service `__ -* `Production directories `__: +* `Jupyter service `__ (`documentation `__) +* `Filesystems `__: - * ``$PSCRATCH``: per-user production directory (TB) - * ``/global/cscratch1/sd/m3239``: shared production directory for users in the project ``m3239`` (50TB) - * ``/global/cfs/cdirs/m3239/``: community file system for users in the project ``m3239`` (100TB) + * ``$HOME``: per-user directory, use only for inputs, source and scripts; backed up (40GB) + * ``${CFS}/m3239/``: `community file system `__ for users in the project ``m3239`` (or equivalent); moderate performance (20TB default) + * ``$PSCRATCH``: per-user `production directory `__; very fast for parallel jobs; purged every 8 weeks (20TB default) -Installation ------------- +.. _building-perlmutter-preparation: + +Preparation +----------- -Use the following commands to download the ImpactX source code and switch to the correct branch: +Use the following commands to download the ImpactX source code: .. code-block:: bash git clone https://github.com/ECP-WarpX/impactx.git $HOME/src/impactx -We use the following modules and environments on the system (``$HOME/perlmutter_impactx.profile``). +On Perlmutter, you can run either on GPU nodes with fast A100 GPUs (recommended) or CPU nodes. -.. code-block:: bash +.. tab-set:: - # please set your project account - export proj= # LBNL/AMP: m3906_g + .. tab-item:: A100 GPUs - # required dependencies - module load cmake/3.22.0 - module load cray-hdf5-parallel/1.12.2.1 + We use system software modules, add environment hints and further dependencies via the file ``$HOME/perlmutter_gpu_impactx.profile``. + Create it now: - # optional: just an additional text editor - #module load nano # TODO: request from support + .. code-block:: bash - # optional: CCache for faster rebuilds - export PATH=/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/ccache-4.5.1-ybl7xefvggn6hov4dsdxxnztji74tolj/bin:$PATH + cp $HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/perlmutter_gpu_impactx.profile.example $HOME/perlmutter_gpu_impactx.profile - # Python - module load cray-python/3.9.13.1 - if [ -d "$HOME/sw/perlmutter/venvs/impactx" ] - then - source $HOME/sw/perlmutter/venvs/impactx/bin/activate - fi + .. dropdown:: Script Details + :color: light + :icon: info + :animate: fade-in-slide-down - # an alias to request an interactive batch node for one hour - # for parallel execution, start on the batch node: srun - alias getNode="salloc -N 1 --ntasks-per-gpu=1 -t 1:00:00 -q interactive -C gpu --gpu-bind=single:1 -c 32 -G 4 -A $proj" - # an alias to run a command on a batch node for up to 30min - # usage: runNode - alias runNode="srun -N 1 --ntasks-per-gpu=1 -t 0:30:00 -q interactive -C gpu --gpu-bind=single:1 -c 32 -G 4 -A $proj" + .. literalinclude:: perlmutter-nersc/perlmutter_gpu_impactx.profile.example + :language: bash - # GPU-aware MPI - export MPICH_GPU_SUPPORT_ENABLED=1 + Edit the 2nd line of this script, which sets the ``export proj=""`` variable. + Perlmutter GPU projects must end in ``..._g``. + For example, if you are member of the project ``m3239``, then run ``nano $HOME/perlmutter_gpu_impactx.profile`` and edit line 2 to read: - # necessary to use CUDA-Aware MPI and run a job - export CRAY_ACCEL_TARGET=nvidia80 + .. code-block:: bash - # optimize CUDA compilation for A100 - export AMREX_CUDA_ARCH=8.0 + export proj="m3239_g" - # compiler environment hints - export CC=cc - export CXX=CC - export FC=ftn - export CUDACXX=$(which nvcc) - export CUDAHOSTCXX=CC + Exit the ``nano`` editor with ``Ctrl`` + ``O`` (save) and then ``Ctrl`` + ``X`` (exit). + .. important:: -We recommend to store the above lines in a file, such as ``$HOME/perlmutter_impactx.profile``, and load it into your shell after a login: + Now, and as the first step on future logins to Perlmutter, activate these environment settings: -.. code-block:: bash + .. code-block:: bash - source $HOME/perlmutter_impactx.profile + source $HOME/perlmutter_gpu_impactx.profile -For Python workflows & tests, also install a virtual environment: + Finally, since Perlmutter does not yet provide software modules for some of our dependencies, install them once: -.. code-block:: bash + .. code-block:: bash + + bash $HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/install_gpu_dependencies.sh + source ${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/venvs/impactx/bin/activate + + .. dropdown:: Script Details + :color: light + :icon: info + :animate: fade-in-slide-down + + .. literalinclude:: perlmutter-nersc/install_gpu_dependencies.sh + :language: bash + + + .. tab-item:: CPU Nodes + + We use system software modules, add environment hints and further dependencies via the file ``$HOME/perlmutter_cpu_impactx.profile``. + Create it now: + + .. code-block:: bash + + cp $HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/perlmutter_cpu_impactx.profile.example $HOME/perlmutter_cpu_impactx.profile + + .. dropdown:: Script Details + :color: light + :icon: info + :animate: fade-in-slide-down + + .. literalinclude:: perlmutter-nersc/perlmutter_cpu_impactx.profile.example + :language: bash + + Edit the 2nd line of this script, which sets the ``export proj=""`` variable. + For example, if you are member of the project ``m3239``, then run ``nano $HOME/perlmutter_cpu_impactx.profile`` and edit line 2 to read: + + .. code-block:: bash + + export proj="m3239" + + Exit the ``nano`` editor with ``Ctrl`` + ``O`` (save) and then ``Ctrl`` + ``X`` (exit). + + .. important:: + + Now, and as the first step on future logins to Perlmutter, activate these environment settings: + + .. code-block:: bash + + source $HOME/perlmutter_cpu_impactx.profile + + Finally, since Perlmutter does not yet provide software modules for some of our dependencies, install them once: + + .. code-block:: bash + + bash $HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/install_cpu_dependencies.sh + source ${CFS}/${proj}/${USER}/sw/perlmutter/cpu/venvs/impactx/bin/activate + + .. dropdown:: Script Details + :color: light + :icon: info + :animate: fade-in-slide-down + + .. literalinclude:: perlmutter-nersc/install_cpu_dependencies.sh + :language: bash + + +.. _building-perlmutter-compilation: + +Compilation +----------- + +Use the following :ref:`cmake commands ` to compile the application executable: + +.. tab-set:: - # establish Python dependencies - python3 -m pip install --user --upgrade pip - python3 -m pip install --user virtualenv + .. tab-item:: A100 GPUs - python3 -m venv $HOME/sw/perlmutter/venvs/impactx - source $HOME/sw/perlmutter/venvs/impactx/bin/activate + .. code-block:: bash - python3 -m pip install --upgrade pip - MPICC="cc -target-accel=nvidia80 -shared" python3 -m pip install --upgrade --no-cache-dir -v mpi4py - python3 -m pip install --upgrade pytest - python3 -m pip install --upgrade -r $HOME/src/impactx/requirements.txt - python3 -m pip install --upgrade -r $HOME/src/impactx/examples/requirements.txt + cd $HOME/src/impactx + rm -rf build_pm_gpu -Then, ``cd`` into the directory ``$HOME/src/impactx`` and use the following commands to compile: + cmake -S . -B build_pm_gpu -DImpactX_COMPUTE=CUDA -DImpactX_PSATD=ON -DImpactX_QED_TABLE_GEN=ON -DImpactX_DIMS="1;2;RZ;3" + cmake --build build_pm_gpu -j 16 + + The ImpactX application executables are now in ``$HOME/src/impactx/build_pm_gpu/bin/``. + Additionally, the following commands will install ImpactX as a Python module: + + .. code-block:: bash + + cd $HOME/src/impactx + rm -rf build_pm_gpu_py + + cmake -S . -B build_pm_gpu_py -DImpactX_COMPUTE=CUDA -DImpactX_PSATD=ON -DImpactX_QED_TABLE_GEN=ON -DImpactX_APP=OFF -DImpactX_PYTHON=ON -DImpactX_DIMS="1;2;RZ;3" + cmake --build build_pm_gpu_py -j 16 --target pip_install + + .. tab-item:: CPU Nodes + + .. code-block:: bash + + cd $HOME/src/impactx + rm -rf build_pm_cpu + + cmake -S . -B build_pm_cpu -DImpactX_COMPUTE=OMP -DImpactX_PSATD=ON -DImpactX_QED_TABLE_GEN=ON -DImpactX_DIMS="1;2;RZ;3" + cmake --build build_pm_cpu -j 16 + + The ImpactX application executables are now in ``$HOME/src/impactx/build_pm_cpu/bin/``. + Additionally, the following commands will install ImpactX as a Python module: + + .. code-block:: bash + + rm -rf build_pm_cpu_py + + cmake -S . -B build_pm_cpu_py -DImpactX_COMPUTE=OMP -DImpactX_PSATD=ON -DImpactX_QED_TABLE_GEN=ON -DImpactX_APP=OFF -DImpactX_PYTHON=ON -DImpactX_DIMS="1;2;RZ;3" + cmake --build build_pm_cpu_py -j 16 --target pip_install + +Now, you can :ref:`submit Perlmutter compute jobs ` for ImpactX :ref:`Python (PICMI) scripts ` (:ref:`example scripts `). +Or, you can use the ImpactX executables to submit Perlmutter jobs (:ref:`example inputs `). +For executables, you can reference their location in your :ref:`job script ` or copy them to a location in ``$PSCRATCH``. + + +.. _building-perlmutter-update: + +Update ImpactX & Dependencies +----------------------------- + +If you already installed ImpactX in the past and want to update it, start by getting the latest source code: .. code-block:: bash cd $HOME/src/impactx - rm -rf build - cmake -S . -B build_perlmutter -DImpactX_COMPUTE=CUDA -DImpactX_PYTHON=ON - cmake --build build_perlmutter -j 32 + # read the output of this command - does it look ok? + git status -To run all tests, do: + # get the latest ImpactX source code + git fetch + git pull -.. code-block:: bash + # read the output of these commands - do they look ok? + git status + git log # press q to exit - # work on an interactive node - salloc -N 1 --ntasks-per-node=4 -t 1:00:00 -C gpu --gpu-bind=single:1 -c 32 -G 4 --qos=interactive -A m3906_g +And, if needed, - # test - ctest --test-dir build_perlmutter -E AMReX --output-on-failure +- :ref:`update the perlmutter_gpu_impactx.profile or perlmutter_cpu_impactx files `, +- log out and into the system, activate the now updated environment profile as usual, +- :ref:`execute the dependency install scripts `. -The general :ref:`cmake compile-time options ` apply as usual. +As a last step, clean the build directory ``rm -rf $HOME/src/impactx/build_pm_*`` and rebuild ImpactX. .. _running-cpp-perlmutter: @@ -130,26 +228,42 @@ The general :ref:`cmake compile-time options ` apply as usual. Running ------- -.. _running-cpp-perlmutter-A100-GPUs: +.. tab-set:: -A100 GPUs (40 GB) -^^^^^^^^^^^^^^^^^ + .. tab-item:: A100 (40GB) GPUs -The batch script below can be used to run a ImpactX simulation on multiple nodes (change ``-N`` accordingly) on the supercomputer Perlmutter at NERSC. -Replace descriptions between chevrons ``<>`` by relevant values, for instance ```` could be ``plasma_mirror_inputs``. -Note that we run one MPI rank per GPU. + The batch script below can be used to run a ImpactX simulation on multiple nodes (change ``-N`` accordingly) on the supercomputer Perlmutter at NERSC. + This partition as up to `1536 nodes `__. + Replace descriptions between chevrons ``<>`` by relevant values, for instance ```` could be ``plasma_mirror_inputs``. + Note that we run one MPI rank per GPU. -.. literalinclude:: ../../../../etc/impactx/perlmutter-nersc/batch_perlmutter.sh - :language: bash + .. literalinclude:: perlmutter-nersc/perlmutter_gpu.sbatch + :language: bash + :caption: You can copy this file from ``$HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/perlmutter_gpu.sbatch``. -To run a simulation, copy the lines above to a file ``batch_perlmutter.sh`` and run + To run a simulation, copy the lines above to a file ``perlmutter_gpu.sbatch`` and run -.. code-block:: bash + .. code-block:: bash + + sbatch perlmutter_gpu.sbatch + + to submit the job. + + + .. tab-item:: A100 (80GB) GPUs - sbatch batch_perlmutter.sh + Perlmutter has `256 nodes `__ that provide 80 GB HBM per A100 GPU. + In the A100 (40GB) batch script, replace ``-C gpu`` with ``-C gpu&hbm80g`` to use these large-memory GPUs. -to submit the job. + + .. tab-item:: CPU Nodes + + The Perlmutter CPU partition as up to `3072 nodes `__, each with 2x AMD EPYC 7763 CPUs. + + .. literalinclude:: perlmutter-nersc/perlmutter_cpu.sbatch + :language: bash + :caption: You can copy this file from ``$HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/perlmutter_cpu.sbatch``. .. _post-processing-perlmutter: @@ -157,9 +271,36 @@ to submit the job. Post-Processing --------------- -For post-processing, most users use Python via NERSC's `Jupyter service `__ (`Docs `__). +For post-processing, most users use Python via NERSC's `Jupyter service `__ (`documentation `__). + +As a one-time preparatory setup, log into Perlmutter via SSH and do *not* source the ImpactX profile script above. +Create your own Conda environment and `Jupyter kernel `__ for post-processing: + +.. code-block:: bash + + module load python + + conda config --set auto_activate_base false + + # create conda environment + rm -rf $HOME/.conda/envs/impactx-pm-postproc + conda create --yes -n impactx-pm-postproc -c conda-forge mamba conda-libmamba-solver + conda activate impactx-pm-postproc + conda config --set solver libmamba + mamba install --yes -c conda-forge python ipykernel ipympl matplotlib numpy pandas yt openpmd-viewer openpmd-api h5py fast-histogram dask dask-jobqueue pyarrow + + # create Jupyter kernel + rm -rf $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/ + python -m ipykernel install --user --name impactx-pm-postproc --display-name ImpactX-PM-PostProcessing + echo -e '#!/bin/bash\nmodule load python\nsource activate impactx-pm-postproc\nexec "$@"' > $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/kernel-helper.sh + chmod a+rx $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/kernel-helper.sh + KERNEL_STR=$(jq '.argv |= ["{resource_dir}/kernel-helper.sh"] + .' $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/kernel.json | jq '.argv[1] = "python"') + echo ${KERNEL_STR} | jq > $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/kernel.json + + exit + -Please follow the same guidance as for :ref:`NERSC Cori post-processing `. +When opening a Jupyter notebook on `https://jupyter.nersc.gov `__, just select ``ImpactX-PM-PostProcessing`` from the list of available kernels on the top right of the notebook. -The Perlmutter ``$PSCRATCH`` filesystem is currently not yet available on Jupyter. -Thus, store or copy your data to Cori's ``$SCRATCH`` or use the Community FileSystem (CFS) for now. +Additional software can be installed later on, e.g., in a Jupyter cell using ``!mamba install -y -c conda-forge ...``. +Software that is not available via conda can be installed via ``!python -m pip install ...``. diff --git a/docs/source/usage/how_to_run.rst b/docs/source/usage/how_to_run.rst index 03cfa4acf..5f83805e4 100644 --- a/docs/source/usage/how_to_run.rst +++ b/docs/source/usage/how_to_run.rst @@ -77,7 +77,7 @@ Note that the actual executable might have a longer name, depending on build opt We used the copied executable in the current directory (``./``); if you installed with a package manager, skip the ``./`` because ImpactX is in your ``PATH``. -On an :ref:`HPC system `, you would instead submit the :ref:`job script ` at this point, e.g. ``sbatch `` (SLURM on Cori/NERSC) or ``bsub `` (LSF on Summit/OLCF). +On an :ref:`HPC system `, you would instead submit the :ref:`job script ` at this point, e.g. ``sbatch `` (SLURM on Perlmutter/NERSC) or ``bsub `` (LSF on Summit/OLCF). .. tip:: @@ -86,4 +86,4 @@ On an :ref:`HPC system `, you would instead submit the :ref:`job sc .. code-block:: bash - mpirun -np 4 ./impactx max_step=10 impactx.numprocs=1 2 2 + mpirun -np 4 ./impactx max_step=10 amr.n_cell=64 64 128 diff --git a/examples/cyclotron/README.rst b/examples/cyclotron/README.rst index e265976d3..fa89cd793 100644 --- a/examples/cyclotron/README.rst +++ b/examples/cyclotron/README.rst @@ -47,20 +47,6 @@ We run the following script to analyze correctness: Visualize --------- -You can run the following script to visualize the beam evolution over time: +.. note:: -.. dropdown:: Script ``plot_cyclotron.py`` - - .. literalinclude:: plot_cyclotron.py - :language: python3 - :caption: You can copy this file from ``examples/cyclotron/plot_cyclotron.py``. - -.. figure:: https://user-images.githubusercontent.com/1353258/180287840-8561f6fd-278f-4856-abd8-04fbdb78c8ff.png - :alt: focusing, defocusing and preserved emittane in our cyclotron cell benchmark. - - cyclotron transversal beam width and emittance evolution - -.. figure:: https://user-images.githubusercontent.com/1353258/180287845-eb0210a7-2500-4aa9-844c-67fb094329d3.png - :alt: focusing, defocusing and phase space rotation in our cyclotron cell benchmark. - - cyclotron transversal beam width and phase space evolution + TODO :)