Merge branch 'development' into implicit_picard

dpgrote · Dec 18, 2023 · 927cef4 · 927cef4
2 parents 6733012 + 244ec1c
commit 927cef4
Show file tree

Hide file tree

Showing 39 changed files with 1,893 additions and 2,408 deletions.
diff --git a/.github/workflows/dependencies/hip.sh b/.github/workflows/dependencies/hip.sh
@@ -41,7 +41,8 @@ sudo apt-get install -y --no-install-recommends \
     rocm-dev        \
     rocfft-dev      \
     rocprim-dev     \
-    rocrand-dev
+    rocrand-dev     \
+    hiprand-dev
 
 # ccache
 $(dirname "$0")/ccache.sh

diff --git a/.gitignore b/.gitignore
@@ -38,7 +38,7 @@ Docs/warpx-doxygen-web.tag.xml
 Docs/openpmd-api-doxygen-web.tag.xml
 Docs/doxyhtml/
 Docs/doxyxml/
-Docs/source/_static/
+Docs/source/_static/doxyhtml
 
 ####################
 # Package Managers #

diff --git a/Docs/source/_static/custom.css b/Docs/source/_static/custom.css
@@ -0,0 +1,6 @@
+.eqno {
+    /* As of 2023 Dec, sphinx_rtd_theme has a bug where equation numbers appear above the equations instead of on the right */
+    /* The following is a make-shift fix, but comes at the cost of "making the header link invisible," though I'm not sure what that means */
+    /* Copied from https://github.com/readthedocs/sphinx_rtd_theme/issues/301#issuecomment-1205755904 */
+    float: right;
+}
diff --git a/Docs/source/conf.py b/Docs/source/conf.py
@@ -134,6 +134,11 @@ def __init__(self, *args, **kwargs):
 html_theme = 'sphinx_rtd_theme'
 
 numfig = True
+math_eqref_format = "{number}"
+numfig_format = {'figure': 'Fig. %s',
+                 'table': 'Table %s',
+                 'code-block': 'Listing %s',
+                 'section': 'Section %s'}
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
@@ -146,6 +151,9 @@ def __init__(self, *args, **kwargs):
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ['_static']
 
+html_css_files = [
+    'custom.css',
+]
 
 # -- Options for HTMLHelp output ------------------------------------------
 

diff --git a/Docs/source/dataanalysis/sensei.rst b/Docs/source/dataanalysis/sensei.rst
@@ -30,7 +30,7 @@ and bridge code making it easy to use in AMReX based simulation codes.
 SENSEI provides a *configurable analysis adaptor* which uses an XML file to
 select and configure one or more back ends at run time. Run time selection of
 the back end via XML means one user can access Catalyst, another Libsim, yet
-another Python with no changes to the code.  This is depicted in figure
+another Python with no changes to the code.  This is depicted in
 :numref:`sensei_arch`. On the left side of the figure AMReX produces data, the
 bridge code pushes the data through the configurable analysis adaptor to the
 back end that was selected at run time.
@@ -99,7 +99,7 @@ The back end is selected and configured at run time using the SENSEI XML file.
 The XML sets parameters specific to SENSEI and to the chosen back end. Many of
 the back ends have sophisticated configuration mechanisms which SENSEI makes
 use of.  For example the following XML configuration was used on NERSC's Cori
-with WarpX to render 10 iso surfaces, shown in figure :numref:`lpa_visit`, using
+with WarpX to render 10 iso surfaces, shown in :numref:`lpa_visit`, using
 VisIt Libsim.
 
 .. code-block:: xml
@@ -123,7 +123,7 @@ run of the desired simulation.
    quadrant has been clipped away to reveal innner structure.
 
 The same run and visualization was repeated using ParaView Catalyst, shown in
-figure :numref:`lpa_pv`, by providing the following XML configuration.
+:numref:`lpa_pv`, by providing the following XML configuration.
 
 .. code-block:: xml
 

diff --git a/Docs/source/index.rst b/Docs/source/index.rst
@@ -107,9 +107,9 @@ Theory
    :hidden:
 
    theory/intro
-   theory/picsar_theory
+   theory/pic
    theory/amr
-   theory/PML
+   theory/boundary_conditions
    theory/boosted_frame
    theory/input_output
    theory/collisions

diff --git a/Docs/source/install/dependencies.rst b/Docs/source/install/dependencies.rst
@@ -18,7 +18,7 @@ Optional dependencies include:
 - for on-node accelerated compute *one of either*:
 
   - `OpenMP 3.1+ <https://www.openmp.org>`__: for threaded CPU execution or
-  - `CUDA Toolkit 11.0+ (11.3+ recommended) <https://developer.nvidia.com/cuda-downloads>`__: for Nvidia GPU support (see `matching host-compilers <https://gist.github.com/ax3l/9489132>`_) or
+  - `CUDA Toolkit 11.7+ <https://developer.nvidia.com/cuda-downloads>`__: for Nvidia GPU support (see `matching host-compilers <https://gist.github.com/ax3l/9489132>`_) or
   - `ROCm 5.2+ (5.5+ recommended) <https://gpuopen.com/learn/amd-lab-notes/amd-lab-notes-rocm-installation-readme/>`__: for AMD GPU support
 - `FFTW3 <http://www.fftw.org>`_: for spectral solver (PSATD) support when running on CPU or SYCL
 

diff --git a/Docs/source/install/hpc/karolina.rst b/Docs/source/install/hpc/karolina.rst
@@ -12,83 +12,60 @@ Introduction
 If you are new to this system, **please see the following resources**:
 
 * `IT4I user guide <https://docs.it4i.cz>`__
-* Batch system: `PBS <https://docs.it4i.cz/general/job-submission-and-execution/>`__
+* Batch system: `SLURM <https://docs.it4i.cz/general/job-submission-and-execution/>`__
 * Jupyter service: not provided/documented (yet)
 * `Filesystems <https://docs.it4i.cz/karolina/storage/>`__:
 
   * ``$HOME``: per-user directory, use only for inputs, source and scripts; backed up (25GB default quota)
-  * ``/scratch/``: `production directory <https://docs.it4i.cz/karolina/storage/#scratch-file-system>`__; very fast for parallel jobs (20TB default)
+  * ``/scratch/``: `production directory <https://docs.it4i.cz/karolina/storage/#scratch-file-system>`__; very fast for parallel jobs (10TB default)
+  * ``/mnt/proj<N>/<proj>``: per-project work directory, used for long term data storage (20TB default)
 
 
 .. _building-karolina-preparation:
 
-Preparation
------------
-
-Use the following commands to download the WarpX source code:
+Installation
+------------
 
-.. code-block:: bash
+We show how to install from scratch all the dependencies using `Spack <https://spack.io>`__.
 
-   git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx
+For size reasons it is not advisable to install WarpX in the ``$HOME`` directory, it should be installed in the "work directory". For this purpose we set an environment variable ``$WORK`` with the path to the "work directory".
 
 On Karolina, you can run either on GPU nodes with fast A100 GPUs (recommended) or CPU nodes.
 
-.. tab-set::
-
-   .. tab-item:: A100 GPUs
-
-      We use system software modules, add environment hints and further dependencies via the file ``$HOME/karolina_gpu_warpx.profile``.
-      Create it now:
-
-      .. code-block:: bash
-
-         cp $HOME/src/warpx/Tools/machines/karolina-it4i/karolina_gpu_warpx.profile.example $HOME/karolina_gpu_warpx.profile
-
-      .. dropdown:: Script Details
-         :color: light
-         :icon: info
-         :animate: fade-in-slide-down
-
-         .. literalinclude:: ../../../../Tools/machines/karolina-it4i/karolina_gpu_warpx.profile.example
-            :language: bash
-
-      Edit the 2nd line of this script, which sets the ``export proj=""`` variable.
-      For example, if you are member of the project ``DD-23-83``, then run ``vi $HOME/karolina_gpu_warpx.profile``.
-      Enter the edit mode by typing ``i`` and edit line 2 to read:
-
-      .. code-block:: bash
+Profile file
+^^^^^^^^^^^^
 
-         export proj="DD-23-83"
+One can use the pre-prepared ``karolina_warpx.profile`` script below,
+which you can copy to ``${HOME}/karolina_warpx.profile``, edit as required and then ``source``.
 
-      Exit the ``vi`` editor with ``Esc`` and then type ``:wq`` (write & quit).
+.. dropdown:: Script Details
+   :color: light
+   :icon: info
+   :animate: fade-in-slide-down
 
-      .. important::
+   .. literalinclude:: ../../../../Tools/machines/karolina-it4i/karolina_warpx.profile.example
+      :language: bash
+      :caption: Copy the contents of this file to ``${HOME}/karolina_warpx.profile``.
 
-         Now, and as the first step on future logins to Karolina, activate these environment settings:
+To have the environment activated on every login, add the following line to ``${HOME}/.bashrc``:
 
-         .. code-block:: bash
-
-            source $HOME/karolina_gpu_warpx.profile
-
-      Finally, since Karolina does not yet provide software modules for some of our dependencies, install them once:
-
-      .. code-block:: bash
+.. code-block:: bash
 
-         bash $HOME/src/warpx/Tools/machines/karolina-it4i/install_gpu_dependencies.sh
-         source $HOME/sw/karolina/gpu/venvs/warpx-gpu/bin/activate
+    source $HOME/karolina_warpx.profile
 
-      .. dropdown:: Script Details
-         :color: light
-         :icon: info
-         :animate: fade-in-slide-down
+To install the ``spack`` environment and Python packages:
 
-         .. literalinclude:: ../../../../Tools/machines/karolina-it4i/install_gpu_dependencies.sh
-            :language: bash
+.. code-block:: bash
 
+   bash $WORK/src/warpx/Tools/machines/karolina-it4i/install_dependencies.sh
 
-   .. tab-item:: CPU Nodes
+.. dropdown:: Script Details
+   :color: light
+   :icon: info
+   :animate: fade-in-slide-down
 
-      CPU usage is documentation is TODO.
+   .. literalinclude:: ../../../../Tools/machines/karolina-it4i/install_dependencies.sh
+      :language: bash
 
 
 .. _building-karolina-compilation:
@@ -98,117 +75,53 @@ Compilation
 
 Use the following :ref:`cmake commands <building-cmake>` to compile the application executable:
 
-.. tab-set::
-
-   .. tab-item:: A100 GPUs
-
-      .. code-block:: bash
-
-         cd $HOME/src/warpx
-         rm -rf build_gpu
-
-         cmake -S . -B build_gpu -DWarpX_COMPUTE=CUDA -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3"
-         cmake --build build_gpu -j 12
-
-      The WarpX application executables are now in ``$HOME/src/warpx/build_gpu/bin/``.
-      Additionally, the following commands will install WarpX as a Python module:
-
-      .. code-block:: bash
-
-         rm -rf build_gpu_py
-
-         cmake -S . -B build_gpu_py -DWarpX_COMPUTE=CUDA -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3"
-         cmake --build build_gpu_py -j 12 --target pip_install
-
-   .. tab-item:: CPU Nodes
-
-      .. code-block:: bash
+.. code-block:: bash
 
-         cd $HOME/src/warpx
-         rm -rf build_cpu
+   cd $WORK/src/warpx
+   rm -rf build_gpu
 
-         cmake -S . -B build_cpu -DWarpX_COMPUTE=OMP -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3"
-         cmake --build build_cpu -j 12
+   cmake -S . -B build_gpu -DWarpX_COMPUTE=CUDA -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3"
+   cmake --build build_gpu -j 48
 
-      The WarpX application executables are now in ``$HOME/src/warpx/build_cpu/bin/``.
-      Additionally, the following commands will install WarpX as a Python module:
+The WarpX application executables are now in ``$WORK/src/warpx/build_gpu/bin/``.
+Additionally, the following commands will install WarpX as a Python module:
 
-      .. code-block:: bash
+.. code-block:: bash
 
-         cd $HOME/src/warpx
-         rm -rf build_cpu_py
+   cd $WORK/src/warpx
+   rm -rf build_gpu_py
 
-         cmake -S . -B build_cpu_py -DWarpX_COMPUTE=OMP -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3"
-         cmake --build build_cpu_py -j 12 --target pip_install
+   cmake -S . -B build_gpu_py -DWarpX_COMPUTE=CUDA -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3"
+   cmake --build build_gpu_py -j 48 --target pip_install
 
 Now, you can :ref:`submit Karolina compute jobs <running-cpp-karolina>` for WarpX :ref:`Python (PICMI) scripts <usage-picmi>` (:ref:`example scripts <usage-examples>`).
 Or, you can use the WarpX executables to submit Karolina jobs (:ref:`example inputs <usage-examples>`).
 For executables, you can reference their location in your :ref:`job script <running-cpp-karolina>` or copy them to a location in ``/scratch/``.
 
 
-.. _building-karolina-update:
-
-Update WarpX & Dependencies
----------------------------
-
-If you already installed WarpX in the past and want to update it, start by getting the latest source code:
-
-.. code-block:: bash
-
-   cd $HOME/src/warpx
-
-   # read the output of this command - does it look ok?
-   git status
-
-   # get the latest WarpX source code
-   git fetch
-   git pull
-
-   # read the output of these commands - do they look ok?
-   git status
-   git log # press q to exit
-
-And, if needed,
-
-- :ref:`update the karolina_gpu_warpx.profile or karolina_cpu_warpx.profile files <building-karolina-preparation>`,
-- log out and into the system, activate the now updated environment profile as usual,
-- :ref:`execute the dependency install scripts <building-karolina-preparation>`.
-
-As a last step, clean the build directory ``rm -rf $HOME/src/warpx/build_*`` and rebuild WarpX.
-
-
 .. _running-cpp-karolina:
 
 Running
 -------
 
-.. tab-set::
+The batch script below can be used to run a WarpX simulation on multiple GPU nodes (change ``#SBATCH --nodes=`` accordingly) on the supercomputer Karolina at IT4I.
+This partition has up to `72 nodes <https://docs.it4i.cz/karolina/hardware-overview/>`__.
+Every node has 8x A100 (40GB) GPUs and 2x AMD EPYC 7763, 64-core, 2.45 GHz processors.
 
-   .. tab-item:: A100 (40GB) GPUs
+Replace descriptions between chevrons ``<>`` by relevant values, for instance ``<proj>`` could be ``DD-23-83``.
+Note that we run one MPI rank per GPU.
 
-      The batch script below can be used to run a WarpX simulation on multiple GPU nodes (change ``#PBS -l select=`` accordingly) on the supercomputer Karolina at IT4I.
-      This partition as up to `72 nodes <https://docs.it4i.cz/karolina/hardware-overview/>`__.
-      Every node has 8x A100 (40GB) GPUs and 2x AMD EPYC 7763, 64-core, 2.45 GHz processors.
+.. literalinclude:: ../../../../Tools/machines/karolina-it4i/karolina_gpu.sbatch
+   :language: bash
+   :caption: You can copy this file from ``$WORK/src/warpx/Tools/machines/karolina-it4i/karolina_gpu.sbatch``.
 
-      Replace descriptions between chevrons ``<>`` by relevant values, for instance ``<proj>`` could be ``DD-23-83``.
-      Note that we run one MPI rank per GPU.
-
-      .. literalinclude:: ../../../../Tools/machines/karolina-it4i/karolina_gpu.qsub
-         :language: bash
-         :caption: You can copy this file from ``$HOME/src/warpx/Tools/machines/karolina-it4i/karolina_gpu.qsub``.
-
-      To run a simulation, copy the lines above to a file ``karolina_gpu.qsub`` and run
-
-      .. code-block:: bash
-
-         qsub karolina_gpu.qsub
-
-      to submit the job.
+To run a simulation, copy the lines above to a file ``karolina_gpu.sbatch`` and run
 
+.. code-block:: bash
 
-   .. tab-item:: CPU Nodes
+   sbatch karolina_gpu.sbatch
 
-      CPU usage is documentation is TODO.
+to submit the job.
 
 
 .. _post-processing-karolina:

diff --git a/Docs/source/latex_theory/Makefile b/Docs/source/latex_theory/Makefile
@@ -7,14 +7,14 @@ all: $(SRC_FILES) clean
 	pandoc Boosted_frame/Boosted_frame.tex --mathjax --wrap=preserve --bibliography allbibs.bib -o boosted_frame.rst
 	pandoc input_output/input_output.tex --mathjax --wrap=preserve --bibliography allbibs.bib -o input_output.rst
 	mv *.rst ../theory
-	cd ../../../../picsar/Doxygen/pages/latex_theory/; pandoc theory.tex --mathjax --wrap=preserve --bibliography allbibs.bib -o picsar_theory.rst
-	mv ../../../../picsar/Doxygen/pages/latex_theory/picsar_theory.rst ../theory
+	cd ../../../../picsar/Doxygen/pages/latex_theory/; pandoc theory.tex --mathjax --wrap=preserve --bibliography allbibs.bib -o pic.rst
+	mv ../../../../picsar/Doxygen/pages/latex_theory/pic.rst ../theory
 	cp ../../../../picsar/Doxygen/images/PIC.png ../theory
 	cp ../../../../picsar/Doxygen/images/Yee_grid.png ../theory
 
 clean:
 	rm -f ../theory/intro.rst
 	rm -f ../theory/warpx_theory.rst
-	rm -f ../theory/picsar_theory.rst
+	rm -f ../theory/pic.rst
 	rm -f ../theory/PIC.png
 	rm -f ../theory/Yee_grid.png
diff --git a/Docs/source/latex_theory/input_output/input_output.tex b/Docs/source/latex_theory/input_output/input_output.tex
@@ -30,7 +30,7 @@ \subsection{Inputs and outputs in a boosted frame simulation}
 
 \subsubsection{Input in a boosted frame simulation}
 \paragraph{Particles - }
-Particles are launched through a plane using a technique that is generic and applies to Lorentz boosted frame simulations in general, including plasma acceleration, and is illustrated using the case of a positively charged particle beam propagating through a background of cold electrons in an assumed continuous transverse focusing system, leading to a well-known growing transverse ``electron cloud'' instability  \cite{Vayprl07}. In the laboratory frame, the electron background is initially at rest and a moving window is used to follow the beam progression. Traditionally, the beam macroparticles are initialized all at once in the window, while background electron macroparticles are created continuously in front of the beam on a plane that is perpendicular to the beam velocity. In a frame moving at some fraction of the beam velocity in the laboratory frame, the beam initial conditions at a given time in the calculation frame are generally unknown and one must initialize the beam differently. However, it can be taken advantage of the fact that the beam initial conditions are often known for a given plane in the laboratory, either directly, or via simple calculation or projection from the conditions at a given time in the labortory frame. Given the position and velocity $\{x,y,z,v_x,v_y,v_z\}$ for each beam macroparticle at time $t=0$ for a beam moving at the average velocity $v_b=\beta_b c$ (where $c$ is the speed of light) in the laboratory, and using the standard synchronization ($z=z'=0$ at $t=t'=0$) between the laboratory and the calculation frames, the procedure for transforming the beam quantities for injection in a boosted frame moving at velocity $\beta c$ in the laboratory is as follows (the superscript $'$ relates to quantities known in the boosted frame while the superscript $^*$ relates to quantities that are know at a given longitudinal position $z^*$ but different times of arrival):
+Particles are launched through a plane using a technique that is generic and applies to Lorentz boosted frame simulations in general, including plasma acceleration, and is illustrated using the case of a positively charged particle beam propagating through a background of cold electrons in an assumed continuous transverse focusing system, leading to a well-known growing transverse ``electron cloud'' instability \cite{Vayprl07}. In the laboratory frame, the electron background is initially at rest and a moving window is used to follow the beam progression. Traditionally, the beam macroparticles are initialized all at once in the window, while background electron macroparticles are created continuously in front of the beam on a plane that is perpendicular to the beam velocity. In a frame moving at some fraction of the beam velocity in the laboratory frame, the beam initial conditions at a given time in the calculation frame are generally unknown and one must initialize the beam differently. However, it can be taken advantage of the fact that the beam initial conditions are often known for a given plane in the laboratory, either directly, or via simple calculation or projection from the conditions at a given time in the labortory frame. Given the position and velocity $\{x,y,z,v_x,v_y,v_z\}$ for each beam macroparticle at time $t=0$ for a beam moving at the average velocity $v_b=\beta_b c$ (where $c$ is the speed of light) in the laboratory, and using the standard synchronization ($z=z'=0$ at $t=t'=0$) between the laboratory and the calculation frames, the procedure for transforming the beam quantities for injection in a boosted frame moving at velocity $\beta c$ in the laboratory is as follows (the superscript $'$ relates to quantities known in the boosted frame while the superscript $^*$ relates to quantities that are know at a given longitudinal position $z^*$ but different times of arrival):
 
 \begin{enumerate}
 \item project positions at $z^*=0$ assuming ballistic propagation