Merge branch 'ComputationalRadiationPhysics:dev' into Fusion_interSpecies

Author: FilipO28555

docs/source/dev/debugging.rst

@@ -26,6 +26,19 @@ An example build command is
 
    pic-build -c "-DPIC_VERBOSE=127 -DPMACC_VERBOSE=127 -DPMACC_BLOCKING_KERNEL=ON -DPMACC_ASYNC_QUEUES=OFF"
 
+
+If you encounter an error during reading or writing of openPMD files, it can be helpful to run openPMD-api with verbose output.
+This can be done by setting the following environment variable either in your environment (e.g. ``picongpu.profile``) or in your submit script ``./tbg/submit.start`` before executing ``picongpu``.
+
+.. code:: bash
+
+   export OPENPMD_VERBOSE=1
+
+
+Additionally, you can activate IO logging in PIConGPU by adding the login level ``32``.
+This can be done via command flags (see above) when using ``pic-build```or via ``ccmake .`` in the ``.build`` directory.
+To debug the performance of openPMD output in PIConGPU, set the CMake variable ``PIC_OPENPMD_TIMETRACE_NUMBER_OF_FILES`` to ``1`` to obtain detailed time information about the I/O steps.
+
 When reporting a crash, it is helpful if you attached the output ``stdout`` and ``stderr`` of such a build.
 
 For further debugging tips and use of tools please refer to `our Wiki <https://github.com/ComputationalRadiationPhysics/picongpu/wiki/Debugging>`_.

docs/source/models/atomic_physics.rst

@@ -76,7 +76,7 @@ Getting Started With FLYonPIC
 
 To use FLYonPIC in a simulation setup, you need to provide a set of atomic input data files, see :ref:`Input Data For FLYonPIC <atomicPhysicsInputData>`, and mark at least one species as an atomic physics ion species and one species as an atomic physics electron species.
 
-See the `compile time test of atomic physics <https://github.com/ComputationalRadiationPhysics/picongpu/tree/dev/share/picongpu/tests/compileAtomicPhysics>`_ for examples using atomic physics.
+See the `compile time test of atomic physics <https://github.com/ComputationalRadiationPhysics/picongpu/tree/dev/share/picongpu/examples/AtomicPhysics>`_ for examples using atomic physics.
 
 .. warning::
 

docs/source/usage/plugins/openPMD.rst

@@ -179,17 +179,14 @@ ADIOS2
 Streaming with the MPI-based SST implementation
 """""""""""""""""""""""""""""""""""""""""""""""
 
-The `SST engine <https://adios2.readthedocs.io/en/latest/engines/engines.html#sst-sustainable-staging-transport>`_ (sustainable staging transport) of ADIOS2 has a number of possible backends, including an MPI data transport (still in development at the time of writing this 2023-01-06).
+The `SST engine <https://adios2.readthedocs.io/en/latest/engines/engines.html#sst-sustainable-staging-transport>`_ (sustainable staging transport) of ADIOS2 has a number of possible data transports (called data planes in terms of ADIOS2), including an MPI data transport.
 It internally uses MPI to set up communcation between data producer (PIConGPU) and consumer via ``MPI_Open_Port()`` and ``MPI_Comm_Accept()``.
 
 Use of this engine requires availability of threaded MPI, which can be activated in PIConGPU via the environment variable ``PIC_USE_THREADED_MPI``.
 Its possible values include ``"MPI_THREAD_SINGLE"``, ``"MPI_THREAD_FUNNELED"``, ``"MPI_THREAD_SERIALIZED"`` and ``"MPI_THREAD_MULTIPLE"``.
 The specified value determines the ``int required`` parameter that will be used in the initialization of MPI via ``int MPI_Init_thread( int *argc, char ***argv, int required, int *provided )``.
 For support of MPI-based SST, the variable must be specified as ``PIC_USE_THREADED_MPI=MPI_THREAD_MULTIPLE``.
 
-Additionally, `a workaround <https://github.com/ornladios/ADIOS2/blob/770d852ac1e5d2faff0b9e62cb8da43d448cca67/docs/user_guide/source/advanced/ecp_hardware.rst>`_ is required in order to let PIConGPU cleanly terminate after using this engine on ECP hardware.
-This workaround can be activated on such systems via ``PIC_WORKAROUND_CRAY_MPI_FINALIZE=1``.
-
 HDF5
 ====
 

etc/picongpu/frontier-ornl/batch_picongpu.profile.example

@@ -30,8 +30,8 @@ export PIC_NODE_OVERSUBSCRIPTION_PT=2
 # The following modules just add to these.
 
 module load PrgEnv-cray/8.6.0 # Compiling with cray compiler wrapper CC
-module load craype-accel-amd-gfx90a
-module load rocm/6.4.1
+module load craype-accel-amd-gfx940
+module load rocm/6.4.2
 
 export MPICH_GPU_SUPPORT_ENABLED=1
 module load cray-mpich/8.1.31
@@ -40,6 +40,11 @@ module load boost/1.86.0
 module load zlib/1.3.1
 module load cmake/3.30.5
 module load git
+module load adios2/2.10.2-mpi
+# Please use a self compiled openpmd-api with adios2 if you need blosc2 support
+module load openpmd-api/0.16.0
+module load cray-fftw
+module load libpng/1.6.39
 
 ## set environment variables required for compiling and linking
 ##   see (https://docs.olcf.ornl.gov/systems/crusher_quick_start_guide.html#compiling-with-hipcc)
@@ -57,12 +62,6 @@ export LDFLAGS="$LDFLAGS -L${MPICH_DIR}/lib -lmpi ${PE_MPICH_GTL_DIR_amd_gfx90a}
 
 # Other Software ##############################################################
 #
-module load libpng/1.6.39
-
-# Please use your self compiled openpmd-api with adios2 and blosc2 support.
-# ORNL is providing spack generated modules without blosc2 support.
-#
-# module load openpmd-api/0.16.X
 
 # Self-Build Software #########################################################
 # Optional, not required.

etc/picongpu/jupiter-jsc/gh200.tpl

@@ -19,15 +19,14 @@
 #
 
 
-# PIConGPU batch script for JEDI's SLURM batch system
+# PIConGPU batch script for JUPITER's SLURM batch system
 
 #SBATCH --account=!TBG_nameProject
 #SBATCH --partition=!TBG_queue
 #SBATCH --time=!TBG_wallTime
 # Sets batch job's name
 #SBATCH --job-name=!TBG_jobName
 #SBATCH --nodes=!TBG_nodes
-#SBATCH --exclusive
 #SBATCH --gres=gpu:!TBG_devicesPerNode
 #SBATCH --ntasks=!TBG_tasks
 #SBATCH --ntasks-per-node=!TBG_devicesPerNode
@@ -46,7 +45,7 @@
 export UCX_RC_TIMEOUT=3000000.00us # 3s instead of 1s
 
 ## calculations will be performed by tbg ##
-.TBG_queue="jureap"
+.TBG_queue="booster"
 
 # settings that can be controlled by environment variables before submit
 .TBG_mailSettings=${MY_MAILNOTIFY:-"NONE"}

etc/picongpu/jupiter-jsc/gh200_picongpu.profile.example

@@ -102,7 +102,7 @@ function getNode() {
         return 1
     fi
     echo "Hint: please use 'srun --cpu_bind=sockets <COMMAND>' for launching multiple processes in the interactive mode"
-    salloc --time=1:00:00 --nodes=$numNodes --ntasks-per-node=4 --gres=gpu:4 --mem=115G -A $account -p jureap bash
+    salloc --time=1:00:00 --nodes=$numNodes --ntasks-per-node=4 --gres=gpu:4 --mem=115G -A $account -p booster bash
 }
 
 # allocate an interactive shell for one hour
@@ -119,7 +119,7 @@ function getDevice() {
         fi
     fi
     echo "Hint: please use 'srun --cpu_bind=sockets <COMMAND>' for launching multiple processes in the interactive mode"
-    srun --time=1:00:00 --ntasks-per-node=$(($numDevices)) --gres=gpu:$(($numDevices)) --mem=115G -A $account -p jureap --pty bash
+    srun --time=1:00:00 --ntasks-per-node=$(($numDevices)) --gres=gpu:$(($numDevices)) --mem=115G -A $account -p booster --pty bash
 }
 
 # Load autocompletion for PIConGPU commands