merging with dev

Author: ethan-lame

.github/workflows/docs_test.yml

@@ -0,0 +1,33 @@
+name: Test Build the Docs
+
+on: [push, pull_request]
+
+jobs:
+  build:
+    runs-on: ${{ matrix.os }}
+    strategy:
+        fail-fast: false
+        matrix:
+            os: ["ubuntu-latest"]
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up python 3.11
+      uses: actions/setup-python@v3
+      with:
+        python-version: "3.11"
+    - name: debug
+      run: |
+        pwd
+        ls
+    - uses: mpi4py/setup-mpi@v1
+    - name: Install dependencies
+      run: |
+        pip install --user . mcdc[docs]
+        pip list 
+    - name: Patch Numba
+      run : |
+        bash .github/workflows/patch.sh
+    - name: Build the Docs
+      run: |
+        cd docs
+        make html

README.md

@@ -17,25 +17,25 @@ Our documentation on installation, contribution, and a brief user guide is on [R
 
 ## Installation
 
-We recommend using [`conda`](https://conda.io/projects/conda/en/latest/user-guide/install/index.html) or some other environment manager to manage the MC/DC installation.
+We recommend using [Python virtual environments (venv)](https://docs.python.org/3/library/venv.html) or some other environment manager (e.g. conda) to manage the MC/DC installation.
 This avoids the need for admin access when installing MC/DC's dependencies and allows greater configurability for developers.
-For most users working on a single machine of which they are administrators, MC/DC can be installed via pip:
+For most users working in a venv, MC/DC can be installed via pip:
 ```bash
 pip install mcdc
 ```
-For developers or users on HPC machines, we recommend that you *not* use the pip distribution and instead install MC/DC and its dependencies via the included [install script](https://mcdc.readthedocs.io/en/latest/install.html), which builds `mpi4py` from source and uses conda to manage your environment. *This is the most reliable way to install and configure MC/DC*. It also takes care of the [Numba patch]() and can configure the [continuous energy data library](), if you have access.
+For developers or users on HPC machines, mpi4py is often distributed as part of an HPC machines given venv.
 
 ### Common issues with `mpi4py`
 
 The `pip mpi4py` distribution commonly has errors when building due to incompatible local MPI dependencies it builds off of. While pip does have some remedy for this, we recommend the following:
 * **Mac users:** we recommend `openmpi` is [installed via homebrew](https://formulae.brew.sh/formula/open-mpi) (note that more reliable mpi4py distribution can also be [found on homebrew](https://formulae.brew.sh/formula/mpi4py)), alternatively you can use `conda` if you don't have admin privileges;
 * **Linux users:** we recommend `openmpi` is installed via a root package manager if possible (e.g. `sudo apt install openmpi`) or a conda distribution (e.g. `conda install openmpi`)
-* **HPC users and developers on any system:** On HPC systems in particular, `mpi4py` must be built using the system's existing `mpi` installation. Installing MC/DC using the [install script](https://mcdc.readthedocs.io/en/latest/install.html) we've included will handle that for you by installing dependencies using conda rather than pip. It also takes care of the [Numba patch]() and can configure the [continuous energy data library](), if you have access.
+* **HPC users and developers on any system:** On HPC systems that do not supply a suitable venv, `mpi4py` may need to be built using the system's existing `mpi` installation. Installing MC/DC using the [install script](https://mcdc.readthedocs.io/en/latest/install.html) we've included will handle that for you by installing dependencies using conda rather than pip. It also takes care of the [Numba patch](https://github.com/CEMeNT-PSAAP/MCDC/blob/main/patch_numba.sh) and can configure the [continuous energy data library](https://github.com/CEMeNT-PSAAP/MCDC/blob/main/config_cont_energy.sh), if you have access.
 
 ### Numba Config
 
 Running MC/DC performantly in [Numba mode](#numba-mode) requires a patch to a single Numba file. If you installed MC/DC with the [install script](https://mcdc.readthedocs.io/en/latest/install.html), this patch has already been taken care of. If you installed via pip, we have a patch script will make the necessary changes for you:
-1. Download the `patch.sh` file [here]() (If you've cloned MC/DC's GitHub repository, you already have this file in your MCDC/ directory).
+1. Download the `patch.sh` file [here](https://github.com/CEMeNT-PSAAP/MCDC/blob/main/patch_numba.sh) (If you've cloned MC/DC's GitHub repository, you already have this file in your MCDC/ directory).
 2. In your active conda environment, run `bash patch_numba.sh`.
 *If you manage your environment with conda, you will not need admin privileges*.
 

docs/source/conf.py

@@ -18,9 +18,11 @@
 # On Read the Docs, need to mock any python packages that would require c
 from unittest.mock import MagicMock
 
-MOCK_MODULES = ["mpi4py", "colorama", "mpi4py.util.dtlib"]
+MOCK_MODULES = ["mpi4py", "colorama", "mpi4py.util.dtlib", "sympy", "matplotlib.pyplot"]
 sys.modules.update((mod_name, MagicMock()) for mod_name in MOCK_MODULES)
+from mpi4py import MPI
 
+MPI.COMM_WORLD.Get_size.return_value = 1
 
 # -- Project information -----------------------------------------------------
 

docs/source/index.rst

@@ -81,7 +81,7 @@ A full exhaustive list of publications can be found on the `CEMeNT site <https:/
     :maxdepth: 1
 
     install
-    user
+    user/index
     contribution
     theory/index
     pythonapi/index

docs/source/install.rst

@@ -4,48 +4,93 @@
 Installation Guide
 ===================
 
-Developers in MC/DC (on any machine) or users on HPC machines should install using the installation script included with the source code; 
-start by :ref:`creating-a-conda-environment`. 
-Installing from source via the installation script is the most resilient way to get properly configured dependencies.
-Most other users can install using pip. 
+Whether installing MC/DC as a user or from source as a developer, 
+we recommend doing so using an environment manager like venv or conda.
+This will avoid the need for any admin access and keep dependencies clean.
+
+In general, :ref:`creating-a-venv-environment` and :ref:`installing-with-pip` is easier and recommended.
+Creating a conda environment and :ref:`installing-with-conda` is more robust and reliable, but is also more difficult. 
+A conda environment is necessary to install MC/DC on LLNL's Lassen machine.
+
+
+
+.. _creating-a-venv-environment:
+
+---------------------------
+Creating a venv environment
+---------------------------
+
+Python `virtual environments <https://docs.python.org/3.11/library/venv.html>`_ are the easy and 
+recommended way to get MC/DC operating on personal machines as well as HPCs;
+all you need is a working Python version with venv installed.
+Particularly on HPCs, using a Python virtual environment is convenient because
+system admins will have already configured venv and the pip within it to load packages and dependencies
+from the proper sources. 
+HPCs often use a module system, so before doing anything else, 
+``module load python/<version_number>``.
+
+A python virtual environment can (usually) be created using
+
+.. code-block:: sh
+
+    python -m venv <name_of_venv>
+
+Once you have created a venv, you will need to activate it
+
+.. code-block:: sh
+
+    source <name_of_venv>/bin/activate
+
+and will need to do so every time a new terminal instance is launched.
+Once your environment is active, you can move on to :ref:`installing-with-pip`.
+
+
+.. _installing-with-pip:
 
 -------------------
 Installing with pip
 -------------------
-Users who:
-
-#. are unix based (macOS, linux, etc.),
-#. have a working version of openMPI (from conda, brew, or apt),
-#. are using an environment manager like conda or have administrator privileges, and
-#. plan to *use* MC/DC, not develop features for MC/DC
-
-can install using pip. 
-We recommend doing so within an active conda (or other environment manager) environment, 
-which avoids the need for any admin access and keeps dependencies clean. 
+Assuming you have a working Python environment, you can install using pip. 
+Doing so within an active venv or conda environment avoids the need for any admin access
+and keeps dependencies clean.
 
+If you would like to run MC/DC as published in the main branch *and* 
+do not need to develop in MC/DC, you can install from PyPI: 
+ 
 .. code-block:: sh
 
     pip install mcdc
 
-Now you're ready to run in pure Python mode!
+If you would like to execute a version of MC/DC from a specific branch or 
+*do* plan to develop in MC/DC, you'll need to install from source: 
 
-.. _creating-a-conda-environment:
+#. Clone the MC/DC repo: ``git clone https://github.com/CEMeNT-PSAAP/MCDC.git`` 
+#. Go to your new MC/DC directory: ``cd MCDC``
+#. Install the package from your MC/DC files: ``pip install -e .``
+#. Run the included script that makes a necessary numba patch: ``bash patch_numba.sh``
 
------------------------------------
-Creating an MC/DC Conda environment
------------------------------------
+This should install all needed dependencies without a hitch. 
+The `-e` flag installs MC/DC as an editable package, meaning that any changes
+you make to the MC/DC source files, including checking out a different
+branch,  will be immediately reflected without needing to do any re-installation.
 
+.. _installing-with-conda:
+
+--------------------------
+Installing MC/DC via conda
+--------------------------
+
+Conda is the most robust (works even on bespoke systems) option to install MC/DC.
 `Conda <https://conda.io/en/latest/>`_ is an open source package and environment management system 
 that runs on Windows, macOS, and Linux. It allows for easy installing and switching between multiple
 versions of software packages and their dependencies. 
-We can't force you to use it, but we do *highly* recommend it, particularly
-if you plan on running MC/DC in `numba mode <https://numba.pydata.org/>`_.
-**The included installation script will fail if executed outside of a conda environment.**
+Conda is really useful on systems with non-standard hardware (e.g. not x86 CPUs) like Lassen, where
+mpi4py is often the most troublesome dependency. 
 
 First, ``conda`` should be installed with `Miniconda <https://docs.conda.io/en/latest/miniconda.html>`_
 or `Anaconda <https://www.anaconda.com/>`_. HPC instructions: 
 
-`Quartz <https://hpc.llnl.gov/hardware/compute-platforms/quartz>`_ (LLNL, x86_64), 
+`Dane <https://hpc.llnl.gov/hardware/compute-platforms/dane>`_ (LLNL, x86_64), 
 
 .. code-block:: sh
 
@@ -62,39 +107,32 @@ or `Anaconda <https://www.anaconda.com/>`_. HPC instructions:
 
 
 Then create and activate a new conda environment called *mcdc-env* in
-which to install MC/DC. This creates an environment with python3.11 
-installed.
+which to install MC/DC. This creates an environment with python3.12 
+installed:
 
 .. code-block:: sh
 
-    conda create -n mcdc-env python=3.11
+    conda create -n mcdc-env python=3.12
     conda activate mcdc-env
 
--------------------------------------------
-Installing from Source on Linux or Mac OS X
--------------------------------------------
-
-All MC/DC source code is hosted on `Github <https://github.com/CEMeNT-PSAAP/MCDC>`_.
-If you have `git <https://git-scm.com>`_, you can download MC/DC by entering the
-following commands in a terminal:
+Then, MC/DC can be installed from source by first cloning the MC/DC repository:
 
 .. code-block:: sh
 
     git clone https://github.com/CEMeNT-PSAAP/MCDC.git
     cd MCDC
 
-
-The MC/DC repository includes the script ``install.sh``, which will 
+then using the the ``install.sh`` within it. The install script will
 build MC/DC and all of its dependencies and execute any necessary patches.
-This has been tested on Quartz, Lassen, and Apple M2 (as of 11/01/2023). 
+This has been tested on Quartz, Dane, Tioga, Lassen, and Apple M2. 
 The ``install.sh`` script **will fail outside of a conda environment**.
 
 On HPC machines, the script will install mpi4py 
 `from source <https://mpi4py.readthedocs.io/en/stable/install.html#using-distutils>`_.
 This means that all appropriate modules must be loaded prior to executing.
 
 On Quartz, the default modules are sufficient (``intel-classic`` and ``mvapich2``). 
-On Lassen, ``module load gcc/8 cuda/11.3``. Then, 
+On Lassen, ``module load gcc/8 cuda/11.8``. Then, 
 
 .. code-block:: sh
 
@@ -107,7 +145,6 @@ On local machines, mpi4py will be installed using conda,
 
     bash install.sh 
 
-
 To confirm that everything is properly installed, execute ``pytest`` from the MCDC directory. 
 
 -------------------------------------
@@ -132,3 +169,42 @@ or run the script after instillation as a stand alone operation with
 
 Both these operations will clone the internal directory to your MCDC directory, untar the compressed folder, then set an environment variable in your bash script.
 NOTE: this does assume you are using bash shell.
+
+
+---------------------------------
+GPU Operability (MC/DC+Harmonize)
+---------------------------------
+
+MC/DC supports most of its Numba enabled features for GPU compilation and execution.
+When targeting GPUs, MC/DC uses the `Harmonize <https://github.com/CEMeNT-PSAAP/harmonize>`_ library as its GPU runtime, a.k.a. the thing that actually executes MC/DC functions.
+How Harmonize works gets a little involved, but in short, 
+Harmonize acts as MC/DC's GPU runtime by using two major scheduling schemes: an event schedular similar to those implemented in OpenMC and Shift, plus a novel scheduler.
+For more information on Harmonize and how we compile MC/DC with it, see this `TOMACs article describing the async scheduler <https://doi.org/10.1145/3626957>`_ or our publications in American Nuclear Society: Math and Comp Meeting in 2025.
+
+If you encounter problems with configuration, please file `Github issues promptly <https://github.com/CEMeNT-PSAAP/MCDC/issues>`_ ,
+especially when on supported super computers (LLNL's `Tioga <https://hpc.llnl.gov/hardware/compute-platforms/tioga>`_, `El Capitan <https://hpc.llnl.gov/documentation/user-guides/using-el-capitan-systems>`_, and `Lassen <https://hpc.llnl.gov/hardware/compute-platforms/lassen>`_).
+
+Nvidia GPUs
+^^^^^^^^^^^
+
+To compile and execute MC/DC on Nvidia GPUs first ensure you have the `Harmonize prerecs <https://github.com/CEMeNT-PSAAP/harmonize/blob/main/install.sh>`_ (CUDA=11.8, Numba>=0.58.0) and a working MC/DC version >=0.10.0. Then,
+
+#. Clone the harmonize repo: ``git clone https://github.com/CEMeNT-PSAAP/harmonize.git``
+#. Install into the proper Python env: ``pip install -e .``
+
+Operability should now be enabled. 
+
+AMD GPUs
+^^^^^^^^
+
+The prerequisites for AMD operability are slightly more complex and
+require a patch to Numba to allow for AMD target triple LLVM-IR.
+It is recommended that this is done within a Python venv virtual environment.
+
+To compile and execute MC/DC on AMD GPUs first ensure you have the `Harmonize prerecs <https://github.com/CEMeNT-PSAAP/harmonize/blob/main/install.sh>`_ (ROCm=6.0.0, Numba>=0.58.0) and a working MC/DC version >=0.11.0. Then,
+
+#. Patch Numba to enable HIP (`instructions here <https://github.com/ROCm/numba-hip>`_)
+#. Clone harmonize and `switch to the AMD <https://github.com/CEMeNT-PSAAP/harmonize/tree/amd_event_interop_revamp>`_ branch with ``git switch amd_event_interop_revamp`
+#. Install Harmonize with ``pip install -e .`` or using `Harmonize's install script <https://github.com/CEMeNT-PSAAP/harmonize/tree/main>`_
+
+Operability should now be enabled.

docs/source/pubs.rst

@@ -45,6 +45,10 @@ Conference on Physics of Reactors. Pittsburgh, Pennsylvania, USA (2022).
 Software Engineering in MC/DC Publications
 -------------------------------------------
 
+J. P. Morgan, I. Variansyah, B. Cuneo, T. S. Palmer and K. E. Niemeyer. 2024. UNDER REVIEW. Performance Portable Monte Carlo Neutron Transport in MCDC via Numba. Preprint DOI 10.48550/arXiv.2306.07847.
+
+B. Cuneo and I. Variansyah. “An Alternative to Stride-Based RNG for Monte Carlo Transport.” In Transactions of The American Nuclear Society, volume 130 (1), pp. 423–426 (2024). DOI 10.13182/T130-44927
+
 J. P. Morgan, I. Variansyah, S. Pasmann, K. B. Clements, B. Cuneo, A. Mote, 
 C. Goodman, C. Shaw, J. Northrop, R. Pankaj, E. Lame, B. Whewell, 
 R. McClarren, T. S. Palmer, L. Chen, D. Anistratov, C. T. Kelley, 

docs/source/user/cpu.rst

@@ -0,0 +1,62 @@
+
+.. _gpu:
+
+=====================
+Running MC/DC on CPUs
+=====================
+
+Executing MC/DC in something like a jupyter notebook is possible but not recommended,
+especially when using MPI and/or Numba.
+The instructions below assume you have an existing MC/DC installation.
+MPI can be quite tricky to configure if on an HPC; if you're having trouble,
+consult our :ref:`install`, your HPC admin, or our `GitHub issues page <https://github.com/CEMeNT-PSAAP/MCDC/issues>`_.
+
+Pure Python Mode
+----------------
+
+To run in pure Python mode (slower, no acceleration)
+
+.. code-block:: python3
+
+    python input.py
+
+Numba Mode
+----------
+
+.. code-block:: python3
+
+    python input.py --mode=numba
+
+When running in Numba mode a significant amount of time is taken compiling Python functions to performant binaries.
+Only the functions used in a specific simulation will be compiled.
+These binaries will be cached, meaning that in subsequent runs of the same simulation the compilation step can be avoided.
+The cache can be used as an effective ahead-of-time compilation scheme where binaries can be compiled once and shared between machines.
+For more information on caching see :ref:`Caching` and `Numba Caching <https://numba.readthedocs.io/en/stable/developer/caching.html>`_.
+
+MC/DC also has the ability to run Numba in a debugging mode.
+This will result in less performant code and longer compile times but will allow for better error messages from Numba and other packages.
+
+.. code-block:: python3
+
+    python input.py --mode=numba_debug
+
+
+For more information on the exact behavior of this option see :ref:`Debugging`
+
+Using MPI
+---------
+
+MC/DC can be executed using MPI with or without Numba acceleration.
+If ``numba-mode`` is enabled the ``jit`` compilation, which is executed on all threads, can take between 30s-2min.
+For smaller problems, Numba compilation time could exceed runtime, and pure python mode could be preferable.
+Below, ``--mode`` can equal python or numba. MC/DC gets MPI functionality via `mpi4py <https://mpi4py.readthedocs.io/en/stable/>`_. 
+As an example, to run on 36 processes in Numba mode with `SLURM <https://slurm.schedmd.com/documentation.html>`_:
+
+.. code-block:: python3
+
+    srun -n 36 python input.py --mode=<python/numba>
+
+For systems that do not use SLURM (i.e., a local system) try ``mpiexec`` or ``mpirun`` in its stead.
+
+CPU Profiling
+-------------