Merge pull request #374 from glotzerlab/conda-for-hpc

Deploy conda packages on HPC platforms.

.gitignore

@@ -2,3 +2,4 @@ __pycache__
 *.sublime-workspace
 *.simg
 Dockerfile
+slurm*.out

CHANGELOG.rst

@@ -11,6 +11,17 @@ Find the complete archive of releases on `Docker Hub`_.
 2024
 ----
 
+Next release
+++++++++++++
+
+*Added*
+
+* Conda package distribution for UMich Great Lakes, Purdue ANVIL, and NCSA Delta.
+
+*Deprecated*
+
+* Docker images
+
 2024-02-02
 ++++++++++
 

README.md

@@ -1,21 +1,57 @@
 # Glotzerlab software
 
-**glotzerlab-software** is a set of container images that provide software developed by the [Glotzer
+**glotzerlab-software** deploys software developed by the [Glotzer
 group](<http://glotzerlab.engin.umich.edu>) at the [University of Michigan](https://www.umich.edu/)
-along with related software commonly used in simulation and data analysis workflows.
+to HPC resources via conda packages and container images. Use **glotzerlab-software** to install
+a MPI and GPU enabled build of HOOMD-blue on a *HPC resource*. Use the standard conda-forge
+provided HOOMD-blue (`mamba install hoomd` with no special configuration) for serial execution
+on individual workstations.
 
 ## Resources
 
 * [Documentation](https://glotzerlab-software.readthedocs.io/):
-  Tutorial, usage information, and list of supported clusters.
+  Tutorial, usage information, and list of supported HPC resources.
 * [glotzerlab-software discussion board](https://github.com/glotzerlab/software/discussions):
   Ask the **glotzerlab-software** user community for help.
-* [glotzerlab/software on Docker Hub](https://hub.docker.com/r/glotzerlab/software):
-  See the list of published images.
+* [HOOMD-blue](https://glotzerlab.engin.umich.edu/hoomd-blue/): The general-purpose particle
+  simulation toolkit that **glotzerlab-software** packages.
 
-## Quick start
+## Quick start [conda]
 
-First, download the stable **glotzerlab/software** image using
+1. Install [miniforge](https://github.com/conda-forge/miniforge).
+2. Replace `$HOME/miniforge3/.condarc` with:
+
+       channel_priority: strict
+       channels:
+         - file://path/to/conda/channel
+         - conda-forge
+
+       disallow:
+       - openmpi
+       - mpich
+       - cuda-cudart-dev
+
+3. Install HOOMD-blue:
+
+       mamba install hoomd
+
+4. Load prerequisite modules. For example:
+
+       module load openmpi cuda
+
+5. Execute Python scripts with MPI and GPU support. For example:
+
+       srun -n 8 python3 script.py
+
+You can confirm that the correct package was installed with `mamba list hoomd`. The full package
+build will contain the cluster name and a `gpu` or `cpu` suffix. For example:
+
+    # Name      Version   Build                          Channel
+    hoomd       X.Y.Z     cluster_gpu_py311h1b32822_0    file://path/to/conda/channel
+
+## Quick start [container]
+
+Download the stable **glotzerlab/software** image using
 [Singularity](https://sylabs.io/singularity/):
 
     $ singularity pull software.sif docker://glotzerlab/software
@@ -29,8 +65,6 @@ Add the ``--nv`` option to **enable NVIDIA GPUs** inside the container:
 
     $ singularity exec --nv software.sif nvidia-smi
 
-## Updates
-
 **glotzerlab-software** updates regularly with the latest versions of included software. Download
 the latest image to update:
 
@@ -42,9 +76,7 @@ See [CHANGELOG.rst](CHANGELOG.rst).
 
 ## Contributing
 
-Contributions are welcomed via
-[pull requests](https://github.com/glotzerlab/software/pulls).
-Please report bugs and suggest feature enhancements via the
-[issue tracker](https://github.com/glotzerlab/software/issues).
-See [ARCHITECTURE.md](ARCHITECTURE.md) for information on how the repository is structured,
-including how to modify the containers.
+Contributions are welcomed via [pull requests](https://github.com/glotzerlab/software/pulls). Please
+report bugs and suggest feature enhancements via the [issue
+tracker](https://github.com/glotzerlab/software/issues). See [ARCHITECTURE.md](ARCHITECTURE.md) for
+information on how the repository is structured, including how to modify the containers.

conda/.condarc

@@ -0,0 +1,2 @@
+conda-build:
+   pkg_format: 2

conda/andes.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+
+# Load modules used to build packages with native MPI support
+module reset
+module load gcc/10.3.0 openmpi/4.1.2
+
+export OUTPUT_FOLDER=/ccs/proj/mat110/software/andes/conda
+
+# Control ninja parallelism in hoomd builds
+export SLURM_CPUS_PER_TASK=8
+
+./build.sh "$@" \
+    --skip-existing \
+    --variants "{'cluster': ['andes'], 'device': ['cpu'], 'gpu_platform': ['none']}" \
+    --output-folder $OUTPUT_FOLDER
+
+chmod g-w $OUTPUT_FOLDER -R
+

conda/anvil.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+#SBATCH --job-name="glotzerlab-software build"
+#SBATCH --account=dmr140129
+#SBATCH --partition=shared
+#SBATCH --ntasks=1
+#SBATCH --cpus-per-task=32
+#SBATCH --time=8:00:00
+
+export OUTPUT_FOLDER=$PROJECT/software/conda
+
+# Load modules used to build packages with native MPI support (no CUDA).
+module reset
+module load gcc/11.2.0 openmpi/4.1.6
+
+export CC=$GCC_HOME/bin/gcc
+export CXX=$GCC_HOME/bin/g++
+
+./build.sh "$@" \
+    --skip-existing \
+    --variants "{'cluster': ['anvil'], 'device': ['cpu'], 'gpu_platform': ['none']}" \
+    --output-folder $OUTPUT_FOLDER
+
+chmod g-w $OUTPUT_FOLDER -R
+

conda/build.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+# Build package format 2
+export CONDARC=.condarc
+
+# Build on a node local SSD.
+export CONDA_BLD_PATH=`mktemp --directory conda-build-${USER}.XXXXXXXX --tmpdir`
+
+function cleanup {
+  rm -rf "$CONDA_BLD_PATH"
+}
+
+trap cleanup EXIT
+
+conda mambabuild  "$@"

conda/conda_build_config.yaml

@@ -0,0 +1,24 @@
+# From conda-forge-pinning
+zip_keys:
+  -
+    - python
+    - numpy
+
+python:
+  # part of a zip_keys: python, numpy
+  - 3.8.* *_cpython
+  - 3.9.* *_cpython
+  - 3.10.* *_cpython
+  - 3.11.* *_cpython
+  - 3.12.* *_cpython
+
+numpy:
+  # part of a zip_keys: python, numpy
+  - 1.22
+  - 1.22
+  - 1.22
+  - 1.23
+  - 1.26
+
+pybind11_abi:
+  - 4

conda/delta.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+#SBATCH --job-name="glotzerlab-software build"
+#SBATCH --account=bbgw-delta-gpu
+#SBATCH --partition=gpuA40x4
+#SBATCH --nodes=1
+#SBATCH --tasks-per-node=1
+#SBATCH --cpus-per-task=16
+#SBATCH --mem=48g
+#SBATCH --gpus=1
+#SBATCH --time=8:00:00
+
+# TODO: rewrite to use CPU account when it is renewed
+export OUTPUT_FOLDER=/projects/bbgw/software/conda
+
+# Load modules used to build packages with native MPI and CUDA support.
+module reset
+module load gcc/11.4.0 openmpi/4.1.6 cuda/12.3.0
+
+./build.sh "$@" \
+    --skip-existing \
+    --variants "{'cluster': ['delta'], 'device': ['gpu'], 'gpu_platform': ['CUDA']}" \
+    --output-folder $OUTPUT_FOLDER
+
+chmod g-w $OUTPUT_FOLDER -R
+chmod g+rX $OUTPUT_FOLDER -R

conda/frontier.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+# Load modules used to build packages with native MPI and HIP support
+module purge
+module load PrgEnv-gnu rocm/5.4.3
+module unload darshan-runtime
+
+export OUTPUT_FOLDER=/ccs/proj/mat110/software/frontier/conda
+
+# Control ninja parallelism in hoomd builds
+export SLURM_CPUS_PER_TASK=16
+
+# The cray-mpich module does not provide this, it is needed to build mpi4py from source.
+export MPICC=$CRAY_MPICH_DIR/bin/mpicc
+export CC=$GCC_PATH/bin/gcc
+export CXX=$GCC_PATH/bin/g++
+export HCC_AMDGPU_TARGET=gfx90a 
+
+./build.sh "$@" \
+    --skip-existing \
+    --variants "{'cluster': ['frontier'], 'device': ['gpu'], 'gpu_platform': ['HIP']}" \
+    --output-folder $OUTPUT_FOLDER
+
+chmod g-w $OUTPUT_FOLDER -R
+

conda/greatlakes.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+#SBATCH --job-name="glotzerlab-software build"
+#SBATCH --account=sglotzer9
+#SBATCH --partition=standard
+#SBATCH --nodes=1
+#SBATCH --tasks-per-node=1
+#SBATCH --cpus-per-task=36
+#SBATCH --mem=144g
+#SBATCH --time=8:00:00
+
+export OUTPUT_FOLDER=/nfs/turbo/glotzer/software/conda
+
+# Load modules used to build packages with native MPI and CUDA support.
+module reset
+module load gcc/10.3.0 openmpi/4.1.6 cuda/12.3.0
+
+export TMPDIR=/tmpssd
+
+./build.sh "$@" \
+    --skip-existing \
+    --variants "{'cluster': ['greatlakes'], 'device': ['gpu'], 'gpu_platform': ['CUDA']}" \
+    --output-folder $OUTPUT_FOLDER
+
+chmod g-w $OUTPUT_FOLDER -R

conda/hoomd/build_hoomd.sh

@@ -0,0 +1,18 @@
+if [[ $1 == "gpu" ]]; then
+    ENABLE_GPU="on"
+fi
+
+cmake -B build \
+      ${CMAKE_ARGS} \
+      -DENABLE_MPI=on \
+      -DENABLE_GPU=${ENABLE_GPU:-off} \
+      -DHOOMD_GPU_PLATFORM=$2 \
+      -DBUILD_TESTING=off \
+      -DENABLE_TBB=off \
+      -DENABLE_LLVM=off \
+      -DPLUGINS="" \
+      -DCMAKE_CXX_FLAGS=-march=native -DCMAKE_C_FLAGS=-march=native \
+      -GNinja
+
+ninja -C build -j $SLURM_CPUS_PER_TASK
+ninja -C build install -j $SLURM_CPUS_PER_TASK

conda/hoomd/meta.yaml

@@ -0,0 +1,48 @@
+{% set name = "hoomd" %}
+{% set version = "4.5.0" %}
+{% set build = 0 %}
+{% set sha256 = "b44f3dc9aa6b60c6bb5fe7c668d72af7eb1a52a4ebf5abe1849e9bb92da88667" %}
+
+
+package:
+  name: {{ name }}
+  version: {{ version }}
+
+source:
+  fn: {{ name }}-{{ version }}.tar.gz
+  url: https://github.com/glotzerlab/hoomd-blue/releases/download/v{{ version }}/{{ name }}-{{ version }}.tar.gz
+  sha256: {{ sha256 }}
+
+build:
+  script_env:
+    - LD_LIBRARY_PATH
+    - MPICC
+    - CC
+    - CXX
+    - CMAKE_PREFIX_PATH
+    - HIP_PLATFORM
+    - HIP_COMPILER
+    - HCC_AMDGPU_TARGET
+    - SLURM_CPUS_PER_TASK
+  number: {{ build }}
+  string: {{ cluster }}_{{ device }}_py{{ py }}h{{ PKG_HASH }}_{{ build }}
+  script: bash ${RECIPE_DIR}/build_hoomd.sh {{ device }} {{ gpu_platform }}
+
+requirements:
+  build:
+    - cmake
+    - ninja
+    - pkg-config
+
+  host:
+    - cereal
+    - eigen
+    - numpy
+    - pybind11
+    - pybind11-abi
+    - python
+
+  run:
+    - python
+    - numpy
+

conda/mpi4py/meta.yaml

@@ -0,0 +1,38 @@
+# Adapted from: https://github.com/conda-forge/mpi4py-feedstock/blob/main/recipe/meta.yaml
+
+{% set version = "3.1.5" %}
+{% set build = 0 %}
+
+package:
+  name: mpi4py
+  version: {{ version }}
+
+source:
+  - fn: mpi4py-{{ version }}.tar.gz
+    url: https://github.com/mpi4py/mpi4py/releases/download/{{ version }}/mpi4py-{{ version }}.tar.gz
+    sha256: a706e76db9255135c2fb5d1ef54cb4f7b0e4ad9e33cbada7de27626205f2a153
+
+build:
+  number: {{ build }}
+  string: {{ cluster }}_py{{ py }}h{{ PKG_HASH }}_{{ build }}
+  script_env:
+    - LD_LIBRARY_PATH
+    - MPICC
+    - CC
+    - CXX
+    - CMAKE_PREFIX_PATH
+  script:
+    - {{ PYTHON }} conf/cythonize.py  # TODO: remove for mpi4py 4.0.0
+    - {{ PYTHON }} -m pip install --no-deps . -vv
+
+requirements:
+  build:
+    - python                                 # [build_platform != target_platform]
+    - cython <3                              # [build_platform != target_platform]
+    - cross-python_{{ target_platform }}     # [build_platform != target_platform]
+  host:
+    - python
+    - pip
+    - cython <3
+  run:
+    - python