-
Notifications
You must be signed in to change notification settings - Fork 56
New Site Configs
The instructions here describe how to generate new site configurations. In addition to configuring new production and testing systems, this is the recommended way for developers to use spack-stack locally on their Linux or MacOS workstations. In general, the recommended approach is to start with an empty/default site config (linux.default or macos.default). The instructions differ slightly for macOS and Linux and assume that the prerequisites for the platform have been installed as described below.
The instructions below are for Apple's Clang compiler on macOS and for GCC on Linux, since these are the easiest and best supported setups. Creating site configs for other compilers is more involved and not described here. It is recommended to peruse the GitHub actions scripts in .github/workflows and .github/actions in the spack-stack repository to see how automated spack-stack builds are configured for CI testing. Further, existing site configurations in configs/sites/tier{1,2} are useful references.
The first two sections describe how to create spack-stack environments on macOS and Linux starting from the default site configs (linux.default or macos.default). The last two sections describe how to add a new site config under configs/sites/tier{1,2} from these environments and how to install certain external packages on systems where sudo installations via the OS package manager are not possible.
On macOS, it is important to use certain Homebrew packages as external packages, because the native macOS packages are incomplete (e.g. missing the development header files); for example, curl, qt, etc. The instructions provided in the following have been tested extensively on many macOS installations. Occasionally, the use of external packages may lead to concretization issues in the form of duplicate packages (i.e., more than one spec per package). This is the case with bison, therefore the package should be installed by spack.
The instructions below assume that Python is built by Spack. That means that when using the Spack environments (i.e., loading the modules for building or running code), the Spack installation of Python with its available Python modules should be used to ensure consistency. However, a Homebrew Python installation may still be needed to build new Spack environments. It can also be beneficial for the user to have a version of Python installed with Homebrew that can be used for virtual environments that are completely independent of any Spack environment.
It is recommended to not use mpich or openmpi installed by Homebrew, because these packages are built using a flat namespace that is incompatible with the JEDI software. The spack-stack installations of mpich and openmpi use two-level namespaces as required.
The Mac platforms are equipped with one of two native architectures: Intel or Arm. The Arm-based Macs come with an Intel architecture emulator named Rosetta. Due to issues encountered with Rosetta we have decided to not support Rosetta meaning that support is limited to just the native (Intel and Arm) architectures. The Arm architecture is denoted by arm64 and aarch64, while the Intel architecture is denoted by x86_64 and i386.
On the Arm based Macs, a number of binaries (Terminal for example) come in a "universal form" meaning they can run as Arm or Intel. MacOS provides a utility called arch which is handy for monitoring which architecture you are running on. For example, entering arch without any arguments will return which architecture is running in your terminal window. Please take care to make sure your terminal is properly configured to run with the native architecture on your Mac.
When running with the Intel architecture, homebrew manages its downloads in /usr/local (as it has been doing in the past). When running with the Arm architecture, homebrew manages its downloads in /opt/homebrew. Other than the different prefixes for Arm versus Intel, the paths for all the pieces of a given package are identical. This separation allows for both Arm and Intel environments to exist on one machine.
For these instructions we will use the variable $HOMEBREW_ROOT to hold the prefix where homebrew manages its downloads (according to the architecture being used).
# If building on Arm architecture:
export HOMEBREW_ROOT=/opt/homebrew
# If building on Intel architecture:
export HOMEBREW_ROOT=/usr/local
Note. By default, every call to brew attempts to update the entire brew installation, which often means that existing spack-stack installations and other builds won't work anymore. With export HOMEBREW_NO_AUTO_UPDATE=1 before running brew, this automatic update is disabled.
These instructions are meant to be a reference that users can follow to set up their own system. Depending on the user's setup and needs, some steps will differ, some may not be needed and others may be missing. Also, the package versions may change over time.
Install Apple's command line utilities using the terminal found in /Applications/Utilities
xcode-select --install
sudo xcode-select --switch /Library/Developer/CommandLineTools
If you encounter build errors for gdal later on in spack-stack, you may need to install the full Xcode application and then switch xcode-select over with sudo xcode-select -s /Applications/Xcode.app/Contents/Developer (change the path if you installed Xcode somewhere else).
On Arm based Macs, prepend the $PATH environment variable to make sure that the Homebrew bin directory is found before /usr/local/bin.
export PATH=$HOMEBREW_ROOT/bin:$PATH
It is highly recommended to ensure that any remnants of a homebrew installation in /usr/local be removed on an Arm based Mac. For example, this situation can come about by migrating your old Mac (which was Intel based) to your new Mac which is Arm based.
Install the following prerequisites via Homebrew, since installing them with Spack and Apple's native clang compiler can be tricky.
brew install coreutils
# For now, use gcc@12
brew install gcc@12
brew install git
brew install git-lfs
brew install lmod
brew install wget
brew install bash
brew install curl
brew install cmake
brew install openssl
brew install libiconv
# Note - need to pin to version 5
brew install qt@5
brew install m4
# Note - For Sonoma only: install pkg-config, diffutils and clingo
brew install pkg-config
brew install diffutils
brew install clingo
On an Intel based Mac, regardless of MacOS version, you will need to also install pkg-config using homebrew. This is done to work around an issue where libraries (eg, openssl) cannot be properly found during code compilation.
# Intel based Mac only
brew install pkg-config
For spack-stack-1.8.0 and newer, you must be using cmake 3.26+. Make sure you upgrade cmake in homebrew.
brew upgrade cmake
After installing bash with homebrew, you need to change your terminal application's default command to use :code:$HOMEBREW_ROOT/bin/bash. For example, with iterm2, you can click on the Preferences item in the iTerm2 menu. Then click on the Profiles tab and enter $HOMEBREW_ROOT/bin/bash in the Command box. This is done to avoid issues with the macOS System Integrity Protection (SIP) mechanism when running bash scripts. See https://support.apple.com/en-us/HT204899 for more details about SIP.
It is recommended to quit the terminal window at this point and then start up a fresh terminal window to make sure you proceed using a terminal that is running the $HOMEBREW_ROOT/bin/bash shell.
Activate the lua module environment (note: This is not persistent and must be done at the beginning of each session you intend to use spack-stack modules).
source $HOMEBREW_ROOT/opt/lmod/init/profile
Install xquartz using the provided binary at https://www.xquartz.org. This is required for forwarding of remote X displays, and for displaying the ecflow GUI, amongst others.
Optional: Install MacTeX if planning to build the jedi-tools environment with LaTeX/PDF support. If the jedi-tools application is built with variant +latex to enable building LaTeX/PDF documentation, install MacTeX and configure your shell to have it in the search path, for example:
export PATH="/usr/local/texlive/2023/bin/universal-darwin:$PATH"
Remember to activate the lua module environment and have MacTeX in your search path, if applicable. It is also recommended to increase the stacksize limit to 65Kb using ulimit -S -s unlimited.
You will need to clone spack-stack (selecting your desired spack-stack branch) and its dependencies and activate the spack-stack tool. It is also a good idea to save the directory in your environment for later use.
git clone [-b develop OR release/branch-name] --recurse-submodules https://github.com/jcsda/spack-stack
cd spack-stack
source setup.sh
Create a pre-configured environment with a default (nearly empty) site config and activate it (optional: decorate bash prompt with environment name; warning: this can scramble the prompt for long lines). The choice of the template depends on the applications you want to run, see here and configs/templates/ in the spack-stack repo for the available options.
spack stack create env --site macos.default [--template unified-dev] --name unified-env.mymacos --compiler=apple-clang
cd envs/unified-env.mymacos/
spack env activate [-p] .
Ignore any warnings that packages_apple-clang.yaml cannot be found in the site config directory.
Still in the environment directory, temporarily set environment variable SPACK_SYSTEM_CONFIG_PATH to modify site config files in site und unset SPACK_DISABLE_LOCAL_CONFIG.
unset SPACK_DISABLE_LOCAL_CONFIG
export SPACK_SYSTEM_CONFIG_PATH="$PWD/site"
Find external packages, add to site config's packages.yaml. If an external's bin directory hasn't been added to $PATH, you will need to prefix the command (PATH=/missing/path/to/bin:$PATH spack external find ...).
spack external find --scope system \
--exclude bison --exclude openssl \
--exclude python --exclude gettext \
--exclude m4
spack external find --scope system grep
spack external find --scope system perl
spack external find --scope system wget
Some dependency paths may be complicated by the use of homebrew casks. These dependencies require PATH modification to enable spack external find.
PATH="$HOMEBREW_ROOT/opt/libiconv/bin:$PATH" \
spack external find --scope system libiconv
PATH="$HOMEBREW_ROOT/opt/curl/bin:$PATH" \
spack external find --scope system curl
PATH="$HOMEBREW_ROOT/opt/qt5/bin:$PATH" \
spack external find --scope system qt
PATH="$HOMEBREW_ROOT/opt/m4/bin:$PATH" \
spack external find --scope system m4
Optional, only if planning to build jedi-tools environment with LaTeX support. The texlive bin directory must have been added to PATH (see above)
spack external find --scope system texlive
On an Intel based Mac, you need to add the following spack config command to prevent spack from building pkg-config. This will force spack to use the pkg-config installed by homebrew (see above).
# Intel based Mac only
spack config --scope system add packages:pkg-config:buildable:false
Next, find compilers, add to site/packages.yaml as externals.
spack compiler find --scope system
Note. When using apple-clang@15.x (or newer) compilers, you need to manually add the following ldflags spec in the site/packages.yaml file. There are known issues with new features in the Apple linker/loader that comes with the 15.x compiler set, and this change tells the linker/loader to use its legacy features which work fine. Apple is aware of this issue (Apple ticket number FB13208302) and working on a solution, so this is a temporary workaround that will be removed once the linker/loader issues are repaired.
apple-clang:
externals:
- spec: apple-clang@=15.0.0
prefix: /usr
extra_attributes:
compilers:
c: /usr/bin/clang
cxx: /usr/bin/clang++
flags:
ldflags: '-Wl,-ld_classic' # Add this ldflags spec
Do not forget to unset the SPACK_SYSTEM_CONFIG_PATH environment variable and restore the SPACK_DISABLE_LOCAL_CONFIG variable before the next steps.
export SPACK_DISABLE_LOCAL_CONFIG=true
unset SPACK_SYSTEM_CONFIG_PATH
Set default MPI library (make sure to use the desired openmpi version)
spack config add "packages:mpi:require:['openmpi@5.0.8']"
If needed, edit site config files and common config files, for example to remove duplicate versions of external packages that are unwanted, add additional specs, etc. (spack.yaml, common/*.yaml, site/*.yaml).
Process the specs and install. It is recommended to save the output of concretize in a log file and inspect that log file using the show_duplicate_packages.py duplicate checker utility. This is done to find and eliminate duplicate package specifications which can cause issues at the module creation step below. Note that in the unified environment, there may be deliberate duplicates; consult the specs in spack.yaml and the GitHub action workflows to determine which ones are desired. Further, the Spack extension spack stack check-preferred-compiler checks that the explicitly preferred/required compilers for the environment and for individual packages are being used.
spack concretize --force --fresh 2>&1 | tee log.concretize
${SPACK_STACK_DIR}/util/show_duplicate_packages.py
spack stack check-preferred-compiler
spack install [--verbose] [--fail-fast] 2>&1 | tee log.install
Create lmod module files.
spack module lmod refresh
Create meta-modules for the compiler and mpi provider.
spack stack setup-meta-modules
Unlike preconfigured environments and Linux environments, MacOS users typically need to activate lmod's module tool within each shell session. This can be done by running source $HOMEBREW_ROOT/opt/lmod/init/profile.
You now have a spack-stack environment that can be accessed by running module use ${SPACK_STACK_DIR}/envs/unified-env.mymacos/modules/Core.
Note. Some Linux systems do not support recent lua/lmod environment modules, which are default in the spack-stack site configs. The instructions below therefore use tcl/tk environment modules.
The following instructions were used to prepare a basic Red Hat 9 system as it is available on Amazon Web Services to build and install all of the environments available in spack-stack. These instructions are also valid for various derivatives of Red Hat Linux.
Install basic OS packages as root. Do not install MPI with yum, this will be done with Spack.
sudo su
yum -y update
# Compilers - this includes environment module support
yum -y install gcc-toolset-13-gcc-c++
yum -y install gcc-toolset-13-gcc-gfortran
yum -y install gcc-toolset-13-gdb
yum -y install gcc-toolset-13-gcc-plugin-devel
# Miscellaneous
yum -y install binutils-devel
yum -y install m4
yum -y install wget
yum -y install cmake
yum -y install git
yum -y install git-lfs
yum -y install bash-completion
yum -y install bzip2 bzip2-devel
yum -y install unzip
yum -y install patch
yum -y install automake
yum -y install xorg-x11-xauth
yum -y install xterm
yum -y install perl-IPC-Cmd
yum -y install gettext-devel
yum -y install texlive
yum -y install bison
# For screen utility (optional)
yum -y remove https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
yum -y update --nobest
yum -y install screen
# Exit root session
exit
Log out and back in to be able to use the tcl/tk environment modules.
The following instructions were used to prepare a basic Ubuntu 20.04 or 22.04 LTS system as it is available on Amazon Web Services to build and install all of the environments available in spack-stack.
Install basic OS packages as root. Do not install MPI with yum, this will be done with Spack.
sudo su
apt-get update
apt-get upgrade
# Compilers. Note you can select other versions of gcc, g++, and gfortran by changing the version number.
apt install -y gcc-13 g++-13 gfortran-13
apt install -y gdb gcc-plugin-dev # check that gcc-plugin-dev is the correct version for your compiler
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-13 100 && \
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 100 && \
update-alternatives --install /usr/bin/gfortran gfortran /usr/bin/gfortran-13 100
# Environment module support
# Note: lmod is available in 22.04, but is out of date: https://github.com/JCSDA/spack-stack/issues/593
apt install -y environment-modules
# Miscellaneous
apt install -y build-essential
apt install -y libkrb5-dev
apt install -y m4
apt install -y git
apt install -y git-lfs
apt install -y bzip2
apt install -y unzip
apt install -y automake
apt install -y autopoint
apt install -y gettext
apt install -y texlive
apt install -y libcurl4-openssl-dev
apt install -y libssl-dev
apt install -y wget
# For building documentation such as jedi-docs using LaTeX
apt install -y texlive
# Exit root session
exit
Log out and back in to be able to use the environment modules.
It is recommended to increase the stacksize limit by using ulimit -S -s unlimited, and to test if the module environment functions correctly (module available).
You will need to clone spack-stack (selecting your desired spack-stack branch) and its dependencies and activate the spack-stack tool. It is also a good idea to save the directory in your environment for later use.
git clone [-b develop OR release/branch-name] --recurse-submodules https://github.com/jcsda/spack-stack
cd spack-stack
source setup.sh
Create a pre-configured environment with a default (nearly empty) site config and activate it (optional: decorate bash prompt with environment name; warning: this can scramble the prompt for long lines). The choice of the template depends on the applications you want to run, see here and configs/templates/ in the spack-stack repo for the available options.
spack stack create env --site linux.default [--template unified-dev] --name unified-env.mylinux --compiler=gcc
cd envs/unified-env.mylinux/
spack env activate [-p] .
Ignore any warnings that packages_apple-clang.yaml cannot be found in the site config directory.
Still in the environment directory, temporarily set environment variable SPACK_SYSTEM_CONFIG_PATH to modify site config files in site und unset SPACK_DISABLE_LOCAL_CONFIG.
unset SPACK_DISABLE_LOCAL_CONFIG
export SPACK_SYSTEM_CONFIG_PATH="$PWD/site"
Find external packages, add to site config's packages.yaml. If an external's bin directory hasn't been added to $PATH, you will need to prefix the command (PATH=/missing/path/to/bin:$PATH spack external find ...).
spack external find --scope system \
--exclude bison --exclude meson \
--exclude curl --exclude openssl \
--exclude openssh --exclude python
spack external find --scope system grep
spack external find --scope system wget
# Note - only needed for generating documentation
spack external find --scope system texlive
Next, find compilers, add to site/packages.yaml as externals. If you have multiple versions of a compiler you will need to manually remove the unwanted versions from site/packages.yaml to prevent their use.
# Add compilers to the top of site/packages.yaml.
spack compiler find --scope system
# Edit site/packages.yaml to delete or comment unwanted compiler versions.
Do not forget to unset the SPACK_SYSTEM_CONFIG_PATH environment variable and restore the SPACK_DISABLE_LOCAL_CONFIG variable before the next steps.
export SPACK_DISABLE_LOCAL_CONFIG=true
unset SPACK_SYSTEM_CONFIG_PATH
Set default MPI library (make sure to use the desired openmpi version)
# Configure preferred mpi for your environment
spack config add "packages:mpi:require:['openmpi@5.0.8']"
If you have manually installed lmod, you will need to update the site module configuration to use lmod instead of tcl. Skip this step if you followed the Ubuntu or Red Hat instructions above.
sed -i 's/tcl/lmod/g' site/modules.yaml
If needed, edit site config files and common config files, for example to remove duplicate versions of external packages that are unwanted, add additional specs, etc. (spack.yaml, common/*.yaml, site/*.yaml).
Process the specs and install. It is recommended to save the output of concretize in a log file and inspect that log file using the show_duplicate_packages.py duplicate checker utility. This is done to find and eliminate duplicate package specifications which can cause issues at the module creation step below. Note that in the unified environment, there may be deliberate duplicates; consult the specs in spack.yaml and the GitHub action workflows to determine which ones are desired. Further, the Spack extension spack stack check-preferred-compiler checks that the explicitly preferred/required compilers for the environment and for individual packages are being used.
spack concretize --force --fresh 2>&1 | tee log.concretize
${SPACK_STACK_DIR}/util/show_duplicate_packages.py
spack stack check-preferred-compiler
spack install [--verbose] [--fail-fast] 2>&1 | tee log.install
Create tcl module files (replace tcl with lmod if you have manually installed lmod).
spack module tcl refresh
Create meta-modules for the compiler and mpi provider.
spack stack setup-meta-modules
You now have a spack-stack environment that can be accessed by running module use ${SPACK_STACK_DIR}/envs/unified-env.mylinux/modules/Core.
Warning. Support for Nvidia compilers is experimental and limited to a small subset of packages of the unified environment. The Nvidia compilers are known for their bugs and flaws, and many packages simply don't build. The strategy for building environments with Nvidia is therefore the opposite of what it is with other supported compilers.
In order to build environments with the Nvidia compilers, a different approach is needed than for our main compilers (GNU, Intel). Since many packages do not build with the Nvidia compilers, the idea is to provide as many packages as possible as external packages or build them with gcc. Because our Spack extension spack stack setup-meta-modules was only recently updated to support combinations of modules built with different compilers, and these instructions have NOT yet been updated accordingly, we are still subject to former restrictions that packages not being built with the Nvidia compilers need to fulfil the two following criteria:
-
The package is used as a utility to build or run the code, but not linked into the application (this may be overly restrictive, but it ensures that the application will be able to leverage all of Nvidia's features, for example run on GPUs).
-
One of the following applies: Either the package is installed outside of the spack-stack environment and made available as an external package. A typical use case is a package that is installed using the OS package manager. Or the package is built with another compiler (typically
gcc) within the same environment, and no module is generated for the package. The Spack modulefile generator in this case ensures that other packages that depend on this particular package have the necessary paths in their own modules. If thegcccompiler itself requires additionalPATH,LD_LIBRARY_PATH, ... variables to be set, then these can be set in the Spack compiler config for the Nvidia compiler (similar to how we configure thegccbackend for the Intel compiler).
With the recently-added support for building packages with different compilers, we will be able to clean up these instructions to use the Nvidia compiler where possible and fall back to the GNU compiler where necessary, and reduce the need to install packages from the package manager. These instructions will be updated with these improvements.
With all of that in mind, the following instructions were tested on an Azure VM running Ubuntu 24.04 to build an environment based on template jedi-mpas-nvidia-dev. These instructions follow the one-off setup instructions above and replace the instructions to create new environments above. The instructions worked as of spack-stack commit 26901af but are very fragile w.r.t. changes in the underlying OS, the Nvidia HPC SDK version, and the spack-stack version.
Follow the instructions above to install the basic packages. In addition, install the following packages using apt:
sudo su
apt update
apt install -y cmake
apt install -y pkg-config
apt install -y libtool
apt install -y libbsd-dev
apt install -y python3-dev
apt install -y openmpi
exit
Download the latest version of the Nvidia HPC SDK following the instructions on the Nvidia website. For nvhpc@24.9:
curl https://developer.download.nvidia.com/hpc-sdk/ubuntu/DEB-GPG-KEY-NVIDIA-HPC-SDK | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-hpcsdk-archive-keyring.gpg
echo 'deb [signed-by=/usr/share/keyrings/nvidia-hpcsdk-archive-keyring.gpg] https://developer.download.nvidia.com/hpc-sdk/ubuntu/amd64 /' | sudo tee /etc/apt/sources.list.d/nvhpc.list
sudo su
apt update
apt install -y nvhpc-24-9
exit
Optionally, to run code that may use the CUDA runtime libraries, also install:
sudo su
apt install -y ubuntu-drivers-common
ubuntu-drivers install nvidia:550
exit
Load the correct module shipped with nvhpc-24-9. Note that this is only required for spack to detect the compiler and openmpi library during the environment configuration below. It is not required when using the new environment to compile code.
module purge
module use /opt/nvidia/hpc_sdk/modulefiles
module load nvhpc-openmpi3/24.9
Clone spack-stack (selecting your desired spack-stack branch) and its dependencies and activate the spack-stack tool.
git clone [-b develop OR release/branch-name] --recurse-submodules https://github.com/jcsda/spack-stack
cd spack-stack
source setup.sh
Create a pre-configured environment with the default (nearly empty) site config for Linux and activate it (optional: decorate bash prompt with environment name). At this point, only the jedi-mpas-nvidia-dev template is supported.
spack stack create env --site linux.default --template jedi-mpas-nvidia-dev --name jedi-mpas-nvidia-env --compiler=nvhpc
cd envs/jedi-mpas-nvidia-env/
spack env activate [-p] .
Ignore any warnings that packages_nvhpc.yaml cannot be found in the site config directory.
Still in the environment directory, temporarily set environment variable SPACK_SYSTEM_CONFIG_PATH to modify site config files in site und unset SPACK_DISABLE_LOCAL_CONFIG.
unset SPACK_DISABLE_LOCAL_CONFIG
export SPACK_SYSTEM_CONFIG_PATH="$PWD/site"
Find external packages, add to site config's packages.yaml. If an external's bin directory hasn't been added to $PATH, you will need to prefix the command (PATH=/missing/path/to/bin:$PATH spack external find ...).
spack external find --scope system \
--exclude bison --exclude cmake \
--exclude curl --exclude openssl \
--exclude openssh --exclude python
spack external find --scope system wget
spack external find --scope system openmpi
spack external find --scope system python
spack external find --scope system curl
spack external find --scope system cmake
spack external find --scope system xz
Next, find compilers, add to site/packages.yaml as externals.
spack compiler find --scope system
Unset the SPACK_SYSTEM_CONFIG_PATH environment variable and restore the SPACK_DISABLE_LOCAL_CONFIG variable.
export SPACK_DISABLE_LOCAL_CONFIG=true
unset SPACK_SYSTEM_CONFIG_PATH
Still in the environment directory, add the following block to spack.yaml (pay attention to the correct indentation, it should be at the same level as specs:):
packages:
c:
prefer: ['%nvhpc']
cxx:
prefer: ['%nvhpc']
fortran:
prefer: ['%nvhpc']
all:
providers:
mpi: [openmpi@3.1.5]
zlib-api: [zlib]
blas: [nvhpc]
nvhpc:
buildable: false
externals:
- spec: nvhpc@24.9 %nvhpc
modules:
- nvhpc/24.9
python:
buildable: false
require:
- '@3.12.3'
curl:
buildable: false
cmake:
buildable: false
pkg-config:
buildable: false
libbsd:
buildable: false
xz:
buildable: false
Then, add the following code block to site/packages.yaml (because spack external find doesn't appear to work for this library):
libbsd:
externals:
- spec: libbsd@0.12.1
prefix: /usr
If you have manually installed lmod, you will need to update the site module configuration to use lmod instead of tcl. Skip this step if you followed the Ubuntu instructions above.
sed -i 's/tcl/lmod/g' site/modules.yaml
It is recommended to save the output of concretize in a log file and inspect that log file using the show_duplicate_packages.py duplicate checker utility. This is done to find and eliminate duplicate package specifications which can cause issues at the module creation step below. Note that in the unified environment, there may be deliberate duplicates; consult the specs in spack.yaml and the GitHub action workflows to determine which ones are desired. Further, the Spack extension spack stack check-preferred-compiler checks that the explicitly preferred/required compilers for the environment and for individual packages are being used.
spack concretize --force --fresh 2>&1 | tee log.concretize
${SPACK_STACK_DIR}/util/show_duplicate_packages.py
spack stack check-preferred-compiler
spack install [--verbose] [--fail-fast] 2>&1 | tee log.install
Create tcl module files (replace tcl with lmod if you have manually installed lmod).
spack module tcl refresh
Create meta-modules for the compiler and mpi provider.
spack stack setup-meta-modules
You now have a spack-stack environment that can be accessed by running module use ${SPACK_STACK_DIR}/envs/jedi-mpas-nvidia-env/modules/Core.
Warning. The instructions above may help to build spack-stack with the Nvidia compilers. Testing of building and running JEDI is ongoing; initial results suggest some of the low-level JEDI software components can be compiled with the Nvidia compilers, but many ctests segfault.
New site configurations under configs/sites/tier{1,2} can be created in different ways. One method is to start from an existing site config for a system that is similar to the new system, and modify the external packages, compilers, paths etc. manually. Another method is to follow the above instructions to install a spack-stack environment starting from the default (empty) site config. Either method requires familiarity with Spack and goes beyond what we can describe in detail here.
In the latter case, the above instructions ensure that most of the information is already contained in the correct files. A viable approach is to create the new site config directory under configs/sites/tier{1,2} and copy the files from envs/my-newly-created-env/site/ to this directory. If a site needs to support environments for multiple compilers, then the site's packages.yaml must be renamed to packages_<compiler>.yaml or packages_<compiler>-<version>.yaml.
Often, a combination of the two approaches works best: starting from a default (empty) site config, create a first functional environment, copy the contents over to the new, permanent site config directory, then compare to an existing site config with a similar setup.
In cases where the native OS package manager can't be used to install prerequisites (external packages), manual installations are needed. Several packages that are typically required in spack-stack environments and known to be difficult to install via Spack are covered in the following subsections.
Building git-lfs with spack isn't straightforward as it requires go-bootstrap and go language support, which many compilers don't build correctly. We therefore recommend treating git-lfs as an external package. On many of the HPC systems, it is already available as a separate module or as part of a git module. On macOS and Linux, it can be installed using brew or other package managers (see above). The following instructions install git-lfs on a CentOS 7.9 system from the OS rpm:
module purge
cd /my/path/to/spack-stack/
mkdir -p git-lfs-2.10.0/src
cd git-lfs-2.10.0/src
wget --content-disposition https://packagecloud.io/github/git-lfs/packages/el/7/git-lfs-2.10.0-1.el7.x86_64.rpm/download.rpm
rpm2cpio git-lfs-2.10.0-1.el7.x86_64.rpm | cpio -idmv
mv usr/* ../
Following this "installation", create a module from the following template:
#%Module1.0
module-whatis "Provides git-lfs-2.10.0 for use with spack."
conflict git-lfs
proc ModulesHelp { } {
puts stderr "Provides git-lfs-2.10.0 for use with spack."
}
# Set this value
set GITLFS_PATH "/my/path/to/spack-stack/git-lfs-2.10.0"
prepend-path PATH "${GITLFS_PATH}/bin"
prepend-path MANPATH "${GITLFS_PATH}/share/man"
Building qt with spack isn't straightforward as it requires many libraries related to the graphical desktop that are often tied to the operating system, and which many compilers don't build correctly. We therefore recommend treating qt as an external package. On many of the HPC systems, it is already available as a separate module or provided by the operating system. On macOS and Linux, it can be installed using brew or other package managers (see above).
On HPC systems without a sufficient Qt5 installation, we install it outside of spack with the default OS compiler and then point to it in the site's packages.yaml. The following instructions install qt@5.15.2 on Discover SCU16 in /discover/swdev/jcsda/spack-stack/qt-5.15.2/5.15.2/gcc_64.
mkdir -p /discover/swdev/jcsda/spack-stack/qt-5.15.2/src
cd /discover/swdev/jcsda/spack-stack/qt-5.15.2/src
wget --no-check-certificate http://download.qt.io/official_releases/online_installers/qt-unified-linux-x64-online.run
chmod u+x qt-unified-linux-x64-online.run
./qt-unified-linux-x64-online.run
Sign into Qt, select customized installation, choose qt@5.15.2 only (uncheck all other boxes) and set install prefix to /discover/swdev/jcsda/spack-stack/qt-5.15.2. After the successful installation, create module /discover/swdev/jcsda/spack-stack/modulefiles/qt/5.15.2 from the following template:
#%Module1.0
module-whatis "Provides a qt-5.15.2 installation for use with spack."
conflict qt
proc ModulesHelp { } {
puts stderr "Provides a qt-5.15.2 installation for use with spack."
}
# Set this value
set QT_PATH "/discover/swdev/jcsda/spack-stack/qt-5.15.2/5.15.2/gcc_64"
prepend-path PATH "${QT_PATH}/bin"
prepend-path LD_LIBRARY_PATH "${QT_PATH}/lib"
prepend-path LIBRARY_PATH "${QT_PATH}/lib"
prepend-path CPATH "${QT_PATH}/include"
prepend-path CMAKE_PREFIX_PATH "${QT_PATH}"
Notes. The dependency on qt is introduced by ecflow, which at present requires using qt@5 - earlier or newer versions will not work. If ./qt-unified-linux-x64-online.run fails to start with the error qt.qpa.xcb: could not connect to display and a role account is being used, follow the procedure described in https://www.thegeekdiary.com/how-to-set-x11-forwarding-export-remote-display-for-users-who-switch-accounts-using-sudo to export the display. A possible warning xauth: file /ncrc/home1/role.epic/.Xauthority does not exist can be ignored, since this file gets created by the xauth command.
Building texlive isn't straightforward as it has many dependencies. Since it is only used to generated documentation for spack-stack (and other projects), i.e. not to compile any code, it makes no sense to build it with spack. We therefore require texlive or any other compatible TeX/LaTeX distribution as an external package.
On many of the HPC systems, it is already available as a separate module or as part of the default operating system. On macOS, the MacTeX distribution provides a full and easy-to-install TeX/LaTeX environment (see above). On Linux, texlive can be installed using the default package manager (see above).
If a different GCC compiler is required than currently available, then a manual installation of GCC is a viable option. An alternative not explorer here is to perform an installation via Spack using the existing GCC compiler. Example instructions for a manual installation:
module purge
mkdir -p /path/to/spack-stack/gcc-13.4.0/src
cd /path/to/spack-stack/gcc-13.4.0/src
wget https://github.com/gcc-mirror/gcc/archive/refs/tags/releases/gcc-13.4.0.tar.gz
tar -xf gcc-13.4.0.tar.gz
cd gcc-releases-gcc-13.4.0
./contrib/download_prerequisites 2>&1 | tee log.download_prerequisites
./configure --prefix=/path/to/spack-stack/gcc-13.4.0 --disable-multilib 2>&1 | tee log.configure
make VERBOSE=1 -j8 2>&1 | tee log.make
make check 2>&1 | tee log.check
make install 2>&1 | tee log.install
Next, create modulefile /path/to/spack-stack/gcc-13.4.0/modulefiles/gcc/13.4.0.lua with the following contents:
help([[
]])
local pkgName = myModuleName()
local pkgVersion = myModuleVersion()
local pkgNameVer = myModuleFullName()
-- conflicts
-- prerequisite modules
load("slurm")
prereq("slurm")
-- compiler environment variables
setenv("F77", "/path/to/spack-stack/gcc-13.4.0/bin/gfortran")
setenv("FC", "/path/to/spack-stack/gcc-13.4.0/bin/gfortran")
setenv("CC", "/path/to/spack-stack/gcc-13.4.0/bin/gcc")
setenv("CXX", "/path/to/spack-stack/gcc-13.4.0/bin/g++")
setenv("SERIAL_F77", "/path/to/spack-stack/gcc-13.4.0/bin/gfortran")
setenv("SERIAL_FC", "/path/to/spack-stack/gcc-13.4.0/bin/gfortran")
setenv("SERIAL_CC", "/path/to/spack-stack/gcc-13.4.0/bin/gcc")
setenv("SERIAL_CXX", "/path/to/spack-stack/gcc-13.4.0/bin/g++")
-- compiler flags and other environment variables
prepend_path("PATH", "/path/to/spack-stack/gcc-13.4.0/bin")
prepend_path("CPATH", "/path/to/spack-stack/gcc-13.4.0/include")
prepend_path("LD_LIBRARY_PATH", "/path/to/spack-stack/gcc-13.4.0/lib/gcc/x86_64-pc-linux-gnu/13.4.0")
prepend_path("LD_LIBRARY_PATH", "/path/to/spack-stack/gcc-13.4.0/lib64")
-- module show info
whatis("Name: " .. pkgName)
whatis("Version: " .. pkgVersion)
whatis("Category: compiler")
whatis("Description: " .. pkgName)
If no suitable OpenMPI provider is available, and/or a Spack installation does not function correctly, the OpenMPI package can be installed manually. Configuring OpenMPI correctly is an advanced topic. There are many options to choose from during the ./configure call. Sometimes, the correct fabric libraries must be loaded beforehand. In any case, knowledge about the underlying fabric on the system is required.
The following instructions install OpenMPI 4.1.8 on a system with a Mellanox interconnect using the previously installed GCC 13.4.0 compiler:
module purge
module load slurm
module use /path/to/spack-stack/gcc-13.4.0/modulefiles
module load gcc/13.4.0
mkdir -p /path/to/spack-stack/openmpi-4.1.8/gcc-13.4.0/src
cd /path/to/spack-stack/openmpi-4.1.8/gcc-13.4.0/src
wget https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.8.tar.gz
tar -xf openmpi-4.1.8.tar.gz
cd openmpi-4.1.8
# Presumably all arguments except --prefix and --with-platform are unnecessary
./configure \
--prefix=/path/to/spack-stack/openmpi-4.1.8/gcc-13.4.0 \
--with-platform=contrib/platform/mellanox/optimized 2>&1 | tee log.configure
make VERBOSE=1 -j8 2>&1 | tee log.make
make check 2>&1 | tee log.check
make install 2>&1 | tee log.install
Next, create modulefile /path/to/spack-stack/openmpi/4.1.8/gcc-13.4.0/modulefiles/openmpi/4.1.8.lua with the following contents:
help([[
]])
local pkgName = myModuleName()
local pkgVersion = myModuleVersion()
local pkgNameVer = myModuleFullName()
-- conflicts
-- prerequisite modules
load("slurm")
prereq("slurm")
load("gcc/13.4.0")
prereq("gcc/13.4.0")
-- compiler environment variables
setenv("MPICC", "/path/to/spack-stack/openmpi-4.1.8/gcc-13.4.0/bin/mpicc")
setenv("MPICXX", "/path/to/spack-stack/openmpi-4.1.8/gcc-13.4.0/bin/mpicxx")
setenv("MPIF77", "/path/to/spack-stack/openmpi-4.1.8/gcc-13.4.0/bin/mpif77")
setenv("MPIF90", "/path/to/spack-stack/openmpi-4.1.8/gcc-13.4.0/bin/mpif90")
-- compiler flags and other environment variables
prepend_path("PATH", "/path/to/spack-stack/openmpi-4.1.8/gcc-13.4.0/bin")
prepend_path("CPATH", "/path/to/spack-stack/openmpi-4.1.8/gcc-13.4.0/include")
prepend_path("LD_LIBRARY_PATH", "/path/to/spack-stack/openmpi-4.1.8/gcc-13.4.0/lib")
prepend_path("LD_LIBRARY_PATH", "/path/to/spack-stack/openmpi-4.1.8/gcc-13.4.0/lib64")
setenv("OMPI_MCA_rmaps_base_oversubscribe", "1")
setenv("PRTE_MCA_rmaps_default_mapping_policy", ":oversubscribe")
-- module show info
whatis("Name: " .. pkgName)
whatis("Version: " .. pkgVersion)
whatis("Category: compiler")
whatis("Description: " .. pkgName)