The spack extension of C2SM and MCH

Spack is the package manager used by C2SM and MeteoSwiss to install and deploy software on supercomputers, local machines and the cloud.

Documentations

Infos about c2sm-supported software and machines

• spack-c2sm latest
• spack-c2sm v0.20.1.4
• spack-c2sm v0.20.1.3
• spack-c2sm v0.20.1.0
• spack-c2sm v0.18.x [deprecated]

General infos about spack

• Official spack v0.21.1
• Official spack v0.20.1
• Official spack v0.18.1 [deprecated]

The first 3 numbers of every spack-c2sm version match with the version of spack it uses as a submodule.

Workflow for Users

We suggest local/individual spack instances and the use of spack environments.

Clone the repository

git clone --depth 1 --recurse-submodules --shallow-submodules -b v0.21.1.3 https://github.com/C2SM/spack-c2sm.git

Setup the shell environment and optionally specify an upstream, where spack will look for installed software, i.e.

. spack-c2sm/setup-env.sh
. spack-c2sm/setup-env.sh /user-environment
. spack-c2sm/setup-env.sh /mch-environment/v6
. spack-c2sm/setup-env.sh /mch-environment/v7

Sourcing this file will put the spack command in your PATH, set up your MODULEPATH to use Spack’s packages, and add other useful shell integration for certain commands, environments, and modules. For bash, it also sets up tab completion. (source: spack docu)

Optionally activate a spack environment

spack env activate <path_to_env>

and starts exploring

spack info <package>
spack spec <spec>

and installing

spack install <spec>
spack dev-build <spec>

packages.

Updating spack-c2sm is in the hands of the user.

git pull
git submodule update --recursive

Before an update we advice to clean your instance

spack uninstall -a
spack clean -a
rm -rf ~/.spack

After an update we advice to rebuild packages, preferably in a new shell so that no outdated shell variables are retained.

Workflow for Local Spack Development (Linting & Testing)

Create a virtual environment

python3 -m venv .venv
source .venv/bin/activate

Install pinned dev tools

pip install --upgrade pip
pip install -r requirements/dev.txt
pip install -r requirements/test.txt

Install pre-commit hooks locally (optional)

pre-commit install

Hooks run automatically on git commit

Run all hooks manually (recommended before push)

pre-commit run --all-files

• Lints and auto-fixes safe issues (like unused imports)
• Checks YAML, formatting, and other configured hooks
• Shows errors/warnings for anything that cannot be auto-fixed

Testing strategy

For more information on the different types of tests see the README in the test directory.

PR tests

To merge PRs into the main branch we require short (taking ~5 minutes) unit and integration tests to run successfully. These tests are ran via a GitHub Action that runs automatically after any push to a PR branch.

Nightly tests

Each day at 3 a.m. system tests run on Säntis (via CSCS CI) and Balfrin (via Jenkins) to test the installation of different packages with Spack on the systems. Since these tests may take longer to complete we decided to not require them to merge PRs, however special comments can still be used on PRs to trigger them nonetheless:

• For Säntis: type cscs-ci run in a PR comment.
• For Balfrin: type launch jenkins in a PR comment.

Releases/Tags

Release tags are created by the Spack-Admin GitHub Team as needed or upon request. The creation of a new release tag is coordinated within the admin team.

When creating a new release tag please follow the naming convention below:

v${SPACK_VERSION}.${SPACK_C2SM_VERSION}

Where SPACK_VERSION corresponds to the upstream Spack version this repo is based on, e.g. SPACK_VERSION=0.22.2. SPACK_C2SM_VERSION is the number of releases based on a specific SPACK_VERSION, starting from 0, which corresponds to the release adapting spack-c2sm to a new upstream Spack version. So, for example v0.22.2.5 is the fifth (sixth if you count the transitional release v0.22.2.0) spack-c2sm release based on Spack v0.22.2.

Command cheat sheet

	Command
Clone	git clone --depth 1 --recurse-submodules --shallow-submodules -b <branch/tag> https://github.com/C2SM/spack-c2sm.git
Update	git pull git submodule update --recursive
Load	. spack-c2sm/setup-env.sh to run without an upstream or . spack-c2sm/setup-env.sh /user-environment to use /user-environment as an upsream spack compiler find autodetects compilers spack external find --all autodetects externally installed packages
Clean	spack uninstall -a uninstalls all packages spack clean -a cleans all misc caches

Spec syntax: <package>@<version>%<compiler>+<variant> ~<variant>^<sub-package> +<sub-package-variant><compiler flags>

	Command
Find	spack find lists all installed packages. spack find <spec> lists all installed packages that match the spec.
Info	spack info <package>
Spec	spack spec <spec> concretizes abstract spec (unspecfied variant = any) Spack is not required to use the default of an unspecified variant. The default value is only a tiebreaker for the concretizer.
Install	spack install <spec>
Locate	spack location --install-dir <spec> prints location of all installs that satisfy the spec
Load env	spack load <spec> loads run environment
Activate env	spack env activate <env_name>
Deactivate env	spack deactivate