Chroma with QUDA

Chroma with QUDA support

Chroma can use GPUs with QUDA in two different flavors.

1. Chroma with QDP++ and QUDA offload for inversions
2. Chroma with QDPJIT for full GPU offload and QUDA for inversions

Which version is optimal depends on the workload. For HMC workloads the QDPJIT version is usually preferable as otherwise the parts remaining on the CPU cause a significant slowdown.

Building Chroma with QUDA

Both version have a few dependencies required for building in common and differ in other parts. In any case we assume that you have

• MPI
• CMake (You can get a recent version from https://cmake.org/download/. The .tar.gz binary distribution unpacks in your home on linux and does not require you to build anything)
• CUDA
• autotools

available on your system.

In addition to above packages you will need

• Chroma https://github.com/JeffersonLab/chroma
• QUDA https://github.com/lattice/quda/
• QMP https://github.com/usqcd-software/qmp
• libxml2 https://github.com/GNOME/libxml2

Note that not all versions are compatible, so please check the versions specified later.

Manual build

For the QDP++ version you will also need

• QDP++ https://github.com/usqcd-software/qdpxx

and for the QDPJIT version you will need

• LLVM https://llvm.org and https://github.com/llvm/llvm-project/ (You need to compile this yourself, a pre-built binary is not sufficient)
• QDPJIT https://github.com/JeffersonLab/qdp-jit

You can get the dependencies (using shallow git clones to save bandwidth) like:

# common
git clone --branch devel https://github.com/JeffersonLab/chroma.git
git clone --branch develop https://github.com/lattice/quda.git
git clone --branch qmp2-5-2  https://github.com/usqcd-software/qmp.git
git clone --depth 1 --branch master https://github.com/GNOME/libxml2.git

# for QDP++ version
git clone --branch qdp1-46-0 https://github.com/usqcd-software/qdpxx.git
https://github.com/JeffersonLab/chroma.git

# for QDPJIT version
git clone --depth 1 --branch llvmorg-11.0.0 https://github.com/llvm/llvm-project.git
git clone --depth 1 --branch devel https://github.com/JeffersonLab/qdp-jit.git

Note that for QUDA, Chroma and QDPJIT this will check out the latest develop branch. This might occasionally not be a stable combination. The following git commits have provided a working build:

Chroma:   b65cb0ee475eb93b918b6c1f797bc819e6b4ebf2
QUDA:     aa7cad2101efd66c7506faf7d26fe8ae080eccac
QDPJIT:   52960068547f88cc9aaad99a5a19262f9e7d2cc4

You can check these out by using (in the corresponding src directory)

git checkout <commit-id>

Experimental automated build with cmake

---

OLD stuff

---

TODO Include details of setting

• <FermAct> from CLOVER/SEOPREC_CLOVER
• <AsymmetricLinop> true/false

as needed for whether we are doing asymmetric or symmetric preconditioning.

Using QMP

With QMP-2.5.1 and above, users can control the logical topology, helping improve inter/intra node layout. In addition to the regular QMP args (-geom x y z t), one can now also pass two new args, for example:

chroma -geom/qmp-geom x y z t -qmp-logic-map 3 2 1 0 -qmp-alloc-map 3 2 1 0

Where the above invocation would result in the time dimension running fastest with the x dimension running slowest.