Building and running the unit tests (gnitest)

We have a set of unit tests in prov/gni/test written for the Criterion framework (https://github.com/Snaipe/Criterion). To simplify building the tests, look for a version of Criterion which is installed on your system, possibly under a libfabric-test directory somewhere. To build the tests, configure with --with-criterion=/path/to/criterion. The unit tests are run from a single program called gnitest.

A successful run of gnitest is required for all pull requests.

Criterion version	commit sha
2.3.1	73b7ac5
2.2.0	fcfcfed
1.2.2

All unit tests run on a single node; data transfer operations happen between different endpoints on the same node (in the same process). In the same directory, there is a simple script called run_gnitest that launches the job on a node of a Cray XC with the recommended arguments for aprun or srun.

Caveats

As of version 2.2.0, by default Criterion runs tests in parallel across the cores of your machine. Our unit tests were not designed to do that (e.g., because they may consume all the NIC resources on a node), so the tests must be run with concurrency 1. The run_gnitest script does this for you, but if you run gnitest by hand, you need to pass the -j1. Alternatively, you can set the environment variable CRITERION_JOBS=1 in your shell.

As of version 2.3.1, for some reason the Criterion framework runs the PRE_ALL report hook twice (from two different pids) even if -j1 is specified. To get around this, you need to set some PMI environment variables. Again, if you use the run_gnitest script, all is good. Otherwise set PMI_NO_PREINITIALIZE=1 and PMI_NO_FORK=1.

Building Criterion

Assume INSTALL_DIR is pointing to the directory in which you install Criterion.

Version 2.3.1

% git clone https://github.com/Snaipe/Criterion.git Criterion-v2.3.1
% cd Criterion-v2.3.1
% git checkout tags/v2.3.1 -b v2.3.1

To configure and install:

# Might need to load a newer version of cmake depending on your system
% module load cmake

% mkdir build

% cd build

% cmake -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR -DCMAKE_CXX_COMPILER=/opt/gcc/default/bin/g++  -DCMAKE_C_COMPILER=/opt/gcc/default/bin/gcc ..

% cmake --build .

% make install

Version 2.2.0

Depending on your system, you may need to download and install PCRE. To configure and install PCRE:

% ./configure LDFLAGS=-Wl,-rpath=$INSTALL_DIR/lib --prefix=$INSTALL_DIR

% make install

To configure and install Criterion:

# Might need to load a newer version of cmake depending on your system
% module load cmake

% mkdir build

% cd build

# Might need to specify paths to gcc/g++ depending on your system
% cmake -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR -DPCRE_INCLUDE_DIR=$INSTALL_DIR/include -DPCRE_LIBRARY=$INSTALL_DIR/lib/libpcre.so -DCMAKE_ASM_COMPILER=/opt/gcc/default/bin/gcc  -DCMAKE_CXX_COMPILER=/opt/gcc/default/bin/g++ -DCMAKE_C_COMPILER=/opt/gcc/default/bin/gcc  ..

% cmake --build .

% make install

Building the gnitest on NERSC edison/cori systems

There is a criterion v2.2.0 + patches installed on edison/cori. Use the following configure option to pick up the Criterion version installed on Cori/Edison:--with-criterion=/global/homes/h/hpp/criterion_install.

Note that owing to the way SLURM is configured at NERSC, as well as other Cray customer sites using SLURM, the following srun args option needs to be changed in the run_gnitest script:

args="-N1 --exclusive --cpu_bind=none -t00:20:00 --ntasks=1 --cpus-per-task=8"

to avoid hangs and errors due to resource allocation failures.

For those using ALPS on a system with MAPN enabled

Don't forget if you are using a Cray XC system running ALPS with MAPN enabled, you will need to add --network to the aprun command line.

For those getting "Warning: Unable to initialize DLA..."

Don't forget to export UGNI_FMA_SHARED=0.