Building running flatfoil on summit

Log on to summit

$ mkdir src
$ cd src
$ git clone git@github.com:psc-code/psc # should end up on the master branch

If you want collisions on, apply this patch:

diff --git a/src/psc_flatfoil_yz.cxx b/src/psc_flatfoil_yz.cxx
index 28fd61b..2a28b7c 100644
--- a/src/psc_flatfoil_yz.cxx
+++ b/src/psc_flatfoil_yz.cxx
@@ -195,13 +195,13 @@ struct PscFlatfoil : Psc<PscConfig>
     sort_interval = 10;

     // -- Collision
-#if 0
+#if 1
     int collision_interval = 10;
 #endif
-#if 1
+#if 0
     int collision_interval = -10;
 #endif
-    double collision_nu = .1;
+    double collision_nu = .000424;
     collision_.reset(new Collision_t{grid(), collision_interval, collision_nu});

     // -- Checks

$ cd psc
$ mkdir build-summit-gpu
$ cd build-summit-gpu

Make sure these modules are loaded (note gcc version in particular)

[kaig1@login2 build-summit-gpu]$ module list

Currently Loaded Modules:
  1) hsi/5.0.2.p5    4) darshan-runtime/3.1.6       7) cmake/3.9.2                      10) hdf5/1.10.0-patch1
  2) xalt/1.1.3      5) DefApps                     8) gcc/6.4.0
  3) lsf-tools/2.0   6) cuda/9.2.64           (H)   9) spectrum-mpi/10.2.0.10-20181214

Create cmake.sh with the following content and run it (. cmake.sh)

$ cmake \
   -DCMAKE_BUILD_TYPE=RelWithDebInfo \
   -DCMAKE_C_COMPILER=mpicc \
   -DCMAKE_CXX_COMPILER=mpicxx \
   -DCMAKE_C_FLAGS_RELWITHDEBINFO="-g -O2" \
   -DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-g -O2" \
   -DCMAKE_CUDA_FLAGS_RELWITHDEBINFO="-g -O2" \
   -DGTEST_ROOT="$HOME/build/googletest/googletest/build" \
   -DUSE_CUDA=ON ..

Build the code

[kaig1@login2 build-summit-gpu]$ make

I made this batch script. I actually ended up running an interactive job, so it's not tested, but I think this is what I did. You probably also want to adjust this back to created a per-jobid directory and run the code there.

#! /bin/bash
#BSUB -nnodes=43
#BSUB -W 0:30
#BSUB -P AST147

cd $MEMBERWORK/ast147

cp ~/src/psc/build-titan/src/psc_flatfoil_yz $PWD

jsrun -n 256 -g 1 -c 1 -a 1  ./psc_flatfoil_yz \
    \
    --mrc_io_nr_writers 8 \
    \
    --mrc_domain_curve_type hilbert \

Output is very slow, so you probably want to make it much less frequent or turn it off entirely.