Skip to content

Second round of benchmarking on SuperMUC-NG

Latest
Latest
Compare
Choose a tag to compare
@mvassaux mvassaux released this 04 Dec 13:28
· 10 commits to master since this release
0.1-beta.0
de31bdb

Second release of SCEMa for benchmarking purposes on SuperMUC-NG.

This new release features two optimisations solving issues found during the first benchmark:

  • avoid communication across nodes within a single LAMMPS simulation by making MD allocation size a factor or a multiple of node core count;
  • parallelise I/O (mkdir and rmdir calls) by disabling logging of MD simulations

Implementation of compact-tension and dogbone test simulations for large-scale simulations of SCEMa on up to a full partition of SuperMUC-NG.

Input files featuring SCEMa configuration file (.json), FE meshes (.msh) and molecular data (.lammpstrj, .bin, etc...) can be found in the attached tarball named inputs_benchmark_SupermucNG_12.2020.tar.gz.

Specific installation and execution directions for SuperMUC-NG can be found below and in the README.md file included in the tarball.

Building SCEMa on SuperMUC-NG

First deal.ii and LAMMPS need to be compiled with the dependencies listed here: https://github.com/UCL-CCS/SCEMa/blob/master/README.md

Compiling deal.ii

More specifically for deal.ii (version 8.5.0 or above) which can be a bit tricky to compile, the following modules need to be loaded on SuperMUC-NG:

module load python/3.6_intel petsc/3.8.4-impi-shared metis/5.1.0-intel-i64-r64 cmake/3.14.4 tbb/2019 gcc/7 mumps/5.0.1-intel-impi-shared scalapack/mkl netcdf/4.6.1-impi-hdf5v1.10-parallel hdf5/1.10.2-intel-impi-frt-threadsafe

Once these are loaded, configuration of deal.ii Makefiles should be straightforward:

cd /path/to/deal.ii/8.5.0/source
mkdir build
cmake -DDEAL_II_WITH_MPI=ON -DDEAL_II_WITH_PETSC=ON ..

From there, all the necessary options should be automatically set to 'ON', and deal.ii can be built:

make -j 8 install

Compiling SCEMa

Once both libraries are installed, edit paths in SCEMa CMakeLists configuration file which points toward:

  • Deal.ii build directory: /path/to/dealii-8.5.0/
  • LAMMPS sources directory: /path/to/lammps/src/
  • Python include directory: /path/to/python/include/pythonX.Y/
  • LAMMPS shared library: /path/to/lammps/src/liblammps.so
  • Python shared library: /path/to/python/lib/libpythonX.Y.so
grep -R "/path/to" /path/to/inputs_benchmark/CMakeLists.txt.generic

and move to SCEMa's directory:

cd /path/to/SCEMa/source/
mkdir build
cd build
cmake /path/to/inputs_benchmark/CMakeLists.txt.generic
make dealammps

Running a simulation of the dogbone test for pure epoxy resin

Before trying to perfom a simulation using SCEMa:

  • deal.ii version 8.5 or above must be installed
  • LAMMPS (preferably version 17Nov16) must be installed
  • dealammps executable (from the SCEMa repository) must be build

Retrieve benchmark inputs tarball archived on the GitHub page of the release and decompress it:

tar -xzvf inputs_benchmark.tar.gz

Create a directory for the simulation data

mkdir dogbone
cd dogbone

Copy the JSON inputs configuration file and the SLURM execution script:

cp ../inputs_benchmark/slurm.dogbone ../inputs_benchmark/inputs_dogbone.json .

Edit generic path to SCEMa directory (/path/to/SCEMa) in both files

Submit the simulation:

sbatch slurm.dogbone
Assets 3
Loading