[FeatureRequest] 3D stencil in benchmarks

[rangure]

Is your feature request related to a problem? Please describe.
In our group, we've been using likwid to measure the practical performance of our hardware. And stencil is one of the common pattern emerge from many applications. It would be great if we could have 3d and 2d stencil that runs with system peak performance for references.

Describe the solution you'd like
Add 3d and 2d stencil into the benchmark suite.

Describe alternatives you've considered
None

Additional context
This may be not trivial as for 3d stencil you need to make sure you're reusing the data efficiently. This means we may need cache blocking to achieve system peak performance. and the specific cache blocking strategies may be system dependent.

[TomTheBear]

I assume you mean stencil kernels for likwid-bench? If so, we are currently re-writing likwid-bench to support multi-dimensional data structures and loop nests. You could theoretically write stencils already in the current version using 1D arrays and custom loop logic.

If you mean general testing codes in C or Fortran, I probably have some versions somewhere but I doubt that they provide optimal performance on all systems. For that, you commonly need spatial and/or temporal blocking.

[rangure]

Thanks for the fast reply. Yes, I would like to have likwid-bench stencil functionalities. I wonder how would the multi-dimensional data structure work. are you going to supply the benchmarks for some fixed patterns using multi dimensional data structures and loop nest or the user need to write the benchmarks themselves.
Yes, I would expect some kind of blocking to achieve peak performance. But according to my understanding tuning the block sizes are not trivial. and I assume the new support in likwid-bench is going to use blocking for multi-dimensional data. Then how could you determine the block sizes. Or the plan is just to perform, for example, copy with multi-dimensional data structures.

[TomTheBear]

Like it is now already, some benchmarks will be included in the repository but the users are free to write their own. The rewrite uses a different file format though (based on YAML).
The kernels contain a definition of the data structures and the calculation. It is all fixed, only little transformations can be done.

For stencils, you can commonly calculate quite optimal blocking factors. We call that layer conditions and even provide an online calculator: https://rrze-hpc.github.io/layer-condition/ . The paper is here or arxiv. This rather simple calculation can be done at assembly level in likwid-bench as well.

[rangure]

The reference is very useful. Maybe this is a little bit deviate from the issue, but do you know in general if it's possible to run single 3d-27-point stencil(which calculate the average of the kernel) sweep on a grid with spatial blocking with full DRAM bandwidth, or any related code example/books.

[TomTheBear]

Here is a list of stencil-related publications we did over the years.

• J. Hammer, G. Hager, J. Eitzinger, and G. Wellein: Automatic Loop Kernel Analysis and Performance Modeling With Kerncraft. Proc. PMBS15, the 6th International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems, in conjunction with ACM/IEEE Supercomputing 2015 (SC15), November 16, 2015, Austin, TX. DOI: 10.1145/2832087.2832092, Preprint: arXiv:1509.03778
• H. Stengel, J. Treibig, G. Hager, and G. Wellein: Quantifying performance bottlenecks of stencil computations using the Execution-Cache-Memory model. Proc. ICS15, 
  DOI: 10.1145/2751205.2751240, Preprint: arXiv:1410.5010
• M. Wittmann, G. Hager, T. Zeiser, J. Treibig, and G. Wellein: Chip-level and multi-node analysis of energy-optimized lattice-Boltzmann CFD simulations. Concurrency and Computation: Practice and Experience (2015). DOI:10.1002/cpe.3489 
  Preprint: arXiv:1304.7664
• J. Treibig, G. Wellein and G. Hager: Efficient multicore-aware parallelization strategies for iterative stencil computations. Journal of Computational Science 2 (2), 130-137 (2011). DOI 10.1016/j.jocs.2011.01.010
• M. Wittmann, G. Hager, J. Treibig and G. Wellein: Leveraging shared caches for parallel temporal blocking of stencil codes on multicore processors and clusters. Parallel Processing Letters 20 (4), 359-376 (2010).
• G. Wellein, G. Hager, T. Zeiser, M. Wittmann and H. Fehske: Efficient temporal blocking for stencil computations by multicore-aware wavefront parallelization. Proc. COMPSAC 2009. DOI: 10.1109/COMPSAC.2009.82
• J. Hornich, J. Hammer, G. Hager, T. Gruber, and G. Wellein: Collecting and Presenting Reproducible Intranode Stencil Performance: INSPECT. Supercomputing Frontiers and Innovations 6(3), 4-25 (2019). ISSN 2313-8734. Available with Open Access. DOI: 10.14529/jsfi190301
• T. M. Malas, G. Hager, H. Ltaief, and D. E. Keyes: Multi-dimensional intra-tile parallelization for memory-starved stencil computations. ACM Transactions on Parallel Computing 4(3), 12:1-12:32 (2017). DOI: 10.1145/3155290, Preprint: arXiv:1510.04995
  T. M. Malas, G. Hager, H. Ltaief, H. Stengel, G. Wellein, and D. E. Keyes: Multicore-optimized wavefront diamond blocking for optimizing stencil updates. SIAM Journal on Scientific Computing 37(4), C439-C464 (2015). DOI: 10.1137/140991133, Preprint: arXiv:1410.3060

Some slides mentioning all the steps of optimizing a 3D 27pt stencil: https://bebop.cs.berkeley.edu/pubs/Jonathan_Carter.pdf