[Bug]: stencilDist arrays not working with GPU support

[jeremiah-corrado]

Summary of Problem

Using the stencil distribution with the GPU locale model causes programs to emit an error from MemConsistency:

$CHPL_HOME/modules/internal/MemConsistency.chpl:116: error: variable 'memory_order_seq_cst' must be defined in the function it is used in for GPU usage

Steps to Reproduce

Attempting to compile the following code from the Chapel blog with CHPL_LOCAL_MODEL=gpu, results in the above error. However, if the stencil distribution is replaced with a block distribution (and the updateFluff call is removed), the program will compile and run successfully.

use StencilDist;

// set up simulation constants
config const xLen = 2.0,
             yLen = 2.0,
             nx = 31,
             ny = 31,
             sourceMag = 100.0,
             maxIters = 100;

const dx = xLen / (nx-1),
      dy = yLen / (ny-1);

// simulation settings
config const makePlots = false;

// define problem space
const space = stencilDist.createDomain({0..<ny, 0..<nx}, fluff=(1,1)),
      spaceInner = space.expand(-1);

proc main() {
    // define pressure and source arrays
    var p, b: [space] real = 0.0;

    // set boundary conditions
    p[..,    0] = 0;                     // p = 0 @ x = 0
    p[.., nx-1] = [j in 0..<ny] j * dy;  // p = y @ x = xLen
    p[0,    ..] = p[1, ..];              // dp/dy = 0 @ y = 0
    p[ny-1, ..] = p[ny-2, ..];           // dp/dy = 0 @ y = yLen

    // set sink/source terms
    b[3*ny/4, nx/4] = sourceMag;
    b[ny/4, 3*nx/4] = -sourceMag;

    solvePoisson(p, b);
}

proc solvePoisson(ref p: [?d] real, const ref b: [d] real) {
    // temporary copy of p
    var pn = p;

    // solve for maxIters iterations
    for 1..maxIters {
        p <=> pn;
        pn.updateFluff();
        poissonKernel(p, pn, b);
        neumannBC(p);
    }
}

proc poissonKernel(ref p: [] real, const ref pn: [] real, const ref b: [] real) {
    forall (i, j) in spaceInner {
        p[i, j] = (
            dy**2 * (pn[i, j+1] + pn[i, j-1]) +
            dx**2 * (pn[i+1, j] + pn[i-1, j]) -
            dx**2 * dy**2 * b[i, j]
        ) / (2.0 * (dx**2 + dy**2));
    }
}

proc neumannBC(ref p: [] real) {
    p[0,    ..] = p[1, ..];    // dp/dy = 0 @ y = 0
    p[ny-1, ..] = p[ny-2, ..]; // dp/dy = 0 @ y = yLen
}

printchplenv...

CHPL_TARGET_PLATFORM: linux64
CHPL_TARGET_COMPILER: llvm
CHPL_TARGET_ARCH: x86_64
CHPL_TARGET_CPU: native
CHPL_LOCALE_MODEL: gpu *
  CHPL_GPU: nvidia *
CHPL_COMM: none
CHPL_TASKS: qthreads *
CHPL_LAUNCHER: none
CHPL_TIMERS: generic
CHPL_UNWIND: none
CHPL_MEM: jemalloc
CHPL_ATOMICS: cstdlib
CHPL_GMP: bundled
CHPL_HWLOC: bundled
CHPL_RE2: bundled
CHPL_LLVM: bundled
CHPL_AUX_FILESYS: none

[e-kayrakli]

Thanks for this issue, @jeremiah-corrado!

Distributed array + GPUs is something we are actively working on. #24429 is another issue that captures that. It is somewhat surprising to see this behavior in StencilDist, though. I was thinking that this is about a missing pragma in StencilDist, but that pragma is there: https://github.com/chapel-lang/chapel/blob/main/modules/dists/StencilDist.chpl#L1592

Tagging @vasslitvinov as the person who's leading the charge here. I personally don't think this particular issue is something we should focus on right now until we fix some of the other challenges you are already working on. But there might be something to learn from it.

[vasslitvinov]

I assigned this to myself. Will investigate.