Local Completion Operation

[jdinan]

Problem Statement

OpenSHMEM 1.4 includes a fence operation to provide ordering, and a quiet operation to provide remote completion at the granularity of an individual context. However, no operation is provided to ensure local completion for nonblocking operations issued on a context (the pipelining example on page 33 of OpenSHMEM 1.4 could take advantage of such an operation).

The merged requests proposal, included such an operation and it was demonstrated to provide a performance benefit for networks such as InfiniBand {paper,slides}.

Proposed Extension

void shmem_ctx_quiet_local(shmem_ctx_t ctx);

The shmem_quiet_local routine ensures local completion of nonblocking Put and Get routines issued by the calling PE on the given context. An operation is locally complete when input and output buffers at the calling PE are no longer in use by the OpenSHMEM library. That is, input buffers have been read and may be overwritten; similarly, output buffers contain the requested data and may be read from.

The shmem_quiet_local routine does not guarantee that remote updates have been completed in memory at the target PE, nor does it guarantee ordering.

Should this routine also guarantee ordering for load and store operations?

[jdinan]

Discussed at 9/4/18 meeting -- Trending negative. Networking folks on the line indicated that there is often little timing difference between local/remote completion.

[naveen-rn]

Following #468 - closing this issue