SHMEM_SYMMETRIC_SIZE Argument

[jdinan]

Issue

SHMEM_SYMMETRIC_SIZE does not adequately specify the format for its argument.

Proposed Solution

We should support floating point numbers with the standard size suffixes of K, M, G, and T corresponding to powers of two bytes, e.g. SHMEM_SYMMETRIC_SIZE=1.5G requests 1.5 * 2^30 bytes.

An unqualified integer, e.g. SHMEM_SYMMETRIC_SIZE=1024, corresponds to a size in bytes and may not contain a decimal point.

Notes from Redmine Discussion Ticket 206

Cray and SGI SHMEM both support a floating point number with K, M, G, and T suffix.

Sandia OpenSHMEM support an integer number with K, M, G, and T suffix.

[jamesaross]

SMA_SYMMETRIC_SIZE is deprecated as of 1.4.

SHMEM_SYMMETRIC_SIZE is precisely defined as any non-negative integer in bytes.

Using floating point numbers implies fractional bytes and the rounding mode has not been defined. It also makes it less precise and unclear. For example, users may interpret G as 10^9 instead of 2^30.

[jdinan]

Updated the text to use SHMEM_SYMMETRIC_SIZE (this was migrated from an older Redmine ticket that was created before the new constants were introduced, hence the stale text).

Good note that we should define rounding/truncation and clarify powers of 2 vs 10. I expect rounding would be need to be phrased as "at least X many Bytes" since implementations may include heap space for internal structures (e.g. SOS allocates a pSync for barrier_all from the symmetric heap) and also round up to the nearest whole page size.

As far as the floating point part goes, this is already supported by the major SHMEM libraries. Could folks in the user community check on whether they use/want this? @nspark

[naveen-rn]

Clarification for powers of 2 vs 10 seems reasonable. But, "at least X many bytes" kind of explanation for the round off seems a bit restrictive on the implementation. If the implementation maintains the SHEAP with some internal data structure or if the user creates multiple smaller chunks of SHEAPs with shmem_align, we may not be able to satisfy the "at least X many bytes" condition.

[tonycurtis]

On Jan 12, 2018, at 9:14 AM, Naveen Namashivayam Ravichandrasekaran ***@***.***> wrote: Clarification for powers of 2 vs 10 seems reasonable. But, "at least X many bytes" kind of explanation for the round off seems a bit restrictive on the implementation. If the implementation maintains the SHEAP with some internal data structure or if the user creates multiple smaller chunks of SHEAPs with shmem_align, we may not be able to satisfy the "at least X many bytes" condition.

The initial allocation size of the symmetric heap wouldn’t affect potential fragmentation of that heap due to shmem_align() calls later, or am I misunderstanding? As a (sometime :) ) user I really wouldn’t want to have to specify the heap size in bytes; e.g. “1.5G" seems much more reasonable, however it should be interpreted. tony

[nspark]

Yes, this K/M/G/T-suffix feature was actually something we've recently talked about wanting. We were thinking along similar lines that that a decimal + suffix value should be at least that many bytes, rounded up to the page size.

@naveen-rn I'm not sure I understand your concerns about multiple heaps and restricting the implementation. For something like the Cray/Intel multiple-heaps extension (#195), each of the heap-size specifications could follow this format; if the implementation could not satisfy the total amount of memory needed for any one of the heaps, then it would report some error as implementations likely do now. It would be on the user not to request too much memory.

For now, I just use Bash shell arithmetic; e.g., SHMEM_SYMMETRIC_SIZE=$((4 << 30)).

[naveen-rn]

The initial allocation size of the symmetric heap wouldn’t affect potential fragmentation of that heap due to shmem_align() calls later, or am I misunderstanding?

@tonycurtis It looks more like an implementation specific detail.

Let us consider this simple example, with SHMEM_SYMMETRIC_SIZE=1M.

#define SIZE 1048576

int foo(void) {
    shmem_init();
#if MONOLITHIC
    if ((char*)shmem_malloc(sizeof(char)*SIZE) == NULL) {
        printf("PE %d ERROR: No SHEAP space available\n", shmem_my_pe());
        shmem_global_exit(1);
    }
#else
    int i;
    for (i = 1; i < SIZE; i++) {
        if ((char*)shmem_malloc(sizeof(char)) == NULL) {
            printf("PE %d ERROR: No SHEAP space available for %d the byte\n",
                    shmem_my_pe(), i);
            shmem_global_exit(1);
        }
    }
#endif

    shmem_finalize();
    return 0;
}

Theoretically, we would expect to get all the 1048576 bytes of SHEAP for allocation from the implementation. When applications allocate large monolithic buffers, implementations could provide all the 1M bytes, but not when multiple smaller chunks are created.

It is good to provide a proper explanation for characters K,M,G(or k,m,g) in the env variable. But let us not phrase it with some exact available size specific requirements (esp with statements like at least X bytes).

@nspark This is not related to the multiple symmetric heap proposal. I was speaking in general about any symmetric heap creation and management.

[naveen-rn]

Explanation from Cray manpage:

SHMEM_SYMMETRIC_SIZE 

Controls the size (in bytes) per PE of the symmetric heap. The value set in this
environment variable is interpreted as a number of bytes, unless the number is
followed by a char that acts as a multiplier, where:
    g or G multiplies by 2**30 (gigabytes)
    k or K multiplies by 2**10 (kilobytes)
    m or M multiplies by 2**20 (megabytes)
For example, the string 20m returns the integer value 20*2**20, or 20 megabytes.
Only one multiplier is recognized, so 20kk will not produce the same value as
20m, nor will invalid strings such as 20MB produce the desired result.

[jdinan]

@naveen-rn The Cray manpage text looks good. Do you want to claim this ticket and bring that forward as a proposal?

[naveen-rn]

@naveen-rn The Cray manpage text looks good. Do you want to claim this ticket and bring that forward as a proposal?

Yes, I can create a PR for this.

[jeffhammond]

Cray man page uses the decimal names but the binary definition. OpenSHMEM should follow the proper names for decimal and binary, e.g. https://en.wikipedia.org/wiki/Gigabyte and https://en.wikipedia.org/wiki/Gibibyte.