Add support for different kinds of memory (HBM, FAM, and others)

[naveen-rn]

Goal

Allow explicit allocation of symmetric heap on different kinds of memory available in the system architecture.

Problem

Emerging systems support multiple kinds of memory with different performance and capacity characteristics. At present, the symmetric heap is created on a memory location determined by the implementation and the users don't have options to select the kind of memory for allocating the symmetric heap.

Proposed Solution - 1 - Environment Variable based Allocation

We propose few runtime changes to allow the users to determine the memory location at which the symmetric heaps can be created and call these user-determined memory locations as Memory Partitions. Each Memory Partitions feature a list of traits to define their characteristics and kind of memory being one of those featured traits.

More details on the proposed changes, HW/SW Impact analysis are available in Redmine Ticket and in this OpenSHMEM-17 workshop paper.

Example

# Environmental variable
SHMEM_SYMMETRIC_PARTITION<ID>=size=<size>[:pgsize=<page_size>][:kind=<mem_kind>][:policy=<mem_policy>]

# C/C++ routines
void *shmem_partition_malloc(size_t size, int partition_id)
void *shmem_partition_align(size_t alignment, size_t size, int partition_id)

Proposed Solution - 2 - OpenSHMEM Teams based memory allocation

This solution is still under discussion. Will be updated once the OpenSHMEM Teams proposal is complete and ready for adding into the specification.

Notes

Symmetric Memory Partition is a feature for allocating SHEAP on different memory kinds explicitly by the users. There are multiple ways to achieve it. Initially it started as env-variable based approach and later evolved into team-based approach. So, the feature itself isn't fixed on any one particular means to achieve the goal. Even both the solution can be adopted.

Blocked by Teams proposal for further discussioin

[naveen-rn]

Initial draft specification based on OpenSHMEM-1.3 specification. Needs update.

mem_partition_initial_draft.pdf

[naveen-rn]

As the teams working group is working on a proposal to introduce PE subsets in OpenSHMEM. The plan is to move towards the team-based symmetric heap. This issue should be an initial starting point for our discussions.

[naveen-rn]

Linking related documents: https://gist.github.com/nspark/b36fd446252ed789005223612808f45e

[naveen-rn]

Proposed Solution - 2 - OpenSHMEM Teams based memory allocation

Requirement

Similar to Proposed Solution - 1, we need to support different kinds of memory including HBM, fabric attached memory, low latency memory and other possible types. While the proposed solution - 1 is simple, it has limited usage model. This solution provides extensive and flexible support for utilizing different types of memory by relating teams, team-based contexts, and team-based symmetric heaps.

Background:

1. Contexts are essentially a team-object with the new OpenSHMEM Teams proposal.
2. There are two types of contexts:

• Contexts created from the global team
• Contexts created from the user-created teams

Proposal:

1. Symmetric memory created with different traits like pagesize, memory type, size, and other properties using a new collective API shmem_create_spaces() are called Memory Spaces.
2. Spaces are created on a specific team-object.

• Spaces have many-to-one mapping to teams, at most one team per space.
• Multiple spaces can be created on a single team.

3. Contexts are expected to be mapped to a space through the teams.

• Contexts have many-to-one mapping to spaces, at most one space is mapped to a context.
• Multiple contexts sharing the same team objects can be mapped to different spaces, but should not break the previous requirement
• Multiple contexts sharing the same team object can be mapped to the same space.

API:

shmem_create_spaces(IN shmem_team_t team, IN shmem_config_space_t traits, OUT shmem_space_t *space);
shmem_ctx_set_space(IN shmem_space_t space, IN shmem_ctx_t ctx);

basic traits available for creating spaces are type of memory (HBM, LOW_LATENCY, DDR), pagesize, size in bytes, memory policy.

Compatibility:

No backward compatibility issues with the current specification.

Under Consideration:

Do we need new explicit context option to decide on the memory type to be used, say SHMEM_CTX_FAM for specifying whether the context created would be then mapped to memory space created on an external memory/FAM? This can also be done internally during the context mapping operation with the spaces.

[naveen-rn]

Attaching slides for providing Support for Different Memory Kinds using Memory Spaces.
extending-memory-spaces.pptx

[shamisp]

As long as the memory model (ordering, alignment requirements, c$ line size, etc) is exactly the same across all memory spaces, it may work. Otherwise it is not really manageable.

[naveen-rn]

Slides describing the need to separate Memory Spaces and Memory Kinds in OpenSHMEM.
Slides used during the Threads WG Meeting 12-10-2019.

Slides: decouple-space-memory-kinds.pdf

[naveen-rn]

ARCHIVED OLD PROPOSAL REFER LINK FOR RECENT PROPOSAL

Updated proposal for Feb-2020 F2F discussions:

Space Proposal:1 - Supporting Team-based SHEAP

Object:

shmem_space_config_t {
    long long               config_mask;
    int                     size;
    enum shmem_space_sz_t   pgsize;
    enum shmem_space_pl_t   alloc_policy;
    enum shmem_space_nd_t   numa_domain;  // need discussion on the numa policy
};

Collective Operations - Space Maintenance:

void shmem_space_create(shmem_team_t team, shmem_space_config_t space_config, shmem_space_t *space)
void shmem_space_attach(shmem_team_t team, shmem_space_t *space)

Local Operations - Space Maintenance:

void shmem_space_destroy(shmem_space_t space)

Local Operations - Context Creation:

Now:
void shmem_team_create_ctx(shmem_team_t team, long options, shmem_ctx_t *ctx)

OpenSHMEM-1.5:
void shmem_team_create_ctx(shmem_team_t team, void *ptr, long options, shmem_ctx_t *ctx)

What we really want:
void shmem_team_create_ctx(shmem_team_t team, shmem_space_t space, long options, shmem_ctx_t *ctx)

Local Operations - Space-based Memory Allocation:

void *shmem_space_malloc(size_t size)
void *shmem_space_calloc(size_t count, size_t size)
void *shmem_space_align(size_t alignment, size_t size)

New Context Options:

1. SHMEM_CTX_NO_GLOBAL // should add other context properties in future for memory
                       // ordering when different types of memory are used

Supported Usage Model:

1. Create multiple spaces per team
2. Attach multiple team per space
3. Allow only one context per space
4. Allow context created using shmem_ctx_create to use default SHEAP and global/static
5. Allow context created using shmem_team_create_ctx to use space object and global/static
6. Allow context created using shmem_team_create_ctx with invalid space to use default SHEAP
7. Allow context created using shmem_team_create_ctx with SHMEM_CTX_NO_GLOBAL just the default SHEAP or space object

Unanswered Questions: (user queries)

1. Do we need to support multiple spaces per team?
2. Do we need to allow child teams to access parent teams SHEAP for data transfer operations?

Followup Proposals:

1. Introducing different kinds of memory - device and HBM
2. Extensions to use FAM

[naveen-rn]

This issue is being discussed in the Memory Spaces WG discussion. Notes on the progress of this new feature is available in the following wiki link. Based on the discussion, we are splitting this huge proposal into three different sections:
Section:1 Determining the relationship between Teams, Contexts, and Spaces
Section:2 Adding support for different memory traits in Spaces
Section:3 Extending Spaces to support flat FAM memory

We will open separate issues for section specific changes.

Other related issues:

1. Integration of Spaces with Teams/Contexts Integration of Spaces with Teams/Contexts #361 - will be handled as part of Section:1
2. NUMA awareness in Spaces Provide NUMA awareness/control of (local) symmetric memory #341 - will be part of Section:2

[naveen-rn]

Marking it as blocked - as we have opened separate sub-section-based issues. Closing all the sub-section-based issues will allows us to close this one.