SRC/UCP/CORE: set FLAG_LOCK for allocated buffers when NONBLOCK is not passed #10830
Conversation
…t passed to ucp_mem_map Signed-off-by: Roie Danino <rdanino@nvidia.com>
| flags |= UCT_MD_MEM_FLAG_NONBLOCK; | ||
| } else if (params->flags & UCP_MEM_MAP_LOCK) { | ||
| } else if ((params->flags & UCP_MEM_MAP_LOCK) || | ||
| (params->flags & UCP_MEM_MAP_ALLOCATE)) { |
There was a problem hiding this comment.
I'm not sure that would always work.
It leads to usage of reg_block_md_map (https://github.com/roiedanino/ucx/blob/reg-nb/src/ucp/core/ucp_mm.c#L866)
which is populated here.
But if UCX_REG_NONBLOCK_MEM_TYPES is empty (as default), that map would be empty.
@yosefe wdyt?
There was a problem hiding this comment.
should be ok, because it just adds context->reg_block_md_map[mem_type] to the existing reg_map.
but what does it fix? Why would we need this change?
There was a problem hiding this comment.
The default for
UCX_REG_NONBLOCK_MEM_TYPESis "host,cuda-managed" its not really empty
it is default only for grace
There was a problem hiding this comment.
The issue is on grace machines the default is not empty and then even if NONBLOCK flag wasn't passed it would still use NONBLOCK registration (unless LOCK flag was set on)
There was a problem hiding this comment.
Meaning, the change will have effect only when UCX_REG_NONBLOCK_MEM_TYPES has a non-empty value.
There was a problem hiding this comment.
but we have UCP_MEM_MAP_LOCK for those who want blocking behavior?
otherwise UCX may decide what is better
There was a problem hiding this comment.
There is more to it.. we need to cancel all effects of REG_NONBLOCK_MEM_TYPES in case of memory allocation. Here, we create another effect (passing UCT lock flag) and we don't prevent the registration md map filtering - that limits registration only to MD that support NONBLOCK registration.
There was a problem hiding this comment.
that was the intent to use ODP on GH machines by default
The issue of ODP+ DPP is being addressed here #10803
4 participants
What?
Set
UCT_MD_MEM_FLAG_LOCKflag when user is allocating a buffer using ucp_mem_map without passingUCP_MEM_MAP_NONBLOCKflagWhy?
Otherwise, non-blocking registration (ODP) can still happen in
ucp_memh_register_internal