-
Notifications
You must be signed in to change notification settings - Fork 0
gnix_rma
Amith Abraham edited this page Oct 6, 2016
·
3 revisions
- RMA Requests support Reads and Writes.
- GNI Supports writing to local and remote addresses with any alignment and length.
- GNI requires reads to use four byte aligned remote address and length.
- For reads smaller than one cacheline, aligned data is read into an intermediate buffer, then partially copied to the user buffer. This is referred to as an
INDIRECT Transfer
. - For larger unaligned reads, the interior, aligned portion of remote data is pulled directly into the user provided buffer. The four bytes at the head and tail of an unaligned read are pulled into an intermediate buffer, then partially copied into the user buffer. This is referred to as a
CHAINED transfer
. - Unaligned reads smaller than the RDMA threshold can perform these 3 distinct transactions (head, middle, tail) in a single GNI chained FMA operation (resulting in a single GNI CQE).
- For unaligned reads larger than than the RDMA threshold, two GNI posts are used. One RDMA TX to transfer the bulk of the data, and then an FMA TX to transfer the head and/or tail data.
- Basic sanity checking. A few examples are listed below, but these may or may not change.
- Find VC for target
- Set up fab request
- Determine if
RDMA
by checking if the length exceeds the rma rdma threshold. - Default
work_fn
is set to begnix_rma_post_req
- Determine if using
Chained Transfer
orINDIRECT Transfer
- If
RDMA
set thework_fn
to begnix_rma_post_rdma_chain_req
- If using ```RDMA` write or read, we need to auto-register the source buffer and get a memory descriptor.
- FI_INJECT - ??? Different behavior for req->rma.loc_addr
- Some completion stuff
Inject interfaces always suppress completions. If
SELECTIVE_COMPLETION
is set, honor any setting. Otherwise, always deliver a completion. - return
gnix_vc_queue_tx_req
This attempts to initiate a TX request. If the TX queue is blocked (due to low resources or a FI_FENCE request), the request is scheduled to be sent later. gnix_vc_queue_tx_req will eventually call on thework_fn
to process the request.
- Basic Sanity Checking
-
gnix_nic_tx_alloc
- Allocate a tx desc for the provided nic - Set the txd completer_fn - gnix_rma_txd_complete is provided.
- Convert key to memory handle
- If FMA - convert kye to memory handle; without CRC validation.
- Finish populating txd
- Fill post descriptor based on transaction type
INDIRECT Get
CHAINED Get