Skip to content

gnix_rma

Amith Abraham edited this page Oct 6, 2016 · 3 revisions
  • RMA Requests support Reads and Writes.
  • GNI Supports writing to local and remote addresses with any alignment and length.
  • GNI requires reads to use four byte aligned remote address and length.
  • For reads smaller than one cacheline, aligned data is read into an intermediate buffer, then partially copied to the user buffer. This is referred to as an INDIRECT Transfer.
  • For larger unaligned reads, the interior, aligned portion of remote data is pulled directly into the user provided buffer. The four bytes at the head and tail of an unaligned read are pulled into an intermediate buffer, then partially copied into the user buffer. This is referred to as a CHAINED transfer.
  • Unaligned reads smaller than the RDMA threshold can perform these 3 distinct transactions (head, middle, tail) in a single GNI chained FMA operation (resulting in a single GNI CQE).
  • For unaligned reads larger than than the RDMA threshold, two GNI posts are used. One RDMA TX to transfer the bulk of the data, and then an FMA TX to transfer the head and/or tail data.

gnix_rma

  1. Basic sanity checking. A few examples are listed below, but these may or may not change.
  2. Find VC for target
  3. Set up fab request
  4. Determine if RDMA by checking if the length exceeds the rma rdma threshold.
  5. Default work_fn is set to be gnix_rma_post_req
  6. Determine if using Chained Transfer or INDIRECT Transfer
  7. If RDMA set the work_fn to be gnix_rma_post_rdma_chain_req
  8. If using ```RDMA` write or read, we need to auto-register the source buffer and get a memory descriptor.
  9. FI_INJECT - ??? Different behavior for req->rma.loc_addr
  10. Some completion stuff Inject interfaces always suppress completions. If SELECTIVE_COMPLETION is set, honor any setting. Otherwise, always deliver a completion.
  11. return gnix_vc_queue_tx_req This attempts to initiate a TX request. If the TX queue is blocked (due to low resources or a FI_FENCE request), the request is scheduled to be sent later. gnix_vc_queue_tx_req will eventually call on the work_fn to process the request.

gnix_rma_post_req

  1. Basic Sanity Checking
  2. gnix_nic_tx_alloc - Allocate a tx desc for the provided nic
  3. Set the txd completer_fn - gnix_rma_txd_complete is provided.
  4. Convert key to memory handle
  • If FMA - convert kye to memory handle; without CRC validation.
  1. Finish populating txd
  2. Fill post descriptor based on transaction type
  • INDIRECT Get
  • CHAINED Get

gnix_rma_post_rdma_chain_req

Clone this wiki locally