Day in the life of fi_send fi_recv

There is a one to one correspondence between application's invocation of fi_send, fi_recv, and a gnix fab_req. This article covers basic code flow through gnix_send and gnix_recv

gnix_send

gnix_send populates a fab request based on the parameters provided to the function call, and finally will make another function call to try to initiate the send. The basic steps of gnix_send are outlined below.

1. Basic Sanity checking

• Check that the provided EP has a send_cq
• Check that size of the send is lower than GNIX_INJECT_SIZE(With FI_INJECT flag present)
• Check FI_TRIGGER

2. Determine whether the send is short (eager) or long (rendezvous). If the message is longer than the rendezvous threshold SMSG cannot be used, so both send and receive must handle this case differently.

• rendezvous = len >= ep->domain->params.msg_rendezvous_thresh;
• If rendezvous, a memory descriptor must be registered.

3. Get Virtual Channel
4. Allocate Fab Request
5. Populate Fab Request (Expand on this)

• If FI_TAGGED req->msg.tag
• If rendezvous req->msg.send_info[0].mem_hndl
• If FI_INJECT
  • Does some stuff (expand here)

6. Initiate gnix_vc_queue_tx_req. This attempts to initiate a TX request. If the TX queue is blocked (due to low resources or a FI_FENCE request), the request is scheduled to be sent later. gnix_vc_queue_tx_req will eventually call on a work_fn to process the request. In the case of gnix_send, the work function provided is gnix_send_req.

gnix_send_req

1. Basic sanity checking.

• Check for EP and NIC

2. gnix_nic_tx_alloc
3. **Populate tdesc ** This is done differently based on the type of transaction (eager or rendezvous). In each case, a different portion of the struct will be populated - tdesc->eager_hdr or tdesc->rndzv_start_hdr

send_req rendezvous

1. Ensure there is an associated memory descriptor.
2. Populate tdesc->rndzv_start_hdr

• flags
• imm
• msg_tag
• mdh
• addr
• len *req_addr

3. GNI_READ_ALIGN_MASK (what is this step?)

• Changes are made to rndzv_start_hdr.head and .tail

4. hdr and hdr_len are set to equal rndzv_start header and its size.
5. Data is set to null.

• data = Null data_len= 0

send_req eager

1. Populate tdesc->eager_hdr

• flags
• imm
• msg_tag
• len

2. hdr, hdr_len, data, and data_len are populated.
   Since this isn't a rndzv the send length should always be the cumulative length of all the send_info lengths

send_req iov

TODO

gnix_recv

1. Basic Sanity Checking

• Check EP has recv_cq
• Check msg_recv_allowed or tagged_recv_allowed
• gnix_msg_addr_lookup (expand on this)

2. gnix_msg_queues (expand here)
3. Acquire lock
4. Look for a matching receive request
5. Populate Local Fields
6. Check if using Peek/Claim,Discard (P/C/D) matching flags (FI_DISCARD, FI_PEEK)
7. Determine what kind of receive request was received, rendezvous or eager?

recv eager

• Send length is truncated to receive buffer size
• Data is copied from eager msg to receive buffer (expand here)
• Generate recv completion
• If using FI_MULTI_RECV another space is left in the receive buffer. Go back to step 7 from previous section and proceed again.
• gnix_fr_free

recv rendezvous

TODO

P/D/C

TODO