Day in the life of fi_send fi_recv

There is a one to one correspondence between application's invocation of fi_send, fi_recv, and a gnix fab_req. This article covers basic code flow through gnix_send and gnix_recv

gnix_send

gnix_send populates a fab request based on the parameters provided to the function call, and finally will make another function call to try to initiate the send. The basic steps of gnix_send are outlined below.

1. Basic Sanity checking. A few examples are listed below, but these may or may not change.

• Check that the provided EP has a send_cq
• Check that size of the send is lower than GNIX_INJECT_SIZE(With FI_INJECT flag present)
• Check FI_TRIGGER

2. Determine whether the send is short (eager) or long (rendezvous). If the message is longer than the rendezvous threshold SMSG cannot be used, so both send and receive must handle this case differently.

• If rendezvous, a memory descriptor must be registered.

3. Get Virtual Channel For a specific Endpoint and Destination Address you will get a specific Virtual Channel.
4. Allocate Fab Request
5. Populate Fab Request (Expand on this)

• What's different with FI_INJECT?

6. Initiate gnix_vc_queue_tx_req. This attempts to initiate a TX request. If the TX queue is blocked (due to low resources or a FI_FENCE request), the request is scheduled to be sent later. gnix_vc_queue_tx_req will eventually call on a work_fn to process the request. In the case of gnix_send, the work function provided is gnix_send_req.

gnix_send_req

1. Basic sanity checking.

• Check for EP and NIC

2. gnix_nic_tx_alloc Allocates a tx desc for the provided nic
3. Populate tdesc This is done differently based on the type of transaction (eager or rendezvous). In each case, a different portion of the struct will be populated - tdesc->eager_hdr or tdesc->rndzv_start_hdr. Refer to sections below for more information on the differences.

• Regardless of message type, the req`` is added into the tdescas well as acompleter_fn(much like thework_fnadded to thereqingnix_send). The completer function provided is gnix_ep_smsg_completers[tag]```.

4. GNI_SmsgSendWTag is called.

send_req rendezvous

1. Ensure there is an associated memory descriptor.
2. Populate tdesc->rndzv_start_hdr
3. GNI_READ_ALIGN_MASK (what is this step?)

• Changes are made to rndzv_start_hdr.head and .tail

4. hdr and hdr_len are set to equal rndzv_start header and its size.
5. Data is set to null.

• data = Null data_len= 0

send_req eager

1. Populate tdesc->eager_hdr
2. The hdr and data are added to the tdesc here. Since this isn't a rndzv the send length should always be the cumulative length of all the send_info lengths

send_req iov

TODO

gnix_recv

1. Basic Sanity Checking

• Check EP has recv_cq
• Check msg_recv_allowed or tagged_recv_allowed
• gnix_msg_addr_lookup (expand on this)

2. gnix_msg_queues - This function sets up receive queues and locks based on whether the message is tagged or not.
3. Acquire lock
4. Look for a matching receive request
5. Populate fab request fields
6. Check if using Peek/Claim,Discard (P/C/D) matching flags (FI_DISCARD, FI_PEEK)
7. Determine what kind of receive request was received, rendezvous or eager?

recv eager

• Send length is truncated to receive buffer size
• Data is copied from eager msg to receive buffer (expand here)
• Generate recv completion
• If using FI_MULTI_RECV another space is left in the receive buffer. Go back to step 7 from previous section and proceed again.
• gnix_fr_free

recv rendezvous

TODO

P/D/C

TODO