Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

why set recv_sge_list start addr refer to buf start addr after post the recv wqes of tx-depth #208

Open
smiceice opened this issue May 15, 2023 · 2 comments

Comments

@smiceice
Copy link

file: perftest/src /perftest_resources.c

int ctx_set_recv_wqes(struct pingpong_context *ctx,struct perftest_parameters *user_param)
{
int i = 0,j,k;
int num_of_qps = user_param->num_of_qps;
struct ibv_recv_wr *bad_wr_recv;
int size_per_qp = user_param->rx_depth / user_param->recv_post_list;

if((user_param->use_xrc || user_param->connection_type == DC) &&
			(user_param->duplex || user_param->tst == LAT)) {

	i = user_param->num_of_qps / 2;
	num_of_qps /= 2;
}

if (user_param->use_srq)
	size_per_qp /= user_param->num_of_qps;
ctx->rposted = size_per_qp * user_param->recv_post_list;

for (k = 0; i < user_param->num_of_qps; i++,k++) {
	if (!user_param->mr_per_qp) {
		ctx->recv_sge_list[i * user_param->recv_post_list].addr = (uintptr_t)ctx->buf[0] +
			(num_of_qps + k) * ctx->send_qp_buff_size;
	} else {
		ctx->recv_sge_list[i * user_param->recv_post_list].addr = (uintptr_t)ctx->buf[i];
	}

	if (user_param->connection_type == UD)
		ctx->recv_sge_list[i * user_param->recv_post_list].addr += (ctx->cache_line_size - UD_ADDITION);

	ctx->rx_buffer_addr[i] = ctx->recv_sge_list[i * user_param->recv_post_list].addr;

	for (j = 0; j < user_param->recv_post_list; j++) {
		ctx->recv_sge_list[i * user_param->recv_post_list + j].length = SIZE(user_param->connection_type,user_param->size,1);
		ctx->recv_sge_list[i * user_param->recv_post_list + j].lkey   = ctx->mr[i]->lkey;

		if (j > 0) {
			ctx->recv_sge_list[i * user_param->recv_post_list + j].addr = ctx->recv_sge_list[i * user_param->recv_post_list + j - 1].addr;

			if ((user_param->tst == BW || user_param->tst == LAT_BY_BW) && user_param->size <= (ctx->cycle_buffer / 2)) {
				increase_loc_addr(&ctx->recv_sge_list[i * user_param->recv_post_list + j],
						user_param->size,
						j,
						ctx->rx_buffer_addr[i],
						user_param->connection_type,ctx->cache_line_size,ctx->cycle_buffer);
			}
		}

		ctx->rwr[i * user_param->recv_post_list + j].sg_list = &ctx->recv_sge_list[i * user_param->recv_post_list + j];
		ctx->rwr[i * user_param->recv_post_list + j].num_sge = MAX_RECV_SGE;
		ctx->rwr[i * user_param->recv_post_list + j].wr_id   = i;

		if (j == (user_param->recv_post_list - 1))
			ctx->rwr[i * user_param->recv_post_list + j].next = NULL;
		else
			ctx->rwr[i * user_param->recv_post_list + j].next = &ctx->rwr[i * user_param->recv_post_list + j + 1];
	}

	for (j = 0; j < size_per_qp ; ++j) {

		if (user_param->use_srq) {

			if (ibv_post_srq_recv(ctx->srq,&ctx->rwr[i * user_param->recv_post_list], &bad_wr_recv)) {
				fprintf(stderr, "Couldn't post recv SRQ = %d: counter=%d\n",i,j);
				return 1;
			}

		} else {

			if (ibv_post_recv(ctx->qp[i],&ctx->rwr[i * user_param->recv_post_list],&bad_wr_recv)) {
				fprintf(stderr, "Couldn't post recv Qp = %d: counter=%d\n",i,j);
				return 1;
			}
		}

		if (user_param->recv_post_list == 1 && (user_param->tst == BW || user_param->tst == LAT_BY_BW) &&
				user_param->size <= (ctx->cycle_buffer / 2)) {
			increase_loc_addr(&ctx->recv_sge_list[i * user_param->recv_post_list],
					user_param->size,
					j,
					ctx->rx_buffer_addr[i],
					user_param->connection_type,ctx->cache_line_size,ctx->cycle_buffer);
		}
	}
	**ctx->recv_sge_list[i * user_param->recv_post_list].addr = ctx->rx_buffer_addr[i];**
}
return 0;

}

because of that line, there will has the same number of wqes as rx-depth use the same buf addr and data will be overwritten

I don't know whether my understanding is correct or not, and whether there are other dependent uses in this line?

@sshaulnv
Copy link
Contributor

Hi @smice-ICE,
thanks for you contribution.
can you mark the line you meant and explain your question more briefly please?

also, we are not preforming validation to the message.

@smiceice
Copy link
Author

smiceice commented Jun 1, 2023

the line is " ctx->recv_sge_list[i * user_param->recv_post_list].addr = ctx->rx_buffer_addr[i];"

in send/recv mode
at first, server side will post tx-depth recv wqes, those wqes addr is start at buf[i]+offset
after then, new rwqes addr is still start at buf[i]+offset
so the first tx-depth wqes and the second tx-depth wqes will use same addr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants