rdma: add option to round robin the ctrl msg, and use shared CQs for control and data endpoints #673
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bwbarrett
previously approved these changes
Oct 23, 2024
AmedeoSapio
force-pushed
the
ctrl_msg_rr_shared_cq
branch
2 times, most recently
from
3940033
to
ae47bd5
Compare
|
bot:aws:retest |
Signed-off-by: Amedeo Sapio <asapio@amazon.com>
PR aws#543 has moved the control message to its own dedicated endpoint on a single rail and with its own CQ. To avoid the cost of polling an additional CQ, this commit is making the control endpoint share the CQ with rail 0. Signed-off-by: Amedeo Sapio <asapio@amazon.com>
PR aws#543 has moved the control message to its own dedicated endpoint on a single rail. As a result, the control message is not sent on all rails in a round-robin fashion anymore. This has impacted performance in some cases, so we want the option to still distribute the control messages across all rails. This commit is providing an environment variable to choose between a) using a single dedicated endpoint for control messages, sharing the CQ with the data endpoint on rail 0; or b) use as many additional endpoints for the control messages as the number of rails, sharing the CQs with the data endpoints, and use the scheduler to round robin the control messages across all rails. Signed-off-by: Amedeo Sapio <asapio@amazon.com>
AmedeoSapio
force-pushed
the
ctrl_msg_rr_shared_cq
branch
from
ae47bd5
to
225a029
Compare
bwbarrett
approved these changes
Oct 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR #543 has moved the control message to its own dedicated
endpoint on a single rail. As a result, the control message is not sent on
all rails in a round-robin fashion anymore. This has impacted performance
in some cases, so we want the option to still distribute the control messages
across all rails.
This PR is providing an environment variable to choose between a) using a
single dedicated endpoint for control messages, sharing the CQ with the data
endpoint on rail 0; or b) use as many additional endpoints for the control
messages as the number of rails, sharing the CQs with the data endpoints, and
use the scheduler to round robin the control messages across all rails.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.