deployer: add info generating commands under resource-allocation #3337
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment was marked as resolved.
This comment was marked as resolved.
consideRatio
force-pushed
the
pr/update-node-info
branch
2 times, most recently
from
d7fcda1 to
83279dd
Compare
GeorgianaElena
approved these changes
Oct 30, 2023
deployer/commands/generate/resource_allocation/daemonset_overhead.py
Outdated
Show resolved
Hide resolved
deployer/commands/generate/resource_allocation/daemonset_overhead.py
Outdated
Show resolved
Hide resolved
deployer/commands/generate/resource_allocation/daemonset_overhead.yaml
Outdated
Show resolved
Hide resolved
|
Thank you for reviewing this @GeorgianaElena! I've not resolved all comments, but will get it done soonish! |
Co-authored-by: Georgiana <georgiana.dolocan@gmail.com>
consideRatio
force-pushed
the
pr/update-node-info
branch
from
5854d47 to
baaea25
Compare
|
Thank you for reviewing this @GeorgianaElena!!!
I'll go for a merge here at this point! |
|
🎉🎉🎉🎉 Monitor the deployment of the hubs here 👉 https://github.com/2i2c-org/infrastructure/actions/runs/6754266612 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds two commands of relevance towards refining the resource-allocation script to meet more needs. The added commands are updating their separate .yaml file's with collected information.
Script overview
daemonset-requestssummarizes the requests from all daemonsets, and clarifies what daemonsets contributed to the requests and what daemonset didn't.The script makes us better understand and clarify differences observed. It helped me verify that the current overhead is mostly cloud provoider specific, to some extent cluster specific (feature and k8s version differences), and to a small extent instance type/node pool specific (gpu drivers).
A caveat of this implementation strategy is that it doesn't intelligently rule out that a daemonset may not only be schedule on some nodes, for example core nodes only, or on nodes with gpus. I've filtered out the gpu related daemonsets manually, and not observed any core node pool only daemonsets.
instance-capacitiesscrapes thekubectl get nodereportedcapacityandallocatable(allocatable capacity), and saves both the lowest and highest reported value to help us overview the spread of these values across clusters. It turns out that there is a slight spread, but its very low.With
update-node-info, why these commands?These are iterations on
update-node-info, but they are added separately asupdate-node-infodirectly hooks into theresource-allocation choicescommand, making an update to it break the existing behavior.