[release-3.6] Replace nvidia-persistenced service with parallelcluster_nvidia service to avoid conflicts with DLAMI #2341
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## release-3.6 #2341 +/- ##
============================================
Coverage 70.01% 70.01%
============================================
Files 13 13
Lines 1834 1834
============================================
Hits 1284 1284
Misses 550 550
Flags with carried forward coverage won't be shown. Click here to find out more. |
francesco-giordano
force-pushed
the
release-3.6
branch
3 times, most recently
from
409b095
to
0ceaf08
Compare
hanwen-pcluste
previously approved these changes
Jun 29, 2023
francesco-giordano
force-pushed
the
release-3.6
branch
from
0ceaf08
to
438a5ce
Compare
…ce to avoid conflicts with DLAMI parallelcluster_nvidia service ensures the creation of the block devices /dev/nvidia0 and it is needed by the slurmd service. parallelcluster_nvidia starts the nvidia-persistenced or run nvidia-smi to avoid race condition with other services. Signed-off-by: Francesco Giordano <giordafr@amazon.it>
francesco-giordano
force-pushed
the
release-3.6
branch
from
438a5ce
to
363d9f0
Compare
gmarciani
reviewed
Jun 29, 2023
| # Check if a process is running | ||
| # | ||
| def is_process_running(process_name) | ||
| ps = Mixlib::ShellOut.new("ps aux | grep '#{process_name}' | egrep -v \"grep .*#{process_name}\"") |
There was a problem hiding this comment.
minor: we could have used pgrep to match processes by name instead of grepping twice on ps.
| NVIDIA | ||
| mode '0644' | ||
| action :create | ||
| variables(is_nvidia_persistenced_running: is_process_running('nvidia-persistenced')) |
There was a problem hiding this comment.
What happens if nvidia-persistenced is run but parallelcluster_nvidia.service is already running?
| NVIDIA | ||
| mode '0644' | ||
| action :create | ||
| variables(is_nvidia_persistenced_running: is_process_running('nvidia-persistenced')) |
There was a problem hiding this comment.
What happens if nvidia-persistenced is run but parallelcluster_nvidia.service is already running?
gmarciani
approved these changes
Jun 29, 2023
enrico-usai
pushed a commit
to enrico-usai/aws-parallelcluster-cookbook
that referenced
this pull request
Jul 7, 2023
`parallelcluster_nvidia` service ensures the creation of the block devices `/dev/nvidia0` and it is needed by the `slurmd` service. `parallelcluster_nvidia` starts the `nvidia-persistenced` or runs `nvidia-smi` to avoid race condition with other services and avoids conflicts when using DLAMI with a gpu instance. ### Tests * Modified ChefSpec to verify new changes. ### References Backport of: aws#2341 Signed-off-by: Enrico Usai <usai@amazon.com>
enrico-usai
changed the title
enrico-usai
pushed a commit
that referenced
this pull request
Jul 7, 2023
`parallelcluster_nvidia` service ensures the creation of the block devices `/dev/nvidia0` and it is needed by the `slurmd` service. `parallelcluster_nvidia` starts the `nvidia-persistenced` or runs `nvidia-smi` to avoid race condition with other services and avoids conflicts when using DLAMI with a gpu instance. ### Tests * Modified ChefSpec to verify new changes. ### References Backport of: #2341 Signed-off-by: Enrico Usai <usai@amazon.com>
rkilpadi
pushed a commit
that referenced
this pull request
Jul 21, 2023
`parallelcluster_nvidia` service ensures the creation of the block devices `/dev/nvidia0` and it is needed by the `slurmd` service. `parallelcluster_nvidia` starts the `nvidia-persistenced` or runs `nvidia-smi` to avoid race condition with other services and avoids conflicts when using DLAMI with a gpu instance. ### Tests * Modified ChefSpec to verify new changes. ### References Backport of: #2341 Signed-off-by: Enrico Usai <usai@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes
nvidia-persistenced.servicewithparallelcluster_nvidiaservice which can:/usr/bin/nvidia-persistencedif no other/usr/bin/nvidia-persistencedare already running./usr/bin/nvidia-smiwhich triggers the/dev/nvidia0creation but does not conflict with other servicesThis allow ParallelCluster to have a service which
slurmdcan depends on. However the service will not have race conditions with other possible customer nvidia daemon.Tests
us-east-1withami-0901c773cf8fa8cb6andPlease review the guidelines for contributing and Pull Request Instructions.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.