NVIDIA GPU in microk8s in WSL #3024
Comments
|
@maximehyh |
|
I hit a similar issue. I tried the following: sudo snap install microk8s --classic
sudo usermod -a -G microk8s ${USER}
sudo chown -R ${USER} ~/.kube
newgrp microk8s
vim /var/snap/microk8s/current/args/kubelet
# Replace --container-runtime=remote with --container-runtime=docker
microk8s stop
microk8s start
cat /var/snap/microk8s/current/args/kubelet
# Make sure container-runtime is docker
microk8s enable gpu
microk8s kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: cuda-vector-add
spec:
restartPolicy: OnFailure
containers:
- name: cuda-vector-add
image: "k8s.gcr.io/cuda-vector-add:v0.1"
resources:
limits:
nvidia.com/gpu: 1
EOFi.e., I switched the runtime to Docker, since my Nvidia Container Runtime works well in WSL2 and hoped it would be able to pick up the GPU from there. However, my pod remains in status I think this would be a really nice use case to enable - many people develop on Windows via WSL2 and Ubuntu. If microk8s with GPU acceleration was a two liner that would be very convenient and potentially attract many new users. It is also entirely possible we just miss a minor detail which prevents this from working. Optional Side Observation: Same Happens With Minikube curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
sudo dpkg -i minikube_latest_amd64.deb
minikube start --driver=docker --container-runtime=docker --addons=ingress --memory=8192
kubectl version
kubectl get pod
kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/master/nvidia-device-plugin.yml
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: cuda-vector-add
spec:
restartPolicy: OnFailure
containers:
- name: cuda-vector-add
image: "k8s.gcr.io/cuda-vector-add:v0.1"
resources:
limits:
nvidia.com/gpu: 1
EOFThe pod also gets stuck in status $ kubectl get pod
NAME READY STATUS RESTARTS AGE
cuda-vector-add 0/1 Pending 0 90s |
Before getting into Microk8s, what does your nvidia-smi in WSL say? |
|
Have the same issue. I observe that before enabling the gpu add-on |
|
Hi @valxv Can you please share the output of Also, could you try enabling GPU as |
|
Hi @neoaggelos Thanks for the prompt response. The I install MicroK8s v1.27.2 following the Get Started instructions. Everything goes smoothly and I get a working MicroK8s instance. And enabling GPU without But operator's pods logs look the same. inspection-report-20230707_090001.tar.gz Please write me down in case you need any additional information. |
|
Pardon my ignorance, but does your GPU support CUDA? I don't see it in https://developer.nvidia.com/cuda-gpus#compute (unless I missed it) If the GPU should be supported, then I believe the right place to open an issue would be https://github.com/nvidia/gpu-operator Hope this helps! |
|
Yes, it is supported. This NVIDIA page you're referring to is a little bit outdated. I don't have any problems running CUDA apps locally and inside Docker containers. The only issue I have is with Kubernetes. :) |
|
OK, I am not too familiar with the details. Thanks @valxv, happy to keep this issue around so that we are in the loop and see if there's anyway to support from the microk8s side. |
|
Does this issue have been resolved ? I'm actually running into the same use case. Even k3s can't detect the gpu into the wsl2 node |
|
This error can be resolved by building the Ubuntu kernel, but the gpu operator still says that it cannot find the gpu node. |
|
Hey is there any Update on this? I am facing the same issue on microk8s 1.24. |
|
Same issue - any ideas? |
|
same issue :( |
|
Same issue for me, see following output. The issue is unable to access the GPU path, which doesn't exist in WSL2. failed to detect NUMA nodes: failed to list numa nodes: open /host-sys/bus/node/devices: no such file or directory (base) eval@rtx-4090-1:$ microk8s enable gpu --driver=host (base) eval@rtx-4090-1:$ microk8s.kubectl get pods -n gpu-operator-resources |
|
meet same issue and i'm here |
|
same issue any idea? WSL with microk8s installed |
|
+1
and |
Hello,
I am looking for a way to run some Machine Learning Inference within a Kubernetes cluster on Windows. Microk8s seemed to be a good solution as I saw that there was a gpu add on. I did some experiments with the documented way of running microk8s on windows using
multipassbut quickly realized that enabling GPU usage within multipass could bring some difficulties (canonical/multipass#2503 (comment)).Knowing that CUDA can be enabled within the WSL and seeing some threads about installing microk8s on WSL (meaning overcoming the lack of
snap) I started to try building a Kubernetes cluster within the WSL using microk8s.Everything went smoothly until I actually deployed the pod that needed access to the GPU. The pod can not managed to be deployed,
describe deploymentgives the following error:0/1 nodes are available: 1 Insufficient nvidia.com/gpu.(I didmicrok8s enable gpuand addedlimits: nvidia.com/gpu: 1in the deployment.yaml as described in the doc).I have not put too much effort yet on trying to make it work as I figured that my assumption that having the GPU available in the WSL would allow to have GPU enabled within Kubernetes could be wrong.
Do you know if such a setup could work?
Below the requested logs.
Thanks!
inspection-report-20220401_223428.tar.gz
The text was updated successfully, but these errors were encountered: