Passthrough on 6.17 kernel

[henrikekblad]

The current script/setup does not seem to work with proxmox kernel 6.17.4-2-pve.

Has anyone figured out what the problem is?

[jammsen]

The current script/setup does not seem to work with proxmox kernel 6.17.4-2-pve.

Has anyone figured out what the problem is?

Hey @henrikekblad this is very vague with no actual infos/facts to base this upon or test this, im not sure how i could even begin to help .... 😕 Screenshots, logs, detailed problem-description, steps to reproduce, please provide some data.

[henrikekblad]

Sorry,

I stumbled over your excellent scripts a while ago and managed to migrate my frigate instance running in a full VM to an lxc container with docker. Everything has been working perfectly! Thank you for the effort putting the scripts together!

The other day I just did a quick test updating the kernel on my Proxmox to the latest version 6.17.4.2 (from 6.14.11-5-pve). Immidiately the CPU went through the roof.

In frigates own UI I saw that the GPU wasn't in use any longer. In heat of the moment I unfortunately did not have the precence to check journald or any other logs. I just quickly reverted back and pinned the old kernel.

Afterwards I saw some other people discussing the new kernel broke passthrought, so I assumed it was "well known":
https://community.frame.work/t/anyone-using-proxmox-ve/74863/22

Have you tried the new kernel yourself and got it working?

[jammsen]

Sorry,

All good 👍

I stumbled over your excellent scripts a while ago and managed to migrate my frigate instance running in a full VM to an lxc container with docker. Everything has been working perfectly! Thank you for the effort putting the scripts together!

Thanks for the flowers 😊 and im very happy to hear that i somehow could help to make life easier for you 😄

The other day I just did a quick test updating the kernel on my Proxmox to the latest version 6.17.4.2 (from 6.14.11-5-pve). Immidiately the CPU went through the roof.

In frigates own UI I saw that the GPU wasn't in use any longer. In heat of the moment I unfortunately did not have the precence to check journald or any other logs. I just quickly reverted back and pinned the old kernel.

Afterwards I saw some other people discussing the new kernel broke passthrought, so I assumed it was "well known": https://community.frame.work/t/anyone-using-proxmox-ve/74863/22

Have you tried the new kernel yourself and got it working?

No i had not the chance to upgrade on my GPU-Server or NAS to this version, i however did that on my Router-Host and i had no problems, very interesting.

I try to not be "not to much bleeding edge" because of stuff like that 😆

Can you confirm its now only working with 6.14.x or also with 6.17?
Very interesting error/problem, just after the runc bug with Docker a few weeks ago 😆

[durzo]

I'm using 6.17.4-1-pve without issue. I'll try to upgrade to 6.17.4-2-pve this weekend and report back

[durzo]

Updated to 6.17.4-2-pve - both nvidia and amd working, passthrough to LXC's working.

[henrikekblad]

Ok, thank for confirimig that it works @durzo. I will have to try it again, while not stressing out...

[henrikekblad]

A quick followup.

I rebooted with the new kernel. CPU peaked again.

When checking the /dev/dri and /dev/kfd rights I noticed that the group had changed to 510. So after editing the lxc conf file with:

lxc.cgroup2.devices.allow: c 510:* rwm

The GPU works again!

So this issue can be closed. But this dicussion might be useful for someone else running into the same problem.

[durzo]

yes my cgroups change every kernel upgrade too, its annoying and I don't know how to prevent it.

[jammsen]

A quick followup.

I rebooted with the new kernel. CPU peaked again.

When checking the /dev/dri and /dev/kfd rights I noticed that the group had changed to 510. So after editing the lxc conf file with:

lxc.cgroup2.devices.allow: c 510:* rwm

The GPU works again!

So this issue can be closed. But this dicussion might be useful for someone else running into the same problem.

yes my cgroups change every kernel upgrade too, its annoying and I don't know how to prevent it.

Hey @durzo & @henrikekblad have you guys seen already #3 ? You think that could be helping you?

[henrikekblad]

@jammsen, It'll surely help in the initial setup.

But if next kernel update changes the cgroup again, It will still require a manual update the lxc conf files.

[durzo]

what @henrikekblad said. every new kernel changes the cgroups and I have to modify my lxc conf files with the new ID's. it doesn't happen often enough so its not a huge deal.

one problem I faced was with nvidia drivers. /dev/nvidia-uvm is not created until the nvidia driver on the proxmox host is asked to use it by a library. this means when my plex lxc boots up, the bind mount for /dev/nvidia-uvm fails because it doesnt exist yet, causing all hw transcoding to fail with an error.

the solution was to call nvidia-smi at every boot on the proxmox host. I did this by creating /etc/cron.d/nvidia-smi with the content:

@reboot root /usr/bin/nvidia-smi > /dev/null

I also have a short delay before any nvidia lxc guests start, just to make sure the cron executes in time.

[jammsen]

what @henrikekblad said. every new kernel changes the cgroups and I have to modify my lxc conf files with the new ID's. it doesn't happen often enough so its not a huge deal.

one problem I faced was with nvidia drivers. /dev/nvidia-uvm is not created until the nvidia driver on the proxmox host is asked to use it by a library. this means when my plex lxc boots up, the bind mount for /dev/nvidia-uvm fails because it doesnt exist yet, causing all hw transcoding to fail with an error.

the solution was to call nvidia-smi at every boot on the proxmox host. I did this by creating /etc/cron.d/nvidia-smi with the content:

@reboot root /usr/bin/nvidia-smi > /dev/null

I also have a short delay before any nvidia lxc guests start, just to make sure the cron executes in time.

Would a nvidia-persistenced service do the same?
Or this workaround:

From: https://digitalspaceport.com/how-to-setup-an-ai-server-homelab-beginners-guides-ollama-and-openwebui-on-proxmox-lxc/

[durzo]

that is the same workaround as i'm already doing - running nvidia-smi on reboot via cron.
I don't really understand why -pm 1 (persistence mode) is needed, as the man page says:

However, it does not persist across reboots. After each reboot persistence mode will default to "Disabled".

and I don't have a problem with it being off, the command still needs to be run once at boot to initialise the uvm device. calling it with or without persistence mode doesn't appear to have any effect whatsoever.

-pm 250 sets a power limit, setting this blindly to 250 might be a bad idea. I have an rtx A400 which has a power limit of 50 watts and already identifies as being unrestricted in that sense when running nvidia-smi

The main thing is needing to run nvidia-smi at boot in order to initialise /dev/nvidia-uvm before your LXC's start. without that, transcoding wont work.

Finally, calling the commands via cron (either /etc/cron.d/ or the users cron via crontab -e) without stdout redirection would cause CROND to email the user with the output every time. To reduce unwanted crud in /var/spool/mail i would use output redirection via >/dev/null