Is NvFBC for X11-only?

[kelvie]

When I try to use NvFBC inside hyprland (a wlroots-based wayland compositor), it seems like it looks for an X server:

[2024:01:24:13:59:45]: Error: Failed to create session: Version mismatch between NvFBC and the X driver interface
[2024:01:24:13:59:45]: Error: Unable to initialize capture method
[2024:01:24:13:59:45]: Error: Platform failed to initialize

Is this expected? What challenges are there to getting this working for wayland (I find that a 4090 can only do up to 4k 20fps if I don't have NvFBC on, unless I misconfigured something).

[cgutman]

NvFBC does not have Wayland or XWayland support according to NVIDIA.

NVIDIA Driver Limitations

• The NvFBC desktop capture library does not have native Wayland support and does not work with Xwayland.

https://us.download.nvidia.com/XFree86/Linux-x86_64/550.40.07/README/wayland-issues.html

[kelvie]

Ah, and I suppose it's generally known that wayland kms or wlgrab is just a lot slower than X11 (at least on nvidia?) Or could there be something wrong with my system, it's pretty beefy, a 5950x with a 4090.

[cgutman]

For Nvidia, yes. For AMD and Intel, no.

The reason is that Nvidia doesn't support VAAPI encoding, so KMS/X11/wlgrab frames need to be read back from the GPU to CPU memory for NVENC encoding. When using KMS/wlgrab+VAAPI or NvFBC+NVENC, we can directly import GPU buffers for encoding without having to read them back to the CPU first.

It should be possible to support importing captured KMS/wlgrab frames into CUDA but it will take some work to implement. I recommend sticking with X11+NvFBC on Nvidia hardware for now.

[cgutman]

I have implemented the NVENC KMS fast path in #2069 and it works great with KMS for me. Please give it a try if you can.

Prebuilt packages are in the Artifacts section here: https://github.com/LizardByte/Sunshine/actions/runs/7683874385?pr=2069

[kelvie]

Well that was amazing, thank you for your hard work, I will be using this to play Palworld on my TV from now on :)

With your branch, I get a full 4k with 120FPS in using wlgrab (with the nvidia 535 driver) from my desktop machine to my phone. CPU is at 30% of a core, and GPU usage is 8%.

[kelvie]

(I didn't test binaries btw, I built it myself from your feature branch)

[fxzzi]

Just tested binaries. Everything working perfectly streaming The Legend of Zelda Breath of the Wild (CEMU) at 1440p60 on Hyprland! :)

Edit: is there some way to implement this capture method for OBS Studio? Or is there limitations around that?

[ReenigneArcher]

@fxzzi they should be able to do something similar.

[cgutman]

Thanks for testing. This is now merged into the nightly branch.