Error: Couldn't initialize EGL display: [00003001] - HW decoding with Nvidia on Linux?

[catthou]

Hi, can't get hardware encoding to work with this problem: Error: Couldn't initialize EGL display: [00003001] I don't know much more than what's in the log. Config, log, and vainfo below.

OS: Gentoo Linux 6.6.21
arch: amd64
Sunshine version: 0.22.0; locally compiled from gentoo repo

Graphics card: Nvidia GTX 1050 Ti
Driver version: 535.161.07
CUDA Version: 12.2

Config:

$ cat sunshine.conf 
sw_preset = veryfast
min_threads = 8
qp = 23
ping_timeout = 3000
origin_web_ui_allowed = pc
fec_percentage = 10
resolutions = [
    1280x720,
    1920x1080,
    2560x1440,
    1366x768
]
output_name = 0
nvenc_preset = 2
fps = [30,60,25]
native_pen_touch = disabled
channels = 2

sunshine.log:

[2024:03:17:11:40:39]: Info: Sunshine version: 0.22.0
[2024:03:17:11:40:39]: Info: System tray created
[2024:03:17:11:40:39]: Error: Failed to create session: This hardware does not support NvFBC
[2024:03:17:11:40:39]: Warning: No render device name for: /dev/dri/card0
[2024:03:17:11:40:39]: Info: Detecting monitors
[2024:03:17:11:40:39]: Info: Detected monitor 0: DVI-D-0, connected: false
[2024:03:17:11:40:39]: Info: Detected monitor 1: HDMI-0, connected: true
[2024:03:17:11:40:39]: Info: Detected monitor 2: DP-0, connected: true
[2024:03:17:11:40:39]: Info: Detected monitor 3: DP-1, connected: false
[2024:03:17:11:40:39]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2024:03:17:11:40:39]: Info: Trying encoder [nvenc]
[2024:03:17:11:40:39]: Info: Screencasting with KMS
[2024:03:17:11:40:39]: Warning: No render device name for: /dev/dri/card0
[2024:03:17:11:40:39]: Info: Found monitor for DRM screencasting
[2024:03:17:11:40:39]: Error: Failed to determine panel orientation, defaulting to landscape.
[2024:03:17:11:40:39]: Info: Found connector ID [35]
[2024:03:17:11:40:39]: Warning: No KMS cursor plane found. Cursor may not be displayed while streaming!
[2024:03:17:11:40:39]: Error: Couldn't initialize EGL display: [00003001]
[2024:03:17:11:40:39]: Error: Couldn't initialize EGL display: [00003001]
[2024:03:17:11:40:39]: Info: Encoder [nvenc] failed
[2024:03:17:11:40:39]: Info: Trying encoder [vaapi]
[2024:03:17:11:40:39]: Info: Screencasting with KMS
[2024:03:17:11:40:39]: Warning: No render device name for: /dev/dri/card0
[2024:03:17:11:40:39]: Info: Found monitor for DRM screencasting
[2024:03:17:11:40:39]: Error: Failed to determine panel orientation, defaulting to landscape.
[2024:03:17:11:40:39]: Info: Found connector ID [35]
[2024:03:17:11:40:39]: Warning: No KMS cursor plane found. Cursor may not be displayed while streaming!
[2024:03:17:11:40:39]: Error: Couldn't initialize va display: unknown libva error
[2024:03:17:11:40:39]: Warning: Monitor 0 doesn't support hardware encoding. Reverting back to GPU -> RAM -> GPU
[2024:03:17:11:40:39]: Warning: No render device name for: /dev/dri/card0
[2024:03:17:11:40:39]: Info: Found monitor for DRM screencasting
[2024:03:17:11:40:39]: Error: Failed to determine panel orientation, defaulting to landscape.
[2024:03:17:11:40:39]: Info: Found connector ID [35]
[2024:03:17:11:40:39]: Warning: No KMS cursor plane found. Cursor may not be displayed while streaming!
[2024:03:17:11:40:39]: Error: Couldn't initialize EGL display: [00003001]
[2024:03:17:11:40:39]: Error: Couldn't initialize EGL display: [00003001]
[2024:03:17:11:40:39]: Error: Couldn't initialize EGL display: [00003001]
[2024:03:17:11:40:39]: Error: Couldn't initialize EGL display: [00003001]
[2024:03:17:11:40:39]: Info: Encoder [vaapi] failed
[2024:03:17:11:40:39]: Info: Trying encoder [software]
[2024:03:17:11:40:39]: Info: Screencasting with KMS
[2024:03:17:11:40:39]: Warning: No render device name for: /dev/dri/card0
[2024:03:17:11:40:39]: Info: Found monitor for DRM screencasting
[2024:03:17:11:40:39]: Error: Failed to determine panel orientation, defaulting to landscape.
[2024:03:17:11:40:39]: Info: Found connector ID [35]
[2024:03:17:11:40:39]: Warning: No KMS cursor plane found. Cursor may not be displayed while streaming!
[2024:03:17:11:40:39]: Info: SDR color coding [Rec. 601]
[2024:03:17:11:40:39]: Info: Color depth: 8-bit
[2024:03:17:11:40:39]: Info: Color range: [JPEG]
[2024:03:17:11:40:39]: Info: [libx264 @ 0x5560a3e27740] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[2024:03:17:11:40:39]: Info: [libx264 @ 0x5560a3e27740] profile High, level 4.2, 4:2:0, 8-bit
[2024:03:17:11:40:39]: Info: [libx264 @ 0x5560a3e27740] frame I:1     Avg QP:31.00  size:  1495
[2024:03:17:11:40:39]: Info: [libx264 @ 0x5560a3e27740] mb I  I16..4:  0.0% 99.9%  0.1%
[2024:03:17:11:40:39]: Info: [libx264 @ 0x5560a3e27740] 8x8 transform intra:99.9%
[2024:03:17:11:40:39]: Info: [libx264 @ 0x5560a3e27740] coded y,uvDC,uvAC intra: 0.1% 0.0% 0.0%
[2024:03:17:11:40:39]: Info: [libx264 @ 0x5560a3e27740] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu:  0%  0% 100%  0%  0%  0%  0%  0%  0%
[2024:03:17:11:40:39]: Info: [libx264 @ 0x5560a3e27740] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu:  0%  0% 100%  0%  0%  0%  0%  0%  0%
[2024:03:17:11:40:39]: Info: [libx264 @ 0x5560a3e27740] i8c dc,h,v,p: 100%  0%  0%  0%
[2024:03:17:11:40:39]: Info: [libx264 @ 0x5560a3e27740] kb/s:717.60
[2024:03:17:11:40:39]: Info: Screencasting with KMS
[2024:03:17:11:40:39]: Warning: No render device name for: /dev/dri/card0
[2024:03:17:11:40:39]: Info: Found monitor for DRM screencasting
[2024:03:17:11:40:39]: Error: Failed to determine panel orientation, defaulting to landscape.
[2024:03:17:11:40:39]: Info: Found connector ID [35]
[2024:03:17:11:40:39]: Warning: No KMS cursor plane found. Cursor may not be displayed while streaming!
[2024:03:17:11:40:39]: Info: 
[2024:03:17:11:40:39]: Info: // Ignore any errors mentioned above, they are not relevant. //
[2024:03:17:11:40:39]: Info: 
[2024:03:17:11:40:39]: Info: Found H.264 encoder: libx264 [software]
[2024:03:17:11:40:39]: Info: Adding avahi service Sunshine
[2024:03:17:11:40:39]: Info: Configuration UI available at [https://localhost:47990]
[2024:03:17:11:40:40]: Info: Avahi service Sunshine successfully established.

VA Info:

$ vainfo --display x11
Trying display: x11
libva info: VA-API version 1.20.0
libva error: vaGetDriverNames() failed with unknown libva error
libva info: User environment variable requested driver 'nvidia'
libva info: Trying to open /usr/lib64/va/drivers/nvidia_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.20 (libva 2.20.1)
vainfo: Driver version: VA-API NVDEC driver [egl backend]
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileVC1Simple              :	VAEntrypointVLD
      VAProfileVC1Main                :	VAEntrypointVLD
      VAProfileVC1Advanced            :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileHEVCMain               :	VAEntrypointVLD
      VAProfileVP9Profile0            :	VAEntrypointVLD
      VAProfileHEVCMain10             :	VAEntrypointVLD
      VAProfileHEVCMain12             :	VAEntrypointVLD
      VAProfileVP9Profile2            :	VAEntrypointVLD

$ vainfo --display drm
Trying display: drm
libva info: VA-API version 1.20.0
libva info: User environment variable requested driver 'nvidia'
libva info: Trying to open /usr/lib64/va/drivers/nvidia_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.20 (libva 2.20.1)
vainfo: Driver version: VA-API NVDEC driver [egl backend]
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileVC1Simple              :	VAEntrypointVLD
      VAProfileVC1Main                :	VAEntrypointVLD
      VAProfileVC1Advanced            :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileHEVCMain               :	VAEntrypointVLD
      VAProfileVP9Profile0            :	VAEntrypointVLD
      VAProfileHEVCMain10             :	VAEntrypointVLD
      VAProfileHEVCMain12             :	VAEntrypointVLD
      VAProfileVP9Profile2            :	VAEntrypointVLD

DRI devices:

$ ls -l /dev/dri/
total 0
drwxr-xr-x  2 root root        100 Mar 17  2024 by-path
crw-rw----+ 1 root video  226,   0 Mar 17 10:35 card0
crw-rw----+ 1 root video  226,   1 Mar 17  2024 card1
crw-rw-rw-  1 root render 226, 128 Mar 17  2024 renderD128