feat(linux): Add Vulkan video encoder for Linux#4603
Open
neatnoise wants to merge 24 commits intoLizardByte:masterfrom
Open
feat(linux): Add Vulkan video encoder for Linux#4603neatnoise wants to merge 24 commits intoLizardByte:masterfrom
neatnoise wants to merge 24 commits intoLizardByte:masterfrom
Conversation
neatnoise
changed the title
neatnoise
changed the title
ReenigneArcher
requested changes
Jan 19, 2026
Member
There was a problem hiding this comment.
This is just a preliminary review. On top of the comments below, I think these files will probably need updates.
update dependencies:
- https://github.com/LizardByte/Sunshine/blob/master/scripts/linux_build.sh
- https://github.com/LizardByte/Sunshine/blob/master/packaging/sunshine.rb
- https://github.com/LizardByte/Sunshine/blob/master/packaging/linux/Arch/PKGBUILD
- https://github.com/LizardByte/Sunshine/blob/master/packaging/linux/copr/Sunshine.spec
- https://github.com/LizardByte/Sunshine/blob/master/.github/workflows/ci-freebsd.yml
- and maybe the flatpak
Member
|
I will convert this to draft for now, please change it back when you're ready for a full review. Thanks! |
Author
|
Its ready |
Contributor
|
I know this PR is about encoding, but I saw changes to capture as well. Do you know if this could support variable framerate capture, similar to how capture works on Windows? In my limited experience Linux always captures at the stream framerate. The ideal capture process would be capturing only when the image has changed (with duplicated frames subject to minimum_fps_target), as well as capturing using a frame interval timer requested by the client or based on the framerate. For example, an Xbox client works best if frames are captured on a timer with an interval of 59.94fps (60000/1001), a desktop client could just use a 60fps interval. One thing you do not want to do is base anything on the host's vsync interval. |
Member
I left a bunch of review comments. |
Author
Yes, the kms_vblank option already behaves this way with variable frame rate similarly to Windows. It encodes that many frames that are generated by a game. It also respects fps frame rate which are set by a client (if the clients sets 60, it won't generate more than 60 fps with some minimal margin) |
Author
|
@ReenigneArcher I made proposed changes. I also added a tiling fix for RNDA 4 when Gnome DE is used. Would be helpful if somebody with Nvidia and Intel GPUs could test the new encoder |
21 tasks
Author
|
@ReenigneArcher I've tested with vulkan ffmpeg build https://github.com/LizardByte/build-deps/actions/runs/21850415771?pr=606 successfully. Tweaked kms_vblank for the game Cronos The New Dawn meanwhile. Tested in many games with various resolutions with and without vsync in-game and kms_vblank enabled and disabled. Tested with vulkan and vaapi encoders. It works solid. |
This comment was marked as off-topic.
This comment was marked as off-topic.
Member
|
This PR needs to be rebased to pick up the right FFmpeg pre-build, before I can run CI. Edit: actually the build-deps commit that added Vulkan hadn't been merged here yet: #4735 |
Contributor
|
Also, don't forget to verify compatibility with portalgrab when rebasing. This is enough to get started, but you might want to refine/verify: |
Author
|
@XT-Martinez It looks like ffmpeg without built-in vulkan support, you need to build it with ffmpeg vulkan supported binaries (like from link above). @psyke83 will do. I will add vulkan support also for portal grab. Actually I have my portal/pipeware implementation here https://github.com/neatnoise/Sunshine/tree/vulkan%2Bpipewire . So It will be similar. |
Author
The best results on my machine are portal+vulkan (vulkan settings low latency + vbr) and existing portal+vaapi. Portal seems to have a lower latency than KMS. |
Contributor
|
FWIW I found the vblank option quite interesting. (But I don't have much time right now to actually check it out.) |
Contributor
|
With the default settings (vbr, ll), the stream is massively violating the target bitrate (70Mbps target is saturating my wifi connection at 250Mbps+). I didn't notice this with prior testing. |
Contributor
|
This is the issue: Your current PR is assigning VBR as 3 erroneously. When set to 4, I'm not seeing the crazy bitrate overshooting. |
psyke83
reviewed
Feb 20, 2026
Author
Fixed |
21 tasks
Member
|
|
Author
|
@ReenigneArcher rebased with the current master. It works fine: |
ReenigneArcher
requested changes
Feb 24, 2026
Member
ReenigneArcher
left a comment
ReenigneArcher
left a comment
There was a problem hiding this comment.
Thank you, I have a review below.
In addition to my review, I have a question. Will there be issues if there is a mismatch in the vulkan header and vulkan loader versions here versus what is used in the build-deps (FFmpeg) repo?
src_assets/common/assets/web/configs/tabs/encoders/VulkanEncoder.vue
Outdated
Show resolved
Hide resolved
src_assets/common/assets/web/configs/tabs/encoders/VulkanEncoder.vue
Outdated
Show resolved
Hide resolved
src_assets/common/assets/web/configs/tabs/encoders/VulkanEncoder.vue
Outdated
Show resolved
Hide resolved
src_assets/common/assets/web/configs/tabs/encoders/VulkanEncoder.vue
Outdated
Show resolved
Hide resolved
Author
The build-deps repo bundles Vulkan Headers/Loader v1.4.344 (statically linked into FFmpeg), while my Sunshine compiles against the system's v1.4.341 (CachyOS distro). The two |
- Remove format qualifiers from compute shader to support both NV12 (8-bit) and P010 (10-bit) with a single SPIR-V binary - Detect P010 from FFmpeg sw_format and use R16/RG16 target views - Rename shader from rgb2nv12 to rgb2yuv to reflect format-agnostic nature
Remove NO_RC_BUF_LIMIT flag from the Vulkan encoder so that rc_buffer_size is set to a single-frame VBV by the generic encoder setup. This constrains the hardware rate controller from producing oversized frames after periods of bitrate undershoot.
When VBR rc_mode is selected, rc_min_rate was pinned to the same value as rc_max_rate and bit_rate. This prevented the encoder from undershooting on simple frames, causing accumulated headroom that led to large bitrate spikes on complex frames. Clear rc_min_rate to 0 in VBR mode so the encoder can freely vary bitrate downward, eliminating the budget buildup that causes overshoot.
Cap rc_max_rate to 120% of target bitrate in VBR mode so FFmpeg's Vulkan encoder passes a meaningful maxBitrate ceiling to the driver. Without this, maxBitrate == averageBitrate gives the driver no explicit limit, causing hardware-dependent spikes (e.g. 26->46 Mbit). Also add NO_RC_BUF_LIMIT flag since FFmpeg's Vulkan encoder does not read rc_buffer_size (it uses virtualBufferSizeInMs which it never populates from AVCodecContext).
- Replace C-style arrays with std::array across vulkan_encode.cpp, rgb2yuv.spv.h, and video.cpp - Refactor vk_vram_t from 35 fields to 19 by grouping into sub-structs (vk_device_t, compute_pipeline_t, cmd_submission_t, target_state_t) - Use C++17 if-init-statement for render_path and merge nested if - Add override to ~vk_vram_t() destructor - Add & operator to init_hw_device function pointer reference - Rename set_frame parameter to avoid shadowing member field - Replace redundant explicit types with auto where cast provides type - Remove redundant VkAccessFlagBits cast - Split multi-variable declarations into separate statements - Replace typedef with using alias in video.cpp - Remove empty style block from VulkanEncoder.vue
Remove the 120% rc_max_rate cap added in 5d1ee1a. Both vulkan-pr and vulkan+control now share the same overshoot fix: only rc_min_rate=0 in VBR mode.
ReenigneArcher
force-pushed
the
vulkan-pr
branch
from
aad9143 to
8fbc7d2
Compare
Bundle ReportChanges will increase total bundle size by 2.02kB (0.26%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: sunshine-esmAssets Changed:
Files in
Files in
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #4603 +/- ##
==========================================
- Coverage 17.91% 16.63% -1.28%
==========================================
Files 106 107 +1
Lines 21980 21883 -97
Branches 9812 9730 -82
==========================================
- Hits 3937 3640 -297
- Misses 13335 15129 +1794
+ Partials 4708 3114 -1594
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Member
|
The failed tests check that the config is setup properly in all places (cpp, html, docs) Also, a number of lint errors: https://github.com/LizardByte/Sunshine/actions/runs/23094267559/job/67084016198?pr=4603 |
- cmake-lint: remove set(Vulkan_FOUND OFF), use SUNSHINE_ENABLE_VULKAN guard - clang-format: apply formatting to vulkan_encode.cpp and config.h - trailing whitespace: clean blank line in video.cpp - docs: add Vulkan Encoder section with vk_tune and vk_rc_mode - tests: add Vulkan Encoder tab-to-section mapping
Author
|
@ReenigneArcher fixed, please check now |
Keep both vulkan-headers (our addition) and python-jinja/python-setuptools (upstream addition) in Arch PKGBUILD.
|
Tried this out on cachyos using rx 9070 xt and working really well on KMS! however portal appears as a green window and immediately disconnects, while wlroots produces a mangled picture running in sway gles2 (expected?). Very positive feedback from my side on KMS, I'll dig into my other issues and get some logs. |
Contributor
|
Tested latest PR changes and it's working well on RX 6600 via KMS and Portal.
It's a known transient issue that's not the fault of this PR. Workaround is available: #4824 (comment) |
Member
ReenigneArcher
left a comment
ReenigneArcher
left a comment
There was a problem hiding this comment.
Just have a few comments. I think this is pretty close to being able to merge!
Sway's GLES2 renderer exports framebuffers in OpenGL's bottom-up convention and signals this via the Y_INVERT flag. The wlroots capture stored this flag but never passed it to the Vulkan encoder, resulting in a mangled picture. Propagate y_invert through img_descriptor_t to the Vulkan compute shader, which now flips the source texture Y coordinate when set.
Author
|
@TerjeTL could you test wlroots with sway gles2? I applied the fix |
|
@neatnoise pulled the fix and made a new build, can't see any errors in the logs but this is the image i get when i run in that config. Not very confident i haven't messed up on my end, only been running Linux for a few weeks, but does work with main branch. Steam Big Picture (supposedly :p) |
Author
|
When wlroots used GBM can silently allocate tiled buffers but report an unknown modifier. Vulkan then imports the buffer assuming linear layout while the actual memory is tiled. I made a change again @TerjeTL . We might stick with KMS and portal for now otherwise. |
Force linear GBM buffer allocation to prevent tiling/modifier mismatch when importing DMA-BUF into Vulkan.
|
@neatnoise That solved the issue on my end, image looks clean now! Thanks for the rapid fixes :) |
neatnoise
force-pushed
the
vulkan-pr
branch
2 times, most recently
from
fcfb1df to
8f98e0d
Compare
… SPIR-V - Move rgb2yuv.comp to src_assets/linux/assets/shaders/vulkan/ - Remove pre-compiled .spv and .spv.h from repo - Add CMake rules to compile .comp -> .spv -> C include at build time - Prefer glslc, fall back to glslangValidator for Ubuntu 22.04 support - Add shader compiler build dependencies for all supported distros
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds Vulkan video encoding support for Linux, providing an alternative to VAAPI encoding.
Changes:
h264_vulkan,hevc_vulkan,av1_vulkan) with zero-copy DMA-BUF import and Vulkan compute RGB-to-YUV conversion — no EGL/GL dependency, all GPU work stays in a single Vulkan queuevulkanencoder options indocs/configuration.mdDependencies:
Requires corresponding PR in LizardByte/build-deps (
ffmpeg-vulkanbranch) to enableBUILD_FFMPEG_VULKANoption, which enables Vulkan encoder support in FFmpeg builds.Testing:
Screenshot
Issues Fixed or Closed
Roadmap Issues
Type of Change
Checklist
AI Usage