Constant stuttering while streaming 4K @ 60 FPS on AMD Radeon RX 6700 XT

[kodemeister]

There is something that puzzles me in recent months. I experience regular stuttering every ~4 seconds with 4K 60 FPS streams. 2K streams stutter as well but it's less noticeable. The framerate oscillates from ~57 to 60 similar to #193 but this one seems to be a completely different issue. My setup is:

• Arch Linux
• AMD Radeon RX 6700 XT
• GigE
• Nvidia Shield
• Sunshine Nightly

capture = kms
encoder = vaapi
sunshine_name = DESKTOP
output_name = 0

Moonlight reports 0.0% frame loss, 1-2 ms of network latency and 2-3 ms of decoding time, so this is definitely not a network or client issue.

I did some benchmarking of FFmpeg/VAAPI encoder performance, in particular measured the execution time of encode function in video.cpp. Here is the frametime graph. X axis denotes a frame number while Y axis shows how much time in milliseconds it took to encode the frame.

As we can see, the GPU usually spends 10-13 ms to encode most of frames. However, there are many peaks above 35 ms that cause frame skipping and constant stuttering. Moreover, all these peaks occur exactly every 256 frames, i.e. about 4.3 seconds at 60 FPS! The number 256 is so magical that it can't be a coincidence. There must be some settings/constants in the underlying encoder implementation that cause peaks to appear so regularly.

BTW streaming at 1080p is not that different. Sure, the stream runs much smoother and the frame encoding time is always below 16 ms but all the peaks are still there.

I managed to find only a silly workaround so far. Before starting the stream, I open a small H.264 video in Haruna (a media player based on mpv), minimize Haruna window and keep it playing in the background. Then I run Sunshine and start streaming as usual. This magically fixes all stuttering issues as shown in the graph below:

This looks much better! No more annoying peaks every 4 seconds.

The media player should use VAAPI hardware decoding otherwise this hack won't work. It seems like even the slightest usage of the hardware decoder somehow forces the hardware encoder to produce much better results. All of this makes me think the issue is not related to Sunshine or FFmpeg but is buried somewhere deeper in libva or amdgpu.

All in all, this is just a hack that might stop working someday. Any ideas what could cause the stuttering and how to properly fix it?

[ReenigneArcher]

I wonder if this would help improve things? LizardByte/Sunshine#1043

Note, it's quite outdated... so it might take a little bit of modification to bring it into current nightly.

[kodemeister]

My final attempt: forcing

ctx->keyint_min = 1024;
ctx->gop_size = 1024;

I've also played with idr_interval, ctx->keyint_min, ctx->gop_size a few months ago. Tried to change them to smaller values like 100 or 1000 but had no luck 😕 There is also a flag AV_FRAME_FLAG_KEY which supposedly forces IDR frames. But it was always unset in my tests:

https://github.com/LizardByte/Sunshine/blob/59ce8deb977225d07539a6c656903650acbd8f3e/src/video.cpp#L312-L319

I can also add that I must have inadvertently fallen back to h264(?) at some point in my earlier attempts. Because in all my recent attempts, moonlight's onscreen statistics (host processing delay) are now toggling between 10-11 ms for 4k@60 (regular frames) and then every ~4 seconds (i.e. 256 frames...) it shoots up to ~35ms.

Yep, you seemed to use H.264, that's why frame timings were less dramatic. HEVC typically requires more processing time and causes larger spikes. On my hardware HEVC also takes ~35ms to encode "slow" frames at 4K 60Hz.

I suppose most people stream at 1080p and simply don't notice such issues. Even "slow" frames seem to always fit 16.6ms limit for both H.264 and HEVC. Unfortunately things become much worse while streaming 1440p and 4K. Slow frames are always consecutive rather than evenly distributed over the entire 256 frame interval. They sum up together to 175-200ms delay which is very noticeable.

[kodemeister]

Okay, I made a minimal reproducible example of latency spikes based on FFmpeg vaapi_encode sample.

@gschintgen @cupown Could you please try the following code and confirm it causes the same latency spikes as Sunshine?

https://gist.github.com/kodemeister/684c312fe420db629c1775a5c06837fe

Compile it:

gcc -O2 vaapi_encode.c -ovaapi_encode -lavcodec -lavutil

And run it:

./vaapi_encode 3840 2160 60 100 output.csv

This will generate random frames, encode them to H.264 video stream 3840x2160 @ 60 FPS of 100 seconds in length, measure the encoding time of each frame and write benchmark results to output.csv.

On my machine it behaves pretty much like Sunshine. The encoder load is about 50-60%. VCLK/DCLK clocks fluctuate between 362/316 MHz and 1400/1225 MHz. Playing H/W-accelerated video in background "fixes" latency spikes as usual.

By default the example limits the encoder rate with given FPS value. If I comment out the throttling code:

sleep_duration = (1.0 / fps - (end_frame_time - start_frame_time)) * 1000000.0;
if (sleep_duration > 0.0)
    usleep((useconds_t)sleep_duration);

Then the latency spikes are gone. The encoder runs at full speed with 100% load and VCLK/DCLK are locked to 1400/1225 MHz.

So yeah, it seems to be a power management issue rather than Sunshine/FFmpeg bug. H/W encoder works well only in transcoding scenarios but not when streaming games. If there is some delay between frames, GPU downclocks too aggressively and causes high encoding latencies.

[gschintgen]

Great! I'll test it later. This is probably also of interest to @DistantThunder who has a 7xxx series AMD card and potentially related issues (#2348).

[gschintgen]

@kodemeister, your vaapi_encode is working nicely! I collected data for h264-1080p, h264p-4k, h265-1080p and h265-4k. (For h265 I replaced h264 by hevc in the source.) In all 4 the 255-frame-rhythm can be seen, but it's most dramatic at 4k resolution.

[kodemeister]

Awesome, thank you for checking! Note how most of points are clustered into two horizontal groups. For example, most of points on the first graph (1080p h264) lie either around ~4ms or ~7ms. This probably corresponds to two levels of your VCLK/DCLK clocks: high (1428/1255 MHz) and low (666/555 MHz).

If others confirm the same behavior, I think I'll report the issue upstream. amd drm repo seems to be the most suitable place for it.

[kodemeister]

The following FFmpeg ticket might be potentially related:

https://trac.ffmpeg.org/ticket/8377

I'm also using kmsgrab + h264_vaapi / hevc_vaapi. Need to check.

[ReenigneArcher]

Issue opened 4 years ago...

[kodemeister]

Tried to reproduce the issue but couldn't even get ffmpeg+kmsgrab+vaapi to work.

cp /usr/bin/ffmpeg .
sudo setcap cap_sys_admin+ep ffmpeg
ffmpeg -y -framerate 60 -device /dev/dri/card1 -vaapi_device /dev/dri/renderD128 -f kmsgrab -i - -f pulse -i default -c:a aac -vf 'hwmap=derive_device=vaapi,scale_vaapi=3840:2160:nv12' -c:v h264_vaapi -qp 28 video.mp4

• On KDE Plasma Wayland ffmpeg fails with error message Framebuffer pixel format 30334241 is not a known supported format. This seems to be a known bug:
  https://fftrac-bg.ffmpeg.org/ticket/10188
• On KDE Plasma X11 ffmpeg works but produces a garbled video and spams with a lot of error messages radeon_enc_encode_params UVD - DCC surfaces not supported. Again, this seems to be a known issue:
  https://invent.kde.org/plasma/krdp/-/issues/14
  Vaapi video encoding (screen recording) causes system crash ValveSoftware/SteamOS#988

There is probably a way to fine-tune VAAPI encoder but I don't have much time for now.

[cupown]

I can reproduce the same exact problem. Also using an RX 6700 XT.

Here a similar plot of avcodec_send_frame, the raw data is in the attached data.txt.

I cannot reproduce your "hacky" fix. In the middle graph, I'm playing a video with MPV, it just make avcodec_send_frame a bit slower with the same spikes.

I also have have 251 frames normal frames and 5 slow frames. It's really annoying and make sunshine hardly usable on my machine.

[cupown]

What is exact model of your GPU? Mine is Sapphire AMD Radeon RX 6700 XT NITRO+.

It's a ASUS Dual, but it's currently using Sapphire NITRO+ vbios in the vain hope of fixing problem with GPU reset and ring timeout. The problem with sunshine also existed with original vbios, I only switched to it recently, and I have this problem since I got this card.

Make sure mpv really uses VAAPI hardware decoding.

You were right, my mpv was not using my GPU, configured it years ago, and I swear it worked at that time. I don't have time to test this workaround with sunshine right now, I will try it again tomorrow.

[cupown]

I can confirm that playing a video with VAAPI decoding remove those spikes. Thanks for the tip.

I also found another source of choppiness. I use wlroot screen capture and sway headless display, and when streaming, video playback is not smooth. When streaming with wlroot a real monitor and using the hacky fix, I do not notice any choppiness. But, as soon as I try with a similar in resolution and frequency headless output, the stream is choppy. I didn't try with KMS as those headless outputs are not detected by it. I need to find another way to create a headless monitor. The cause seem to be the following issue: https://gitlab.freedesktop.org/wlroots/wlroots/-/issues/3751

Anyway, it would be nice to know more about the bug. Is it only on (all?) 6700 XT? Is it a bug in sunshine, ffmpeg, VAAPI implementation or in VCE?

[kodemeister]

I can confirm that playing a video with VAAPI decoding remove those spikes.

Awesome, glad it worked for you 😃

I need to find another way to create a headless monitor.

I'm using HDMI dummy plug for streaming. I've also set up a simple shell script to automate running mpv, turning on/off physical displays, changing the resolutions etc. If you're interested, feel free to check my dotfiles:

https://github.com/kodemeister/dotfiles/tree/master/sunshine/.config/sunshine

Anyway, it would be nice to know more about the bug. Is it only on (all?) 6700 XT? Is it a bug in sunshine, ffmpeg, VAAPI implementation or in VCE?

Unfortunately I don't have any clues about it 😕 I created this discussion in hope someone else would also run into this issue and shares their hardware details.

@gschintgen faces the same stuttering on 6650 XT so it isn't specific to 6700 XT. Probably the entire 6xxx series is affected.

I've always had stutters on different versions of Sunshine, Mesa, Linux kernel. DE/WM and display server also don't seem to matter. I had exactly the same spikes on GNOME Wayland, KDE Wayland and KDE X11. I've always used kmsgrab capture method though. Don't know much about wlr.

I'm inclined to think the issue is related to poor VAAPI implementation or VCN 3.0 rather than to Sunshine or FFmpeg. Perhaps Radeon 7xxx series with VCN 4.0 doesn't have this issue. But we need more people to confirm this.

[gschintgen]

I've always had stutters on different versions of Sunshine, Mesa, Linux kernel. DE/WM and display server also don't seem to matter.

See also: #2286 and #2333. I'm quite happy with that PR.

[cupown]

I'm using HDMI dummy plug for streaming

I find it stupid to use some hardware stuff on a libre OS, I just ended up forcing an EDID on one of my free DV port and forcing it on, that does the same thing.

I've also set up a simple shell script to automate running mpv, turning on/off physical displays, changing the resolutions etc

I also do that, also switched from using MPV to ffmpeg directly: ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -stream_loop -1 -re -i video.mp4 -f null -

See also: LizardByte/Sunshine#2286 and LizardByte/Sunshine#2333. I'm quite happy with that PR.

I was already using your PR, I extended it a bit to work with wlroot, but seem like you also did that recently.

[gschintgen]

I can reproduce this pattern, but to a less dramatic extent:

The less dramatic extent is probably related to the fact that I'm only streaming at 1080p, 60Hz. (AMD 6650XT, VA-API, mesa 23.3.6.)

IMO this should be converted from a discussion to an issue. (Even though it could turn out to be an upstream issue.)

[gschintgen]

Here is when I'm temporarily streaming my desktop at 4k@60:

Again, not as dramatic, but it does confirm that 256 frames spike.

I suppose that VA-API has a number of tunables that Sunshine does not yet expose in its UI.

[gschintgen]

FWIW the pattern also shows up in moonlight's onscreen statistics. Recently the "host processing" parts have been added to the linux side of Sunshine (latest nightly builds). I can observe that the minimum and average host processing times are mostly constant, but every few seconds the maximum increases quite a bit. But it's not even enough to pull the average higher.

[gschintgen]

Hi @kodemeister and @cupown , I poured some more hours into yesterday, until I had a breakthrough. Ffmpeg can actually pass some additional flags to AMD's VAAPI by (mis)using compression_level. This has recently been documented here: https://trac.ffmpeg.org/wiki/Hardware/VAAPI.

With most flag combination that I tried there was always that annoying spike pattern. But with the quality preset they went away! At the price of idle latency increasing from ~10ms to ~12.5ms (at 4k@60). In other words I set compression_level equal to 5 in `video.cpp. (5=4(quality) + 1(validity)).
Caveat: I was only streaming my desktop with not much going on.

This might also be of interest to @psyke83.

It might also be interesting to know how old your firmware versions are. While researching various AMD issues I found one issue that was partly related to an old firmware. Mine is rather old since I'm still on Ubuntu 22.04 and waiting for 24.04's release (kernel and mesa are rather recent, but firmware is often neglected as it turns out). It would be nice to have reproducer from the command line using only ffmpeg. I'm still suspecting that under some circumstances sunshine/ffmpeg/vaapi is emitting keyframes even if configured not to. But I did not find the part in video.cpp where I could add .is_idr() to each encoded frame. In the original report there are exactly 250 regular frames before the weirdness starts again. This corresponds exactly to the interval that AMF uses for AV1.

[kodemeister]

The same here. compression_level=5 works only on empty desktop but produces spikes in games.

I think I tried all possible values of compression_level. In summary:

• Speed and balanced profiles always have spikes, no matter what VBAQ and pre-encode bits I've used.
• Quality profile, no VBAQ, no pre-encode (compression_level=5) has no spikes on static content but does have them on dynamic content.
• Quality profile, VBAQ, no pre-encode (compression_level=21) produces irregular spikes on both static and dynamic content.
• Quality profile, w/ or w/o VBAQ, pre-encode (compression_level=13 or 29) doesn't have such drastic spikes but slow and resource-consuming.

As we can see, the spikes are less noticeable but the average latency is higher. So you might want to set compression_level to 13 or 29 but the background video hack will likely work better.

[gschintgen]

next idea: I'm seeing heavy fluctuations in VCLK (666 to 1428 MHz) and DCLK (555 to 1255 MHz) in amdgpu_top (edit: in high-frequency mode that updates faster). Not in a 4-second-rhythm though. Both clocks are related to the decoder/encoder parts of the GPU. I don't see an obvious way to disable automatic power management of it. But there is some documentation about it: https://wiki.archlinux.org/title/AMDGPU#Manual_(default)

As soon as I stop the stream, both clocks are mostly at 666 / 555 MHz.

[gschintgen]

Any additional load on the VCN seems to do the trick. Encoding works too, at least for desktop streaming (didn't test gaming):

ffmpeg -vaapi_device /dev/dri/renderD128 -f x11grab -video_size 320x240 -i :0 -vf 'hwupload,scale_vaapi=format=nv12' -c:v hevc_vaapi -f null /dev/null

Also: if I set the gpu at max frequencies using

 echo profile_peak | sudo tee /sys/class/drm/card0/device/power_dpm_force_performance_level

most of the clocks seem to be locked at their maximum (as per amdgpu_top): GFXCLK, SOCCLK, UMCCLK but the codec-related clocks VCLK and DCLK are still fluctuating.

[cupown]

I believe at this point an issue should be opened at mesa and/or AMD drm. It's not normal that behavior change when putting load with VAAPI.

Also, I wonder, does none of this happen with OBS? They have used and worked with VAAPI for streaming a lot more, but I don't see any issue or PR mentioning such issue.

[gschintgen]

I've also been wondering about OBS. But as long as the frames are only delayed but not outright dropped, it may not be an issue for OBS usecases?

I'm also wondering about AMF but I don't know how that could be tested in Linux without starting a coding project. Apparently it has a ultralowlatency flag...

[psyke83]

I'm also wondering about AMF but I don't know how that could be tested in Linux without starting a coding project. Apparently it has a ultralowlatency flag...

AFAIK, AMF is only used by the amdgpu-pro driver on Linux, but Sunshine is not coded to support enumeration of the AMF codecs on Linux, so it won't be straightforward to test (or perhaps worthwhile at all).

I know that @kodemeister does not believe this to be a bandwidth issue, but can someone report on the behaviour with the CQP rate control method clamped to the lowest quality possible? For example:

diff --git a/src/video.cpp b/src/video.cpp
index 6c1938c2..aa620c91 100644
--- a/src/video.cpp
+++ b/src/video.cpp
@@ -809,6 +809,9 @@ namespace video {
         { "low_power"s, 1 },
         { "async_depth"s, 1 },
         { "idr_interval"s, std::numeric_limits<int>::max() },
+        { "qmin"s, 50 },
+        { "qmax"s, 51 },
+        { "rc_mode"s, "cqp"s },
       },
       // SDR-specific options
       {},
@@ -828,6 +831,9 @@ namespace video {
         { "async_depth"s, 1 },
         { "sei"s, 0 },
         { "idr_interval"s, std::numeric_limits<int>::max() },
+        { "qmin"s, 50 },
+        { "qmax"s, 51 },
+        { "rc_mode"s, "cqp"s },
       },
       // SDR-specific options
       {},
@@ -847,6 +853,9 @@ namespace video {
         { "async_depth"s, 1 },
         { "sei"s, 0 },
         { "idr_interval"s, std::numeric_limits<int>::max() },
+        { "qmin"s, 50 },
+        { "qmax"s, 51 },
+        { "rc_mode"s, "cqp"s },
       },
       // SDR-specific options
       {},

It might also be worth incrementing the quality slowly (by reducing qmin/qmax) to see if CQP behaves better at equivalent quality levels to your regular bandwidth target -- but keep in mind that CQP is not suitable for enforcing a specific bandwidth limit as needed by Sunshine.

[gschintgen]

I think the options should be added to the fallback options. If I remember correctly (at this point that's a big if...) AMD will always fall through to that block due to the unsupported low_power option. @cupown mentioned that they use cqp to reduce overshoot issues. This issue is different as far as I can tell. I have netdata running in the background on my host and the bandwidth seems to be nicely controlled for. At least for hevc.

(I stopped using h264 entirely due to weird issues with multiple clients, in particular with low-powered ones. Only recently I put 2 and 2 together: it may have been the overshoot problem that overwhelmed the clients. I also had lots of MAX_SHARD messages in sunshine log. While investigating the present issue with some more debugging output I noticed that the MAX_SHARD messages popped up exactly at the times that an IDR keyframe was emitted, either on client request or due to me playing around with gop_size. h265 seems to be way less troublesome.)

[gschintgen]

Nevermind my remark about the fallback options. I added the three cqp lines to the common option (and capitalized CQP) and tested it. Bandwidth is reduced and fluctuating, as expected, and also Sunshine's log confirms that the RC mode is CPQ.

The latency spikes that this issue is about are completely unchanged. around 10ms and then every 4seconds a spike to ca. 35ms. (I eyeballed it using moonlight's onscreen stats, but it's the same pattern that has also been confirmed by frametime analysis as in the original report.)

[kodemeister]

Reported the issue upstream:

https://gitlab.freedesktop.org/drm/amd/-/issues/3336

Thanks everyone for your valuable comments and feedback! 🙇

[gschintgen]

Great work!

[awalende]

The fixes sadly don't work for me.
I run an AMD 6800XT from XFX.
KDE Plasma 6 on Wayland, everything is up to date.

On one terminal I run:
ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -stream_loop -1 -re -i movie.mkv -f null -

While on the other:
./vaapi_encode 3840 2160 60 100 output3.csv

I see spikes jumping from 2ms to 15ms in a periodic fashion. I attached my csv here.
output3.csv

Anything missing when doing the workaround?

[gschintgen]

It's this one: https://gitlab.freedesktop.org/drm/amd/-/issues/3336#note_2386629

It's a test patch (a very useful one!) that unconditionally enables a low latency mode on Amd GPUs. This mode is not yet available via mesa.

[DistantThunder]

@ReenigneArcher indeed, it's a Mesa patch that used to build and install libva from 24.0.6.

[awalende]

I can confirm that patching mesa with this small patch-file fixed it for me as well.

Initially it fixed the spikes but I still had microstutters. Turns out, that this came from using Wayland. I switched back to x11 and locked the FPS on Client and Server side to 60 FPS via DXVK_FRAME_RATE=60. Butter smooth experience so far with a handful games I had laying around.

I don't think that this patch will be included soon in upstream mesa. I simply modified the existing PKGBUILD from the arch source repository and included the patch file. Maybe I will add an package to AUR (something like mesa-lowlatency-git) or lock the custom build mesa on my system.

[DistantThunder]

@awalende I confirm the issue with micro-stutter that make the steam "jump" frames forward. But I can't tell if this is due to that "Network Jitter" stuff.

[Riebeck-one]

@awalende Hi,
I see you're using Arch linux like me. I tried to apply the patch myself, and I managed to compile everything to some extent. However, when I install the packages I compiled, especially mesa-1:24.0.6-2-x86_64.pkg.tar.zst, it crashes my desktop upon restart, and I have to download the official Mesa package again.
You mentioned that you made a custom PKGBUILD file. Would you be willing to share it? I'm not sure what I'm doing wrong.

[gschintgen]

I took some notes on how to install the patched mesa on Ubuntu (22.04):
https://gist.github.com/gschintgen/ea148962f024b1c75e3c9ad5a114afa5

[cupown]

A patch for low latency encode was merged in mesa: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30039

Enable it with AMD_DEBUG=lowlatencyenc

[kodemeister]

Awesome, thank you for noticing this! The patch will likely be included in the upcoming Mesa 24.2 release.

I can confirm that running AMD_DEBUG=lowlatencyenc sunshine works pretty well. Tested with mesa-tkg-git package from Chaotic-AUR.

[ReenigneArcher]

Could someone add a note about this environment variable to our docs?

[gschintgen]

Yes, we could also add by default to the .desktop files and systemd units and maybe also add logging code to the binary.
(edit: by "yes" I mean that I'll do it. At the very least amend the docs.)

[gschintgen]

It should also be possible to just set the variable directly in sunshine using setenv(), no?

Unfortunately I can't really test it since my AMD box will be stuck on Ubuntu 22.04 for another month.
(For various reasons, notably work, I do need to have some kind of stability and consistency, hence the slow moving LTS.)

[ReenigneArcher]

Yes, I think we probably could do that

[gschintgen]

That's great news! Thanks for the heads up.

…

On July 16, 2024 8:49:37 PM GMT+02:00, cupown ***@***.***> wrote: A patch for low latency encode was merged in mesa: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30039 Enable it with `AMD_DEBUG=lowlatencyenc` -- Reply to this email directly or view it on GitHub: https://github.com/LizardByte/Sunshine/discussions/2193#discussioncomment-10065908 You are receiving this because you were mentioned. Message ID: ***@***.***>

[kodemeister]

Closing this discussion as resolved. Sunshine now automatically enables low-latency mode on AMD GPUs (#3115). Mesa 24.2 or newer is required. The fix is already available in nightly and should be included to the next Sunshine release. Thanks everyone who helped with this annoying issue! 🙇