Work around warping on nvidia with Atomic KMS + GBM KMS

[tarek-y-ismail]

Closes #4035

What's new?

• Adds a fallback path in gbm_bo_with_modifiers_or_linear to account for nvidia non-compliance
• Adds a codepath in DMABufEGLProvider to copy buffers via the CPU
• Adds a quirk to detect to determine which codepath to use for buffer transfers, and passes that down to DMABufEGLProvider
• Any transfer involving nvidia goes through the GPU for now until we get to the bottom of the bug.

How to test

On a machine with an intel iGPU and an nvidia dGPU, with one output connected to each:

1. Launch mir as follows:

miral-app --platform-display-libs=mir:atomic-kms --platform-rendering-libs=mir:gbm-kms --display-config=sidebyside --driver-quirks=gbm-buffer-transfer-strategy:nvidia:nvidia:dma

2. Launch one of firefox, ghostty, gtk4-demo as follows

__EGL_VENDOR_LIBRARY_FILENAMES=/usr/share/glvnd/egl_vendor.d/50_mesa.json  DRI_PRIME=1 <command>

3. Drag the window onto the screen connected to the nvidia card
4. If the app still renders correctly, resize the width it using the mouse
5. Observe as the window warps in a sawtooth pattern as you change the width
6. Repeat steps 1-4, but remove the last flag or change it to --driver-quirks=gbm-buffer-transfer-strategy:nvidia:nvidia:cpu
7. Observe as the window renders correctly.

Checklist

• Adequate documentation added (How should we document this?)

[Copilot]

Pull request overview

This PR implements a workaround for rendering issues on NVIDIA GPUs when using Atomic KMS with GBM KMS. The primary issue was a "warping sawtooth pattern" that occurred when resizing windows on NVIDIA-connected displays in a hybrid GPU setup.

Changes:

• Added a BufferTransferStrategy enum to support CPU-based buffer transfers as an alternative to DMA transfers
• Modified buffer allocation to handle NVIDIA's non-compliant GBM behavior when allocating with modifiers
• Added a driver quirk detection system to automatically apply the CPU transfer strategy for NVIDIA GPUs

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/platforms/renderer-generic-egl/rendering_platform.cpp	Added default CPU buffer transfer strategy for generic EGL renderer
src/platforms/renderer-generic-egl/buffer_allocator.cpp	Updated texture allocation to pass full Buffer instead of NativeBufferBase
src/platforms/gbm-kms/server/kms/quirks.h	Added BufferTransferStrategy enum and quirk tracking
src/platforms/gbm-kms/server/kms/quirks.cpp	Implemented quirk parsing and application for NVIDIA buffer transfer strategy
src/platforms/gbm-kms/server/kms/platform.cpp	Added fallback allocation paths and integrated quirks into buffer provider creation
src/platforms/gbm-kms/server/buffer_allocator.cpp	Updated texture allocation to pass full Buffer instead of NativeBufferBase
src/platform/graphics/linux_dmabuf.cpp	Implemented CPU transfer path, refactored DMA transfer logic, and added strategy selection
include/platform/mir/graphics/linux_dmabuf.h	Added BufferTransferStrategy enum and updated DMABufEGLProvider interface

[Saviq]

How should we document this?

These seem appropriate?

• https://canonical-mir.readthedocs-hosted.com/latest/explanation/platform-support/
• https://canonical-mir.readthedocs-hosted.com/latest/explanation/driver-quirks/

[RAOF]

The quirk is pragmatic, but it would ideally take into account both GPUs - as far as I can tell, this bug only manifests when importing from Intel to NVIDIA (it imports fine from AMD to NVIDIA on my system).

[tarek-y-ismail]

The more I work on this problem, the weirder it gets:

• Apparently the warping occurs when we import from nvidia to nvidia for some reason.
  Forcing that to use the CPU path via --driver-quirks=gbm-buffer-transfer-strategy:nvidia-drm:nvidia-drm:cpu fixes the warping. That points to the following code path:

  mir/src/platform/graphics/linux_dmabuf.cpp

  Lines 1796 to 1800 in 1a04fce

  	if (dmabuf_tex->on_same_egl_display(dpy))
  	{
  	auto tex = dmabuf_tex->as_texture();
  	return std::shared_ptr<gl::Texture>(std::move(dmabuf_tex), tex);
  	}

• Logs show the source and importing sides are flipped. Isn't the buffer provider the source?
  Rendering a firefox window on the iGPU, and dragging it to the iGPU's output causes transfers from the dGPU to the iGPU??
• Rendering a firefox window on the dGPU and dragging it from the dGPU output to the iGPU output causes the iGPU output to stop displaying frames altogether, even if firefox is shutdown and relaunched on the iGPU
  The splash screen seems to trigger this as well. This might be related to the next point too.
• Still have freezes during shutdown. Dunno from where exactly but last time I checked it was the iGPU rendering thread getting stuck in eglSwap. Won't snoop after it right now until the main bug is fixed

EDIT: Attaching a log. Opened qterminal, then firefox rendering on the iGPU and displaying on the dGPU. Then dragged the window to the iGPU's output and back. Closed firefox. Relaunched it without specifying GPUs (which usually renders on the dGPU). Repeated the roundtrip to the iGPU output and back. Shut down.

20260119_141900-mir-atomic-kms-hybrid-log.txt

[tarek-y-ismail]

Looks like DMA transfers from nvidia to intel in the case where firefox is rendering on nvidia and should display on intel cause the intel compositing thread to completely freeze somewhere in the mesa stack in dri_flush. Again, going through the CPU alleviates this.

Edit: Forgot to add, checking dmesg logs shows no errors being thrown, mesa debugging symbols don't help too. I think I might have to build from source to investigate this further. But I think I've spent enough time on this issue as is.

[tarek-y-ismail]

Well, this is awkward. I just tried running kgx rendering on the nvidia GPU and it still suffers from warping. Its colors are broken compared to KDE as well. Pluma and gedit work fine though.

[github-actions]

TICS Quality Gate

❌ Failed

mir

Coding Standards: ❌ 9 Blocking Issues

❌ Condition “No new Coding Standard Violations for level 1, 2, 3 with respect to Previous analysis” failed 2 times.

File	Issues
	🪲 Total	❌ Blocking
src/platform/graphics/linux_dmabuf.cpp	47	+7
src/platforms/gbm-kms/server/kms/quirks.cpp	3	+2

See the results in the TICS Viewer

The following files have been checked for this project

• include/platform/mir/graphics/linux_dmabuf.h
• include/platform/mir/graphics/quirk_common.h
• src/platform/graphics/linux_dmabuf.cpp
• src/platform/graphics/quirk_common.cpp
• src/platforms/atomic-kms/server/kms/quirks.cpp
• src/platforms/gbm-kms/server/buffer_allocator.cpp
• src/platforms/gbm-kms/server/kms/platform.cpp
• src/platforms/gbm-kms/server/kms/quirks.cpp
• src/platforms/gbm-kms/server/kms/quirks.h
• src/platforms/renderer-generic-egl/buffer_allocator.cpp
• src/platforms/renderer-generic-egl/rendering_platform.cpp

TICS / TICS / Run TICS analysis