Bug: SIGSEGV in PipeWire MemFd buffer handling via portal ScreenCast

[lamco-office]

Summary

Server crashes with SIGSEGV when receiving MemFd-type buffers through the PipeWire portal ScreenCast capture path. The crash occurs because PipeWire's MAP_BUFFERS auto-mapping can produce stale pointers for MemFd buffers received via portal FD connections.

Affected versions

v1.4.2 and earlier (when using portal ScreenCast capture path)

Who is affected

Users on compositors where the portal backend provides MemFd buffers rather than DmaBuf. This is most commonly observed with XDPH (Hyprland's XDG Desktop Portal backend) on PipeWire 1.6.x, since XDPH defaults to MemFd unless the client advertises DmaBuf modifier support.

Affected configuration:

• Capture path: portal ScreenCast → PipeWire (not wlr-screencopy)
• Buffer type: MemFd (PipeWire SPA_DATA_MemFd, type 2)
• Trigger: stream lifecycle event (disconnect, renegotiation) that invalidates auto-mapped pointers

NOT affected:

• Users on the wlr-screencopy capture path (bypasses PipeWire entirely)
• Users receiving DmaBuf buffers (type 3) — these already use manual mmap
• Users receiving MemPtr buffers (type 1) — direct pointer, no mapping involved

Root cause

PipeWire's MAP_BUFFERS stream flag auto-mmaps MemFd buffers at stream setup time. When the portal FD connection experiences lifecycle events (disconnect, renegotiation), the internal clear_buffers() calls munmap() but the buffer's data pointer field is not nulled. The libspa data() accessor only checks for null — it returns this stale, unmapped pointer as valid. Attempting to memcpy from it triggers SIGSEGV.

Status

Fixed in lamco-pipewire 0.3.2. Pending next server release.

[lamco-office]

Fix details (lamco-pipewire 0.3.2)

The MemFd buffer handler in the PipeWire process callback previously attempted to use data.data() (the auto-mapped pointer from MAP_BUFFERS) as the primary path, falling back to manual mmap_fd_buffer() only when the pointer was null. The problem is that stale pointers are non-null — they pass the null check but point to unmapped memory.

The fix: MemFd buffers now always use manual mmap_fd_buffer() instead of relying on the auto-mapped data.data() pointer. The manual mmap maps the file descriptor, copies the frame data, and unmaps within the same callback invocation. No pointer survives beyond the callback scope, so there is no window for staleness.

This is a consumer-side fix. The behavior that produces stale pointers exists in PipeWire's buffer management and affects any consumer using MAP_BUFFERS with MemFd buffers received through portal FD connections. The DmaBuf handler was already using manual mmap and was not affected.

Configurations affected in detail

All portal backends use identical buffer type selection logic: if the PipeWire format negotiation includes SPA_FORMAT_VIDEO_modifier, the backend provides DmaBuf; otherwise, it provides MemFd. The crash requires:

1. Portal ScreenCast capture path active — the server must be using portal-based capture, not wlr-screencopy
2. Portal backend providing MemFd buffers — XDPH defaults to MemFd; other backends (Mutter, KWin, xdpw) also fall back to MemFd when DmaBuf modifiers are not negotiated
3. Stream lifecycle event — something must invalidate the auto-mapped pointer (stream disconnect, format renegotiation, buffer reallocation)

In practice, Hyprland/XDPH users are the most exposed because XDPH defaults to MemFd buffers. Users on GNOME or KDE with DmaBuf-capable hardware typically receive DmaBuf buffers and are unaffected.

Release status

The fix is published in lamco-pipewire 0.3.2 on crates.io. It will ship in the next lamco-rdp-server release (v1.4.3 or v1.5.0).