fix: prevent choppy qsv transcodes with vfr content

[aclerici38]

Description

Fixes #8657
Possibly #21484

Disclaimer: I am far from an expert in ffmpeg, hw stacks, etc. This is just an issue that's been bothering me and I took a stab at fixing it. let me know if it's completely off base!

HW Transcoding (QSV specifically for me) on some videos (newer iPhone vids that use variable fps are particularly bad) produces an extremely choppy video across all clients I tried (firefox/chrome, iOS app, immich-gallery on tvOS). For some reason the hw encode/decode paths on these videos produce outputs with out-of-order or nearly identical timestamps. Swapping the -fps_mode arg from passthrough to cfr (constant frame rate) works around this by forcing ffmpeg to interpolate or duplicate frames as needed to output a vid at the specified fps. The trade-off is potential for frame duplication/dropping to enforce CFR.
NOTE: it's possible to just set CFR and ffmpeg will match the original's framerate, but for videos with variable frame rates this can cause the transcode to be much larger than it needs to be. For example, in the video below the frame rate is spec'd at 120 but the average frame rate is about 29 (3597000/123509). It makes more sense to me to use the average framerate for a lossy transcode.

Alternate solutions: In my testing disabling B-frames also helped the choppiness. I feel setting cfr is a more complete fix, however.

This example vid was filmed on an iphone 13. I cut it to 10 seconds and removed the location metadata. It's too big to upload straight to github unfortunately.
original.mov
Here are the timestamps:

ffprobe -v error -select_streams v:0 -show_entries frame=pts_time -of csv=p=0 original.mov | head -10
0.000000,,,,
0.041667,,,,
0.083333,,,,
0.125000,,,,
0.166667,,,,
0.208333,,,,
0.241667,,,,
0.275000,,,,
0.308333,,,,
0.341667,,,,

Before, immich would invoke this ffmpeg command to transcode:

ffmpeg -n 10 /usr/bin/ffmpeg -hwaccel qsv -hwaccel_output_format qsv -async_depth 4 -noautorotate -qsv_device /dev/dri/renderD128 -threads 1 -i /data/library/anthony/2025/12/IMG_3104.mov -y -c:v hevc_qsv -c:a copy -movflags faststart -fps_mode passthrough -map 0:0 -map_metadata -1 -map 0:1 -bf 7 -refs 5 -g 256 -tag:v hvc1 -v verbose -vf scale_qsv=-1:1440:async_depth=4:mode=hq,hwmap=derive_device=opencl,tonemap_opencl=desat=0:format=nv12:matrix=bt709:primaries=bt709:transfer=bt709:range=pc:tonemap=reinhard:tonemap_mode=lum:peak=100,hwmap=derive_device=qsv:reverse=1,format=qsv -preset 4 -global_quality:v 28 /data/encoded-video/d5e4be30-b38f-4c6b-a7b1-0721a11992c5/e9/8a/e98a9912-82db-4bd0-952b-e9f12f6d4214.mp4

Which produced this choppy output
https://github.com/user-attachments/assets/8c0494ff-7bf6-4fd9-99a1-48019d958f5e
And has some frames out of order along with 5 crammed into just 0.0003s

ffprobe -v error -select_streams v:0 -show_entries frame=pts_time -of csv=p=0 before-transcode.mp4 | head -10
0.000000,
0.225195,
0.225130,
0.225260,
0.225065,
0.225391,
0.241667,
0.275000,
0.308333,
0.491862,

After the change to use cfr the ffmpeg command becomes

ffmpeg -n 10 /usr/bin/ffmpeg -hwaccel qsv -hwaccel_output_format qsv -async_depth 4 -noautorotate -qsv_device /dev/dri/renderD128 -threads 1 -i /data/library/anthony/2025/12/IMG_3104.mov -y -c:v hevc_qsv -c:a copy -movflags faststart -fps_mode cfr -r 3597000/123509 -map 0:0 -map_metadata -1 -map 0:1 -bf 7 -refs 5 -g 256 -tag:v hvc1 -v verbose -vf scale_qsv=-1:1440:async_depth=4:mode=hq,hwmap=derive_device=opencl,tonemap_opencl=desat=0:format=nv12:matrix=bt709:primaries=bt709:transfer=bt709:range=pc:tonemap=reinhard:tonemap_mode=lum:peak=100,hwmap=derive_device=qsv:reverse=1,format=qsv -preset 4 -global_quality:v 28 /data/encoded-video/d5e4be30-b38f-4c6b-a7b1-0721a11992c5/e9/8a/e98a9912-82db-4bd0-952b-e9f12f6d4214.mp4

The encoded vid is no longer choppy
https://github.com/user-attachments/assets/82d5c6f4-658c-49d5-bbcf-c39586483be2
And the timestamps are normal-looking

ffprobe -v error -select_streams v:0 -show_entries frame=pts_time -of csv=p=0 cfr.mp4 | head -10
0.000000,
0.034337,
0.068673,
0.103010,
0.137347,
0.171683,
0.206020,
0.240357,
0.274693,
0.309030,

How Has This Been Tested?

• Pushed an image to docker.io/ant385525/immich-server:cfr-fix and retranscoded my library, verified all videos play well
• Reviewed ffprobe outputs to see corrected timestamps

Screenshots (if appropriate)

Checklist:

• I have performed a self-review of my own code
• I have made corresponding changes to the documentation if applicable
• I have no unrelated changes in the PR.
• I have confirmed that any new dependencies are strictly necessary.
• I have written tests for new code (if applicable)
• I have followed naming conventions/patterns in the surrounding code
• All code in src/services/ uses repositories implementations for database calls, filesystem operations, etc.
• All code in src/repositories/ is pretty basic/simple and does not have any immich specific logic (that belongs in src/services/)

Please describe to which degree, if any, an LLM was used in creating this pull request.

Used claude code to debug the video files and parse ffprobe/ffmpeg output. Impl logic is my own

[immich-push-o-matic]

Label error. Requires exactly 1 of: changelog:.*. Found: 🗄️server. A maintainer will add the required label.

[mertalev]

Hmm, forcing the transcode to a constant frame rate doesn't seem ideal. Does vfr work?

[aclerici38]

I tried that first, it didn't unfortunately
https://github.com/user-attachments/assets/79a1bce9-414f-4022-8ed5-5763482092d1

ffprobe -v error -select_streams v:0 -show_entries frame=pts_time -of csv=p=0 vfr-fail.mp4 | head -10
0.000000,
0.225195,
0.225130,
0.225260,
0.225065,
0.225391,
0.241667,
0.275000,
0.308333,
0.491862,

ffmpeg -n 10 /usr/bin/ffmpeg -hwaccel qsv -hwaccel_output_format qsv -async_depth 4 -noautorotate -qsv_device /dev/dri/renderD128 -threads 1 -i /data/library/anthony/2025/12/IMG_3104.mov -y -c:v hevc_qsv -c:a copy -movflags faststart -fps_mode vfr -map 0:0 -map_metadata -1 -map 0:1 -bf 7 -refs 5 -g 256 -tag:v hvc1 -v verbose -vf scale_qsv=-1:1440:async_depth=4:mode=hq,hwmap=derive_device=opencl,tonemap_opencl=desat=0:format=nv12:matrix=bt709:primaries=bt709:transfer=bt709:range=pc:tonemap=reinhard:tonemap_mode=lum:peak=100,hwmap=derive_device=qsv:reverse=1,format=qsv -preset 4 -global_quality:v 28 /data/encoded-video/d5e4be30-b38f-4c6b-a7b1-0721a11992c5/e9/8a/e98a9912-82db-4bd0-952b-e9f12f6d4214.mp4

[aclerici38]

Disabling bframes also helped before, but in my mind the cfr change was better as I didn't really understand why disabling bframes helped and if the fix was specific to qsv. Maybe that's not the case

[aclerici38]

I looked into this a bit more and disabling bframes isn't a fix. It definitely makes things a lot better but there's still issues playing some transcodes. I'll keep looking into this, from what I've read online people have pretty strong opinions that cfr isn't a great thing to use I just haven't found a better solution

(This is immich v2.5.5 with bframes disabled)
In the transcoded video the 3rd and 4th frames shown are separated by 0.065ms and the 4th and 5th frames are 16.7ms apart.

ffprobe -v error -select_streams v:0 -show_entries frame=pts_time -of csv=p=0 playback.mp4 | head -1038 | tail -8
8.583333,
8.591667,
8.608333,
8.608398,
8.616667,
8.633333,
8.641667,
8.650000,

Here are the PTS timestamps from the original

ffprobe -v error -select_streams v:0 -show_entries frame=pts_time -of csv=p=0 "IMG_0073 (1).mov" | head -1038 | tail -8
8.587083,,,,
8.595417,,,,
8.604167,,,,
8.612083,,,,
8.620417,,,,
8.629167,,,,
8.637500,,,,
8.645833,,,,

[mertalev]

Does adding -fflags +igndts in the input options work? If it does, it's a much more efficient fix.

[aclerici38]

Aha, I think fps_mode=vfr and disabling bframes is working! I'm running it through my library to test, I'll check back tomorrow. Would that be an acceptable fix? I'm thinking to globally setfps_mode=vfr and disable b-frames on qsv. I'm not 100% if other hw accel stacks have this same issue after reading through the linked issues again.

Unfortunately +igndts did not change anything. I tested with -fflags +igndts, -fflags +genpts, and -fflags +igndts+genpts. The original file has valid timestamps so I'm pretty sure it's something qsv/ffmpeg related and not having to do with the demux bit.

ffprobe -v error -select_streams v:0 -show_entries frame=pts_time -of csv=p=0 /work/v2_bf7_igndts.mp4 | head -20 | tail -10
0.491797,
0.491927,
0.491732,
0.492057,
0.508333,
0.541667,
0.575000,
0.758529,
0.758464,
0.758594,

ffprobe -v error -select_streams v:0 -show_entries frame=pts_time -of csv=p=0 /work/v2_bf7_genpts.mp4 | head -20 | tail -10
0.491797,
0.491927,
0.491732,
0.492057,
0.508333,
0.541667,
0.575000,
0.758529,
0.758464,
0.758594,

ffprobe -v error -select_streams v:0 -show_entries frame=pts_time -of csv=p=0 /work/v2_bf7_both.mp4 | head -20 | tail -10
0.491797,
0.491927,
0.491732,
0.492057,
0.508333,
0.541667,
0.575000,
0.758529,
0.758464,
0.758594,

[mertalev]

Setting fps_mode=vfr is probably fine. B-frames can be disabled by default for QSV (with a comment explaining why), but you should be able to set them as normal.