[RFC] hwdec: add v4l2request hwcontext #14511
Conversation
Current out-of-tree FFmpeg V4L2 Request API hwaccel use the drm hwdevice type, however, patches being prepared for upstream FFmpeg changes this to use a new v4l2request hwdevice type. Add basic support for using the v4l2request hwdevice type. Signed-off-by: Jonas Karlman <jonas@kwiboo.se>
Kwiboo
force-pushed
the
v4l2request-hwdevice
branch
from
73dcf9b
to
c108232
Compare
|
Download the artifacts for this pull request: |
There was a problem hiding this comment.
Thanks for starting the conversation here. Obviously, we need to see how the ffmpeg side reviews go and what the final state is, but actually getting upstream support for v4l2request is a big deal.
I don't know what additional state from AVCodecHWConfig you need to make a decision on initialising the hwdec, but if you do need more, you will need to take the approach of adding more fields to hwdec_info and passing the additional metadata in add_all_hwdec_methods in video/decode/vd_lavc.c. That's currently the only place where avcodec_get_hw_config is called.
| #if HAVE_V4L2REQUEST | ||
| /* | ||
| * AVCodecHWConfig contains a combo of a pixel format and hwdevice type, | ||
| * correct type must be created here or hwaccel will fail. | ||
| * | ||
| * FIXME: Create hwdevice based on type in AVCodecHWConfig | ||
| */ | ||
| int ret = av_hwdevice_ctx_create(&p->hwctx.av_device_ref, | ||
| AV_HWDEVICE_TYPE_V4L2REQUEST, | ||
| NULL, NULL, 0); | ||
| #else |
There was a problem hiding this comment.
I know it's a draft but the main thing is that if it really ends up being a completely distinct hwdevice type, we will want a separate hwdec for it. Of course, you can refactor to share code but this #ifdef approach cannot actually be used because, as you say, it breaks hwdec_drmprime for everything else.
There was a problem hiding this comment.
Agree, and this is what really confused me when testing the v4l2reqeust hwdevice changes with mpv, this file seem more related to the imgfmt and not the hwcontext type when using a --hwdec=v4l2request parameter.
Testing this with or without the changes in this file you must still use --hwdec=v4l2request to use the new hwcontext, the only difference was if the hwaccel in ffmpeg would work or if it would fail and complain that an unexpected hwdevice type is used.
On e.g. a RPi5 board I can use following to playback a HEVC video file (after #14508).
mpv --hwdec=v4l2request --hwdec-codecs=all --gpu-hwdec-interop=drmprime-overlay --drm-draw-plane=overlay --drm-drmprime-video-plane=primary <hevc file>
There was a problem hiding this comment.
This is because historically, every hwaccel had a unique pix_fmt associated with it, and the pix_fmt actually ends up driving most of the logic, because it encapsulates where the frame data lives and how it can be manipulated. The hwdevice type only really matters for ensuring the right device is instantiated that can work with ffmpeg to produce the frames.
In your case, I guess you're adding what is basically the first new hwdevice type that doesn't use a new pix_fmt - so all the format related logic is largely unchanged. Still, you'll want to have a distinct hwdec and refactor the code to be shared. We should not fudge the relationship between hwdecs and hwaccels.
There was a problem hiding this comment.
Still, you'll want to have a distinct hwdec and refactor the code to be shared. We should not fudge the relationship between hwdecs and hwaccels.
I think this is where there may be an architectural flaw in mpv, the hwdec parameter seem to be more tied to what hwcontext gets used not what hwdec is used, the hwdec used seem more tied to the imgfmt of the hwcontext.
There is also a create_dev ops on the hwcontext that probably could/should be used at hwdec level instead of duplicated/separate logic. Ideally the hwcontext could/should be passed along as parameter/input to the hwdec. If mpv wants to treat the hwdec as the hwcontext+imgfmt combo that is also fine.
I am really hoping that a mpv developer will eventually continue working on any changes required based on the outcome of ffmpeg reviews, mainly wanted to start this discussion early so that others can start preparing for upcoming changes, and I was initially expecting that mpv already had a separation between hwdevice and hwdec/imgfmt 😄
The code here is more minimum effort to make the new hwdevice type work and limit the bad impact to when mpv is built with --enable-v4l2request.
|
Separately, does this approach have agreement from the various SoC communities? The rockchip and rpi forks have been around for ages and there are a bunch of applications and application forks that assume the |
Fully agree, last time around (2019 and 2020) there was limited interest and after RPi4 launched there was hope/indication that RPi could assist on getting something upstream, unfortunately that never happened. RFC: http://ffmpeg.org/pipermail/ffmpeg-devel/2019-April/242316.html This time around I have tried to fix feedback from the RFC and v1, include improvements along the way made by others in LibreELEC team and add similar performance improvements (using multiple requests) made in the RPi fork (that fork mainly focused on
The main thing is that the Main inconvenience in mpv seem to be that there is no real distinction between the Kodi already used the Adding a new |
The current required use of Decoding with V4L2 Request API has really nothing to do with a drm device, and was originally only used because of little to no knowledge into the inner workings of FFmpeg, the only thing that was known was that we wanted to produce frames using the Two reasons why we should not continue to use
|
|
OK. I can try and take a look in the next few weeks. There's no kernel patches required right? It's enough to have the ffmpeg patches on top of upstream? |
Great, thanks!
Correct, there should not be any need for kernel patches, uapi headers should all be upstream and last change was included in v6.5-rc1 (addition of AV1 and a new parameter in a HEVC control). The ffmpeg patches have ifdefs to adopt depending on the installed version of kernel uapi headers. The ffmpeg patches should work with current mainline kernel v4l2 stateless drivers, I am testing with I do not have any hw to test the |
|
@Kwiboo Ok. I got your patches applied and working on my Chromebook Plus (RK3399) running Arch (BTW). What video decoding formats should work? I can only get h.264 and mpeg2 to work. I also haven't seen any patches sent to ffmpeg-devel. |
Sorry for the delay, I was not able to finish getting the patches out the door before some vacation time.
FFmpeg patches is currently limited to mpeg2, h264 and hevc for now, I did not manage to finish looking over the vp8/vp9/av1 support before vacation time, they will be added in a follow up ffmpeg series (or a later patch series revision). Current kernel drivers for rk3399 should support mpeg2, h264, vp8 and vp9. To support h264 hi10 and hevc on rk3399 two kernel series for rkvdec is needed, new revision with minimal changes should hit linux-media list soon, hoping tomorrow, and hopefully they are merged in time for linux v6.12.
Also note that the latest set of ffmpeg patches now enforce a limitation that is reported by the kernel driver related to frame sizes. Kernel may need a fix to correctly report supported frame sizes, for now 16 pixel aligned width/height videos should work with existing kernel drivers. |
|
Great. I've got a PoC here of an approach that I'm happy with; I should have some patches up later this week. |
This is a follow up to a very old series from April 2019 [1] and
December 2020 [2], adding V4L2 Request API hwaccels for stateless
decoding of MPEG2, H.264 and HEVC.
These hwaccels has in one form or another been used in LibreELEC
community, nightly and release images since Dec 20th 2018.
HISTORY
The initial v4l2-request hwaccel code was mainly created as a proof of
concept at the end of 2018 by me and Jernej Skrabec. Back when Bootlin's
crowdfunding campaign for upstream Linux kernel driver for Allwinner VPU
had started to bear fruit.
At the time we had very little knowledge on how to properly interact
with any of the V4L2 APIs, and the existing V4L2 M2M code in FFmpeg
seemed very complex to start working with, so we started from scratch.
(That is the main reason why these hwaccels still does not use any of
the existing V4L2 M2M code in FFmpeg.)
The hwaccels had one major limitation, it waited on the kernel to
complete decoding before continuing with next request. Due to hwaccels
still was able to decode up to 4k 50-60fps on Allwinner and Rockchip
boards, time was never spent trying to improve this limitation.
The initial version of these hwaccels was merged as patches into
LibreELEC in April 2019 [3], with my FFmpeg tree as main source for any
update of FFmpeg v4l2-request patches included in LibreELEC.
A RFC was sent in April 2019 [1], with very little feedback and the
required kernel headers not being merged until a year later there was
never any new RFC revision submitted.
After the release of the Raspberry Pi 4 in June 2019 there was hope that
RPi folks would rework and/or improve our proof-of-concept to something
better that could be upstreamed, as history has show this did not happen.
And instead we have just ended up with a second FFmpeg v4l2-request
implementation that has only really been used for HEVC decoding on RPi.
Due to personal reasons I took a long break from doing open-source
contributions during ~2020-2023. During this time Jernej's FFmpeg tree
became the main source for FFmpeg v4l2-request patches in LibreELEC [4].
In December 2020 a v1 of this series was sent [2], at that time kernel
headers for H.264 had been merged into Linux kernel, and I cannot
remember why it never went any further.
Last update to the v4l2-request patches in LibreELEC was done by me in
November 2023 [5].
Along the way there has been noteworthy contributions including from
Boris Brezillon, Ezequiel Garcia, Alex Bee and Benjamin Gaignard.
PRESENT DAY
The version submitted in this series have seen major refactoring to the
common code, to make it more ready for upstreaming and also a rework of
how buffers and requests are handled.
The limitation of waiting for decoding to complete has been removed and
now multiple pending requests can be in-flight at the same time. This
was a requirement to better support HEVC decoding on RPi.
The older versions also incorrectly required use of a DRM hwdevice,
however V4L2 decoding has nothing to do with the DRM subsystem in Linux.
Instead in this version a new V4L2 Request API hwdevice has been added.
Thanks to this it is now possible to specify what media device to use
for decoding, in case multiple decoders exists on a system. E.g. using a
-init_hw_device v4l2request:/dev/media1 parameter.
This version only add support for MPEG2, H264 and HEVC. Support for VP8,
VP9 and AV1 is planned and be added in next revision or in a follow up
series.
HOW TO USE
To use the V4L2 Request API hwaccels you must build FFmpeg on a system
with recent Linux kernel headers, v6.0+. It also requires libdrm and
libudev to successfully build.
This can then be runtime tested on multiple Allwinner and Rockchip
devices. To runtime test this on a RPi 4 or 5 you should use latest
rpi-6.6.y kernel.
ffmpeg -hwaccel v4l2request -hwaccel_output_format drm_prime \
-i <input-path> -map 0:v -f null -
This series has been tested with cedrus driver on Allwinner H6, hantro
and rkvdec driver on Rockchip RK3399, and rpivid driver on RPi 4/5.
PRs have also been opened for Kodi and mpv to assist with the transition
from DRM to a new V4L2REQUEST hwdevice type.
- Kodi: xbmc/xbmc#25467
- mpv: mpv-player/mpv#14511
With those PRs applied it should be possible to playback video using
kodi-gbm or mpv, see the PRs above for more details.
It should also be possible to run fluster test suites with following PR:
- fluster: fluendo/fluster#179
I am expecting that there will be a new revision adding VP8, VP9 and AV1
support in a week or two.
Please get back with any type of feedback!
A copy of this series can also be found at [6].
[1] https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2019-April/242316.html
[2] https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2020-December/273579.html
[3] LibreELEC/LibreELEC.tv#3405
[4] https://github.com/LibreELEC/LibreELEC.tv/commits/master/packages/multimedia/ffmpeg/patches/v4l2-request
[5] LibreELEC/LibreELEC.tv#8356
[6] https://github.com/Kwiboo/FFmpeg/commits/v4l2request-2024-v2/
Signed-off-by: Jonas Karlman <jonas@kwiboo.se>
With all the machinery in place, we can now add the v4l2request hwdec with a different hw device type, and a different initialisation path for it. This is a basic WIP version of the change that includes the initialisation logic that Jonas put together in his PR: mpv-player#14511 I've tested this with my RK3399 based machine - have not tested the overlay as I haven't had time to pull out a suitable SBC.
|
Alright. I've put up #14639 which adds the necessary logic to handle the different device types. And then I have an initial version of the v4l2request hwdecs based on your code here but now no longer breaking the regular drmprime support. https://github.com/philipl/mpv/tree/v4l2request It still needs more work to refactor properly, but please try it out. |
|
Perfect, will test it out and get back with results. FFmpeg v4l2-request patches is now on ffmpeg-devel list at https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2024-August/332034.html |
With all the machinery in place, we can now add the v4l2request hwdec with a different hw device type, and a different initialisation path for it. This is a basic WIP version of the change that includes the initialisation logic that Jonas put together in his PR: mpv-player#14511 I've tested this with my RK3399 based machine - have not tested the overlay as I haven't had time to pull out a suitable SBC.
@philipl I gave it a go and seem to work for the most part, had to add See https://github.com/Kwiboo/mpv/commits/v4l2request-test-20240808/ for the code that I tested. I have tested using vanilla aarch64 ubuntu 24.04 on a RK3399 device (ROCK Pi 4B+ v1.73) together with However there are a few minor and/or strange issues:
On console using In |
Playback of jellyfish-3-mbps-hd-h264.mkv under Could be possible that |
With all the machinery in place, we can now add the v4l2request hwdec with a different hw device type, and a different initialisation path for it. This is a basic WIP version of the change that includes the initialisation logic that Jonas put together in his PR: mpv-player#14511 I've tested this with my RK3399 based machine - have not tested the overlay as I haven't had time to pull out a suitable SBC.
|
Did you try with |
Thanks, that worked much better! We have noticed that the new ffmpeg v4l2-request code help trigger a decoding issue only seen using |
Great. We strongly recommend that people use |
|
I don't have hardware to test, but I don't think gpu{-next} should be presenting frames jumping back and forth even if that is the case. I wonder if it's similar to #13811 |
I cannot reproduce with your samples, but I've seen that with interlaced dvb-t samples on amd. Sometimes frames start jumping back and forth, other times it just stutter a lot, and video decode engine is at 100%. It happens only with limited number of samples. This is good example, https://github.com/GPUOpen-LibrariesAndSDKs/AMF/files/11082371/Sample2_.2750.zip if you start playback from beginning. Sometimes seeking recovers it. I'm not sure if this is comparable to your case. I'm sure at one point (years ago) those samples were working fine, so it got regressed in AMD driver, but question is if ffmpeg is not doing something unexpected. It happens also with LAV Filters on Windows, so it is not mpv related. But since we are discussing completely unrelated platforms, it's probably different issues... See: |
With all the machinery in place, we can now add the v4l2request hwdecs with a different hw device type, and a different initialisation path. This applies to both the drmprime and drmprime_overlay hwdecs. At the moment, the device initialisation is done in the bare minimum way, but it can be extended to take a device path (for example) if that makes sense as we better understand what meaningful configuration will be. This change is based on the code proposed by Jonas Karlman <jonas@kwiboo.se> in mpv-player#14511
|
@llyyr @kasper93 the issue I noticed without use of Also I do not think that this wayland rendering issue is really an issue worth pursing, with use of |
|
This RFC is now obsolete, see #14690 for a proper implementation that can work with both |
Current out-of-tree FFmpeg V4L2 Request API hwaccel patches abuse the drm hwdevice type.
Patches being prepared for upstream FFmpeg changes this to use a new dedicated v4l2request hwdevice type.
This PR add basic support for using the new v4l2request hwdevice type, however, it cripples old use of
AV_HWDEVICE_TYPE_DRM, and possible break use with e.g.rkmppdecoders.Please assist on how the the hwdevice created based on the picked
AVCodecHWConfigcan be re-used instead of recreating a new hwdevice.Ideally
mpvshould behave similar to e.g.ffmpeg -init_hw_device v4l2request, or implicitly create the hwdevice based on theAVCodecHWConfig.device_type.With the updated patches one would instead of
drmusev4l2requestas the-hwaccel, e.g.:Created as a draft PR because work-in-progress FFmpeg patches are still being worked on.