Skip to content

multi-scaler: corrupt images with I420 inputs #20

@ghost

Description

When configuring the multi-scaler kernel to enable support for the I420 color format, passing I420 images to the vvas_xabrscaler gstreamer element leads to corrupt images that show regular green stripes over the whole image. These images usually come from motion jpeg videos, but the behaviour can be reproduced with any I420 input:

I420_NV12
GStreamer-Pipeline to reproduce the above image (color conversion only):
gst-launch-1.0 videotestsrc num-buffers=1 ! "video/x-raw,format=I420,width=640,height=480" ! vvas_xabrscaler xclbin-location=/lib/firmware/xilinx/custom-smartcam-firmware/custom-smartcam-firmware.xclbin kernel-name=v_multi_scaler:{v_multi_scaler_1} ! "video/x-raw,format=NV12, width=640, height=480" ! jpegenc ! filesink location=I420_NV12.jpg

I420_I420_scale
Gstreamer-Pipeline with downscaling, no format change:
gst-launch-1.0 videotestsrc num-buffers=1 ! "video/x-raw,format=I420,width=640,height=480" ! vvas_xabrscaler xclbin-location=/lib/firmware/xilinx/custom-smartcam-firmware/custom-smartcam-firmware.xclbin kernel-name=v_multi_scaler:{v_multi_scaler_1} ! "video/x-raw,format=I420, width=320, height=240" ! jpegenc ! filesink location=I420_I420_scale.jpg

These stripes always have a width of 16 input pixels and occur always when using I420 (planar 4:2:0 YUV) inputs, no matter the output format/scale (as long as it is not an identity transform). NV12 inputs (semiplanar 4:2:0 YUV) as well as RGB work fine. This is independent of the output format - using input format NV12, RGB or BGR with an I420 output (or other outputs) produces the correct result:
NV12_I420_scale
gst-launch-1.0 videotestsrc num-buffers=1 ! "video/x-raw,format=NV12,width=640,height=480" ! vvas_xabrscaler xclbin-location=/lib/firmware/xilinx/custom-smartcam-firmware/custom-smartcam-firmware.xclbin kernel-name=v_multi_scaler:{v_multi_scaler_1} ! "video/x-raw,format=I420, width=320, height=240" ! jpegenc ! filesink location=NV12_I420_scale.jpg

Scaling the input image by a factor also reduces the size of the stripes by the same factor.

This was initially observed on a custom platform using Vitis 2022.2, but could be reproduced on the KV260 starter kit using the 22.1 BSP. To reproduce the bug on the kv260 using the 2022.1 toolchain:

  • build the kv260_ipMipiRx_vcu_DP platform from https://github.com/Xilinx/kria-vitis-platforms.git
  • use that platform to build the multichannel_ml example from VVAS 2.0.
    • enable I420/Y_U_V8_420 image format in v_multi_scaler_config.h: #define HAS_Y_U_V8_420 1.
    • make some adaptations to adapt the example to the smaller FPGA (reduce number of dpu cores, scaler frequency). See attached patch: adapt_to_kv260.txt
  • create a petalinux project from the Xilinx 2022.1 kv260 BSP.
  • include the bitstream and xclbin into the petalinux project by following the guide
  • include the vvas recipes from vvas-platforms/Embedded/zcu104_vcuDec_DP/petalinux/project-spec/meta-user/recipes-vvas/ into the project and add the vvas packages to the rootfs
  • enable packagegroup-petalinux-gstreamer.
  • build and flash the disk image, load the firmware with xmutil.
  • execute the gst-launch-1.0 commands above to reproduce the images.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions