-
Notifications
You must be signed in to change notification settings - Fork 23
Description
When configuring the multi-scaler kernel to enable support for the I420 color format, passing I420 images to the vvas_xabrscaler gstreamer element leads to corrupt images that show regular green stripes over the whole image. These images usually come from motion jpeg videos, but the behaviour can be reproduced with any I420 input:

GStreamer-Pipeline to reproduce the above image (color conversion only):
gst-launch-1.0 videotestsrc num-buffers=1 ! "video/x-raw,format=I420,width=640,height=480" ! vvas_xabrscaler xclbin-location=/lib/firmware/xilinx/custom-smartcam-firmware/custom-smartcam-firmware.xclbin kernel-name=v_multi_scaler:{v_multi_scaler_1} ! "video/x-raw,format=NV12, width=640, height=480" ! jpegenc ! filesink location=I420_NV12.jpg

Gstreamer-Pipeline with downscaling, no format change:
gst-launch-1.0 videotestsrc num-buffers=1 ! "video/x-raw,format=I420,width=640,height=480" ! vvas_xabrscaler xclbin-location=/lib/firmware/xilinx/custom-smartcam-firmware/custom-smartcam-firmware.xclbin kernel-name=v_multi_scaler:{v_multi_scaler_1} ! "video/x-raw,format=I420, width=320, height=240" ! jpegenc ! filesink location=I420_I420_scale.jpg
These stripes always have a width of 16 input pixels and occur always when using I420 (planar 4:2:0 YUV) inputs, no matter the output format/scale (as long as it is not an identity transform). NV12 inputs (semiplanar 4:2:0 YUV) as well as RGB work fine. This is independent of the output format - using input format NV12, RGB or BGR with an I420 output (or other outputs) produces the correct result:

gst-launch-1.0 videotestsrc num-buffers=1 ! "video/x-raw,format=NV12,width=640,height=480" ! vvas_xabrscaler xclbin-location=/lib/firmware/xilinx/custom-smartcam-firmware/custom-smartcam-firmware.xclbin kernel-name=v_multi_scaler:{v_multi_scaler_1} ! "video/x-raw,format=I420, width=320, height=240" ! jpegenc ! filesink location=NV12_I420_scale.jpg
Scaling the input image by a factor also reduces the size of the stripes by the same factor.
This was initially observed on a custom platform using Vitis 2022.2, but could be reproduced on the KV260 starter kit using the 22.1 BSP. To reproduce the bug on the kv260 using the 2022.1 toolchain:
- build the kv260_ipMipiRx_vcu_DP platform from https://github.com/Xilinx/kria-vitis-platforms.git
- use that platform to build the multichannel_ml example from VVAS 2.0.
- enable I420/Y_U_V8_420 image format in
v_multi_scaler_config.h:#define HAS_Y_U_V8_420 1. - make some adaptations to adapt the example to the smaller FPGA (reduce number of dpu cores, scaler frequency). See attached patch: adapt_to_kv260.txt
- enable I420/Y_U_V8_420 image format in
- create a petalinux project from the Xilinx 2022.1 kv260 BSP.
- include the bitstream and xclbin into the petalinux project by following the guide
- include the vvas recipes from
vvas-platforms/Embedded/zcu104_vcuDec_DP/petalinux/project-spec/meta-user/recipes-vvas/into the project and add the vvas packages to the rootfs - enable
packagegroup-petalinux-gstreamer. - build and flash the disk image, load the firmware with xmutil.
- execute the
gst-launch-1.0commands above to reproduce the images.