Getting distorted image after saving when using NV12 format.

[PriyanshuSudhakar26]

gst-launch-1.0 filesrc location='video2.mp4' ! parsebin ! vah264dec ! "video/x-raw(memory:VAMemory)" ! gvapython module=full.py function=process_frame ! vapostproc ! "video/x-raw(memory:VAMemory), width=320, height=320" ! queue ! gvadetect model=yolov3.xml model-proc=yolov3.json threshold=0.8 device=GPU ! fakesink async=true

This is the pipeline I am using and I want to save each frame using gvapython , but I am getting distorted images after I save it.

here is the python code

import cv2
import numpy as np
import os
from gstgva import VideoFrame

frame_counter = 0

def process_frame(frame):
    global frame_counter

    # Retrieve video info
    video_info = frame.video_info()
    width = video_info.width
    height = video_info.height

    frame_counter += 1

    with frame.data() as frame_data:
        bgr = cv2.cvtColor(frame_data, cv2.COLOR_YUV2BGR_NV12)   
        os.makedirs("sent", exist_ok=True)
        cv2.imwrite(f"sent/frame_{frame_counter}.jpg", frame_data)

    return frame

[brmarkus]

You could check with a gst pipeline graph which format actually was negotiated and provided to each plugin. Or check the format finfo within video_info.

What do you wanted to do with bgr = cv2.cvtColor(frame_data, cv2.COLOR_YUV2BGR_NV12)?

[PriyanshuSudhakar26]

@brmarkus

enum GST_VIDEO_FORMAT_NV12 of type GstVideo.VideoFormat
the format is NV12 only

bgr = cv2.cvtColor(frame_data, cv2.COLOR_YUV2BGR_NV12)

here basically I am trying to first convert to bgr and then save it as a jpg

[brmarkus]

You are storing frame_data, but not bgr.

[PriyanshuSudhakar26]

@brmarkus
yeah , sorry I have tried saving bgr also , did not work.

[brmarkus]

Can you get more details about the received format?
Can you just try e.g. bgr = cv2.cvtColor(frame_data, cv2.COLOR_YUV2BGR) and then cv2.imwrite()?

[PriyanshuSudhakar26]

@brmarkus
bgr = cv2.cvtColor(frame_data, cv2.COLOR_YUV2BGR) this throws an error

ERROR: pipeline doesn't want to preroll.
<enum GST_VIDEO_FORMAT_NV12 of type GstVideo.VideoFormat>
ERROR: from element /GstPipeline:pipeline0/GstGvaPython:gvapython0: Traceback (most recent call last):
  File "/work/full.py", line 19, in process_frame
    bgr = cv2.cvtColor(frame_data, cv2.COLOR_YUV2BGR)
cv2.error: OpenCV(4.10.0) /io/opencv/modules/imgproc/src/color.simd_helpers.hpp:92: error: (-2:Unspecified error) in function 'cv::impl::{anonymous}::CvtHelper<VScn, VDcn, VDepth, sizePolicy>::CvtHelper(cv::InputArray, cv::OutputArray, int) [with VScn = cv::impl::{anonymous}::Set<3>; VDcn = cv::impl::{anonymous}::Set<3, 4>; VDepth = cv::impl::{anonymous}::Set<0, 2, 5>; cv::impl::{anonymous}::SizePolicy sizePolicy = cv::impl::<unnamed>::NONE; cv::InputArray = const cv::_InputArray&; cv::OutputArray = const cv::_OutputArray&]'
> Invalid number of channels in input image:
>     'VScn::contains(scn)'
> where
>     'scn' is 1

[brmarkus]

There is a similar discussion here, e.g. "#197 (comment)".

You need to find out more about the exact format, first.

E.g. write the raw frame data and then use a raw-image-viewer to get to know the exact format, like