optimized I420 to RGB in mrayGstreamerUnity plugin?

[pinjiang]

Hi, mrayy
I would like to raise a question. Why in the C# we append "videoconvert ! video/x-raw,format=I420 " to gstreamer pipeline strings, and later converted from I420 to RBG in the C++ plugin.

We found the efficiency of the I420 to RBG conversion in C++ is very low. We'd like to convert to RBG directly with gstreamer plugin since we are playing high resolution video which means a lot of pixels.

Is there any reason doing that?

[NoTuxNoBux]

IIRC the reason is that the ImageInfo class used to convert the image pixel data to a format suitable for use in Unity only supports I420 right now. If the format is anything else, the C++ plugin won't know how to crop (or otherwise handle) the data.

Operating on RBG or other formats is different from I420 and requires a different approach because of the layout. There is no technical limitation to supporting RBG or other formats here (and providing a fast path if you don't need cropping), so the "reason" is probably just that @mrayy did not need anything but I420 at the time he built this plugin (I420 is very commonly used as camera image format, so also in streaming video from cameras).

To summarize: if you need a different format than I420, you'll probably need to add support for it in the C++ plugin, after which you can remove the I420 conversion from the pipeline string (I did something similar to support NV12 for my company to avoid conversions and optimize streaming from a camera that natively doesn't support I420).

[mrayy]

I420 is much lighter format for passing the data between gstreamer to unity. Using this format would reduce the rendering latency significantly compared to RGB format.
In the plugin I am not converting from I420 to RGB using C++ actually, but rather it gets handled directly via the shader, which is significantly faster than doing the conversion on the CPU.
If you want to use RGB format, you can simply modify the pipeline:
video/x-raw, format=RGB
And remove the shader from the post processing shader attribute.

[ahmadreza-hadidi]

hey @mrayy why you convert height of I420 images to "height/1.5"? i found that the image quality decreased by this.