Transmit OSD video to UDP via gstream instead of monitor

[ylankovs]

Hey, firstly this is a very awesome project and together with wifibroadcast it creates a strong tool for FPV.

I'm using NVIDIA Jetson TX2 (8GB) with JetPack version 7.1 (# R32 (release), REVISION: 7.1, GCID: 29818004, BOARD: t186ref, EABI: aarch64, DATE: Sat Feb 19 17:07:00 UTC 2022). I'm using an onboard CSI camera from the development kit and Pixhawk autopilot.

I have a bunch of questions,

1. I run the following command to capture video from CSI and direct to h264 localhost port,
   gst-launch-1.0 nvarguscamerasrc ! nvvidconv ! omxh264enc ! 'video/x-h264, framerate=(fraction)30/1, stream-format=(string)byte-stream' ! h264parse ! rtph264pay mtu=1400 config-interval=1 pt=96 ! udpsink host=127.0.0.1 port=5600

Then, I'm using mavproxy route to redirect telemetry from autopilot usb to port,
mavproxy.py --baudrate 115200 --out 127.0.0.1:14551 --state-basedir=/tmp

Then, I've compiled osd project and just simply running ./osd command to get the video with telemetry OSD as the output. The QUESTION is as follows, when the video just appears I can see the picture fine, but within 1-2 seconds it goes gray with some artifacts like on the television screen. I keep osd running and cover the camera lens with my finger and then quickly remove my finger and I can see the clear picture again for 1-2 seconds and then it goes grey again, why does this happen???
2. I require to do the following,

• capture video from camera into jetson
• put OSD on the video in jetson
• direct video with OSD to UDP stream
• capture video from jetson on a client device

I will also require the telemetry to be sent in both directions via the same UDP ethernet channel, however, your projects are based on wifi, and I'm trying to understand how I can send my traffic via UDP ethernet, instead of wifi broadcast? It'd be good to start with understanding on how to run osd project and instead of seeing the video on the jetson screen it's sent to a udp source?

Thanks for your time, I would really appreciate to have some pointers on the questions above as I'm fairly new to the topic and trying to wrap my brain around it.

[svpcom]

1. Reduce video bitrate to 4mbit/s and check link state via wfb-cli gs on the ground side.
2. You can avoid use of mavproxy and put in /etc/wifibroadcast.cfg on the drone side:

   [drone_mavlink]
   peer = 'serial:ttyUSB0:115200'
   

[ylankovs]

1. I have tried using bitrate=4000000 (which is the default value) for omxh264enc and bitrate=4000 for x264enc for different instances of gst-launch-1.0 commands issued. However, neither run had any success with removing the grey.

I have gone back to the basic example in the readme with OSD displayed over the black screen,

gst-launch-1.0 videotestsrc pattern=black ! videoconvert !
video/x-raw,format=NV12,framerate=30/1,width=1280,height=720 !
x264enc bitrate=4000 tune=zerolatency ! rtph264pay config-interval=1 !
udpsink host=127.0.0.1 port=5600

However, running this example and then executing ./osd does not show any video at all. I have the messages appearing, i.e.

OSD_PORT=14551
RTP_PORT=5600
VIDEO_CODEC=h264
...
gst udpsrc0: PAUSED -> PLAYING

and that's it, no video window appears. What is the issue there? Are there any logs for osd, I couldn't locate any?

Let's try to resolve this first and then tackle the broadcast of osd+video.

Thanks a lot for the responses and time spending on answering my questions!

[reedev]

Are you starting the video service too? The osd is not responsible for the video but only for the telemetry etc. Right, @svpcom ?

[svpcom]

On RPI osd separated from video player. But when you use generic linux system with X11 then osd embeds video player (it renders osd text on video stream inside gstreamer)

[svpcom]

Put this file into osd sources instead of cairo_overlay.c and try again.
cairo_overlay.c.txt
It seems not all systems properly supports vaapipostproc gstreamer module

[reedev]

Aha, I understand. Thanks for the explanation.

[svpcom]

Run gst-inspect-1.0 omxh264enc to see what parameter set encoding bitrate

[svpcom]

Also check that NetworkManager and wpa_supplicant are properly configured: https://github.com/svpcom/wifibroadcast/wiki/Setup-HOWTO