Enhancement/Design: Push/Pull?

[jlpoolen]

Push/Pull For new Content?

For a robust surveillance system, I had in mind that there should be the four areas:

1. Moonfire-nvr: full preservation of camera output,

2. Motion: a module which determine if an event warrants attentions, e.g. motion (See Issue "signals" schema (for motion detection and such) #28 and Issue on-camera motion detection with ONVIF #29.)

3. AI: and artificial intelligence ("AI") module that detects what objects are found, e.g. person, car, cat, rat

4. Messaging: an alarm/messaging system based on results from AI, the logical intelligence component

Moonfire-nvr accomplishes well the preservation component; it is the foundation. And the Web interface is great to be able to extract snippets once you have decided what the start and stop times should be. I'm still looking into the motion and AI and currently I am focused on the AI component since it requires complex hardware to handle the intensive computational AI requires. Motion could be further broken down into two components: the camera's motion output and after-processing motion detection with software such as The Motion Project.

I think it would be desirable to have Moonfire-nvr be able to advertise or stream/push new content it receives. An alternative is to have the separate processes ping Moonfire-nvr to keep abreast of new events. The latter seems to be a "pull" concept, and the former a "push" concept. The push model seems easier, just offer a fire hose output, i.e. an rtsp server?, that lets other modules connect with. But offering a fire hose of data burdens the system's input/output with all these streams. I'm starting to see why gstreamer's paradigm of stream pipeline is a desirable architecture. A plugin model seems desirable.

I'm finding that I have Reolink's software determine events and save snippets, usually 15-30 seconds MP4 files in a directory. During rain, I can accumulate over 800 files for 11 cameras in a period of 6 hours. I currently use the output of Reolink as organized in a directory and exposed through Windows Explorer to be a first-look manifest of events. If I see files saved from adjacent cameras around the same time, then I know something triggered Moonfire's motion detection and I use Window's explorer with a module that displays a preview image of a particular frame of a selected file to show me quickly what is in the scene of that file.

I just wanted to get the concept "push" and "pull" in relation to Moonfire-nvr documented and see what Scott thinks. But, it's something that is going to have to be considered as further development evolves.

[jlpoolen]

Addendum: For Windows users, the K-lite package is invaluable for managing videos files through Windows Explorer.

K-Lite Codec Pack or K-Lite Mega Codec Pack

[scottlamb]

I agree about the value of analytics-based annotations and notifications. Motion, depending:

• On some systems, it might be all we can do. If CPU/GPU can't keep up with H.264 decoding, we're stuck with the camera's analytics, often just motion. Or if there's no ML accelerator, we might be able to do on-NVR motion analysis but not anything more sophisticated.
• On others, it might be a useful preprocessing stage for ML-based analytics. Frigate does this. It uses the motion analytics to reduce the number of frames sent through object detection and to crop those frames down to just the region of most interest (increasing accuracy when the object detection model runs on tiny 300x300 images).
• On others, it might be unnecessary. If you can do analytics on e.g. 5 fps/camera with a 704x480 image (or even a 1920x1080 image), it'd be simpler to just do that.

I think it would be desirable to have Moonfire-nvr be able to advertise or stream/push new content it receives. An alternative is to have the separate processes ping Moonfire-nvr to keep abreast of new events. The latter seems to be a "pull" concept, and the former a "push" concept. The push model seems easier, just offer a fire hose output, i.e. an rtsp server?, that lets other modules connect with. But offering a fire hose of data burdens the system's input/output with all these streams. I'm starting to see why gstreamer's paradigm of stream pipeline is a desirable architecture. A plugin model seems desirable.

Moonfire's server already offers one way for a client to watch one or more live video streams via live.m4s. This is what the UI uses; other things can too. We could also add support for acting as a RTSP and/or WebRTC server (although it's not at the top of my todo list now).

Separately, I want to add a WebSocket feed for metadata, such as changes to recordings/signals/events, reconfiguration, and the like. That's #40. This is less firehose and more leaky faucet. This might be close to the next thing I work on, as it's somewhat at the union of two things I want: analytics and rock solid web-based configuration.

We could also publish metadata changes on MQTT as Frigate does for its events. This would be an easy way to integrate with Home Assistant, which a lot of people use for notifications.

My current plan for on-NVR analytics is to have a separate process that watches the video streams of interest and updates events accordingly. It's really not a problem for the Moonfire server to relay the video streams to it. Proxying bytes over loopback is rounding error compared to actually analyzing those bytes (including H.264 decoding, scaling, and ML). Keeping it in a separate process has some benefits. We could spread it across machines if desired. We could iterate on it more quickly while keeping the main server running. We could

[xromansx]

An interesting info that the Motion (software) has custom version with onvif event motion detection.

[jlpoolen]

@scottlamb , I feel badly about being nonresponsive on this great project. But, I am in the throws of a complicated air space analysis re: flights and need to finalize an analysis shortly, so I have not made time to even upgrade my Pi and your software --- what I have is several years old and has been reliably working and doing its job (I do want to try the dual camera upgrade, but that involves upgrading my entire Raspberry Pi and a migration).