All notable changes to the SentinelCam project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
Much of the following is more properly categorized as still in the wishlist phase of design.
- Continue refinements to
Outpostimplementation.- Modernize object detector capabilities with support for newer algorithms.
- Implement filtering mechanism based on object detection results.
- Begin designing support for a model deployment framework that can be used to support custom lenses as another layer beneath object detection.
- Support multiple result sets from both
Outpostevent management, and from running sentinel tasks. Needed to support the capture of multiple neural nets producing results in parallel from a single event or task. - Aditional refinements for the sentinel module.
- Implement result signaling to Twilio, to Node-RED via MQTT, and to imagehub logging to help support reponsiveness to events in progress as well as for knowledge and state management throughout the larger system.
- Provide an abstraction to support a set of reusable design patterns for the most common ring buffer control techniques.
- Implement internal housekeeping to periodically purge oldest job history. This is needed to avoid an obvious memory leak. Perform as a routine task during quiet periods. Incorporate a job history report of statistics and performance metrics as an output. This could ultimately fuel a sentinel health check, perhaps built to support agency based on self-diagnosis.
- Support a job runtime limit as a configurable setting per task engine? Provide tolerance based on the queue length for tasks waiting in that job class.
- datapump needs data sink storage and data analysis with clean-up and reporting as a nightly task. Will need control panel instrumentation for this as well, including perhaps charts of the storage breakdown, utilization, and available capacity of the data sinks.
- Add missing health-check monitor from the camwatcher to detect and restart a stalled imagenode.
- Just a general note of caution. Run this at your own risk. All major components are under active development. SentinelCam is an on-going research experiment which may, at times, be somewhat unstable around the edges.
- Vaccinations against zombie subscriber syndrome.
- Cleaned up sentinel job history dump.
- Correct tracking timestamps published for outpost DepthAI pipelines.
- Addressed oversleeping in the sentinel Task Engines and Job Monitor thread.
- Include face recon status as criterion in sentinel
DailyCleanuptask. See explanation in the YAML configuration file within the Tasks folder. - Refactored sentinel task chaining logic for efficiency. Execute chained tasks immediately on the same engine if the target job class is supported.
- Added systemd timer setups for daily maintenance sentinel task.
- Moved results selection and sorting for each frame outisde the watchtower event review loop. This allowed for reaching expected performance goals, and required the reintroduction of a delay between frames to slow down the replay so that it aligns with the capture rate.
- The watchtower now selects an appropriate white or black text color for result labels based on a lumninace factor of the randomly selected background color assigned to each distinct item.
- A complete refactoring of the
DataFeedfor conciseness and clarity. Will now raise aTimeoutErrorexception for an unresponsive datapump connection, including a close and reconnect on the 0MQ socket. - Change from UTC to localtime for all timestamps.
- Bug sweep on the watchtower wall console. Working now. Has core functionality in place for both live viewing, and previous event display.
- Clean-up on the watchtower wall console. Has 99.94% of the core functionality needed for both live viewing, and previous event display.
- Revised EOJ status message from sentinel for event syncrhonization with the watchtower.
- Another premature push. Untested work-in-progress on the watchtower wall console. This has nearly, ~80%, of all the core functionality required for its primary use cases. The full feature list is still only conceptual at this point.
- An extremely early push, for safe keeping, of the watchtower wall console. Designed for the Raspberry Pi 7-inch touchscreen display, this is a combination live outpost viewer and prior event display tool showing image analysis results. Though a working proof of concept, this barely qualifies as a prototype; just a little buggy and critical funcionality is missing.
- The
DataFeednow raises aImageSetEmptyexception when requesting the image list and no images were captured for the requested event.
- Defined
FaceListclass to encapsulate access and updates against thefacelist.csvfile of faces which are in use or awaiting analysis. Storage is in a CSV-format text file used to support management of the facial recognition and learning pipeline. This also serves as an event lock preventing data deletion whenever present within the dataset. - Added
FaceSweepandFaceDataUpdatetask definitions to flesh out the facial recognition and learning pipeline. The former identifies new candidate images to be considered for inclusion in the next model update. The latter writes selected candidates into thefacedata.hdf5file of embeddings used for modeling.
- Cleanup and shakedown of facial recon pipeline.
- Corrected sentinel ring buffer loading when task is configured with
ringctrl: trk. Now loads only the subset of unique images for the tracking type. - Delay clearing of sentinel on-deck presence until task has either reported a successful start or failed during initialization.
- The sentinel now supports easily configurable pipeline definitions, through task chaining and
aliasing. This is managed by new
Taskattributes in the YAML file. - Defined an
EventListclass to encapsulate the most common event selection and processing methods: by date or date range, by specific date and event, and from a from text file with a list of events to process. An optional tracking type parameter is supported to refine the selection.
- Facial reconnaissance pipeline now incorprates Euclidean distance metrics against the individual baselines kept for known individuals. OpenFace embeddings are used for both this purpose and also for training the SVM classifier. This ensemble approach both bolsters classification results and helps address the open set recognition problem inherent in the overall design.
- The
DataFeednow raises aTrackingSetEmptyexception when attempting to retrieve tracking data that does not exist.
- Include support for carrying object ID references within tracking data updates from tasks running on the sentinel.
- Now properly reporting messages with ERROR and WARNING logging levels from tasks running on the sentinel. Previously, these were being logged with a level of INFO.
- Include alpha version of facial reconnaissance pipeline.
- Send empty image from datapump when size of JPEG file is zero.
- Corrected tracking references for camwatcher updates when task ringctrl is
trk.
- Event index modified to include captured camera image dimensions (width, height). This is helpful for buffer allocations during downstream processing.
- Support an alternate image cursor for populating the sentinel ring buffer. This allows image retrieval to be restricted to only images included in a specific result set. The default for this is to provide all images captured for an event.
- Exit the sentinel task gracefully whenever image retrieval fails.
- The sentinel on-deck status was not being properly cleared for jobs running in secondary classes.
- Face detection pipeline introduced. This demonstrates a sentinel task designed to run against just a subset of event images. Skip-ahead logic is used to advance the ring buffer start dynamically, so that only frames with a previously detected "person" object are analyzed.
- Corrected timestamp on logged
SpyGlassresults to match timestamp of frame being analyzed. This was previously being stamped with the time results were received, resulting in an noticeable lag. Bounding boxes were sometimes being drawn behind moving objects, following them like some kind of ghostly electronic shadow. - Support an event type selection as a part of sentinel task configuration. Each task
receives a set of tracking data, which defaults to
'trk'. This change allows tasks to either process every frame in the event, or selectively analyze only a subset of frames based on results in a previously collected dataset. - Added basic task performance instrumentation to the sentinel end-of-job message.
- Corrected alignment logic between results and images when presented for video review. Factoring in estimations around elapsed time within the event has not been helpful.
- Timestamp mapping for sentinel tasks was incorrectly based on the first
trkrecord, rather than the first frame. - Fixed a bug in how the
JobManagerfor the sentinel manages the task list. Failed task initialization could sometimes lead into a spiral of death and destruction. - The sentinel was occasionally attempting to feed a ring buffer no longer in use, when a task had selected an early exit. This exposed a bug where ring buffer operations were being executed against tasks just ending, resulting in failures.
- Spit and polish for camwatcher v3 support. Bug clean-up sweep.
- Minor updates to
video_review_df.pyfor selecting alternate result sets.
- Restructured python module organization for the data sink codebase.
- Fleshed out camwatcher setups and migrated into a YAML document.
- Child process image subscribers are now pre-loaded when the camwatcher initializes. These are kept resident between events for faster response to new activity.
- A list of known
Outpostnodes has been added to the camwatcher settings. Subscriptions to these are established automatically at startup. A new camera node can still introduce itself dynamically. This change allows the camwatcher and imagenode applications to restart independently of each other. Previously, all camera nodes had to be restarted whenever the camwatcher was bounced. - Now providing sentinel task results via 0MQ log publishing. This content includes analysis results and status messages, along with internal errors and warnings. The sentinel does not write to a logfile on local disk. All logging is published for any interested subscribers.
- A subprocess agent was added to the camwatcher for subscribing to sentinel logging messages and capturing task analysis data.
- Post-event processing logic was added to the camwatcher. This is used to automatically submit a
parameterized task to the sentinel for a complete analysis of all event data. Results are stored
as a supplement to the original captured tracking data provided by the
Outpostnodes.
- More hardening in the data layer. Improved exception handling, with a more graceful failover from
the datapump to a
DataFeedrequester.
- Data layer resilience.
CamDataclass was failing when event detail CSV files were missing. Now properly returns an emptypandas.DataFramefor this condition. - Additional tightening of the sentinel for efficiency and stability. Fixed a bug in the start logic when task has no eventID.
- Provide support for sentinel task engines to have complete control over the ring buffer, including issuing start commands, and changing context to a new event.
- Stress testing the sentinel module with multiple task engines. Support a ring buffer model for customization by task engine if desired. Confirm affinity to task engine by job class.
- Some early code clean-up of the sentinel module. Added a HISTORY command to dump the current comprehensive list of job request status details to the logger in JSON format.
- First early working prototype of the sentinel module. This is an image analytical engine that accepts job service requests over ZeroMQ. Parallelization is provided by a multi-processing design, allowing multiple tasks to run at once. Employs a dedicated I/O thread to supply image requests for use in analysis tasks through a set of ring buffers in shared memory.
- Corrected camwatcher filename generation for JPEG files when timestamp has no fractional second.
- Outpost state machine refinements. Begin adding missing logic to gaps in scene management functionality; this addresses the runaway spyglass bug.
- Added an event delete command to datapump. This runs as a background task and will purge all stored data for a specific event.
- Revisions and corrections to OAK camera neural net retrieval.
- Added
video_review_df.pymodule, leaving original version in place for reference. This uses theDataFeedfor operation within a WSGI container, and represents the next logical step in the evolution of this function.
- Added OAK-1 camera support for running DepthAI pipelines as the primary data collection device.
- Support motion-only mode for event logging without object detection or tracking.
- Integrated object tracking support is now optional.
- Now using MessagePack for marshalling IPC exchanges between
SpyGlassandOutpost.
- Replaced non-sensical approach to Outpost state management with something sane, and correct.
- Code revised for operation within OpenVINO environment. Tested with an Intel NCS2 accelerator.
- Changed motion detector to use the OpenCV baseline MOG2 background subtraction library.
- Added support for dlib correlation tracker.
- Additional work on the multiprocessing handshake. State management seems to have a loose tent stake. Somewhere. I'm beginning to think that the object detector is intermittently failing and returning bad data. Still looking for the real issue.
- Refinements to dance choreography between the Outpost and SpyGlass.
- Added heartbeat logging from the outpost. This simply reports the current image publishing frame rate at 5 minute intervals. This will be saved by the camwatcher whenever its internal logging level is set to INFO. It may be smarter to direct this data down to the imagehub for access from the librarian.
- Over-publishing image data with 0MQ is not smart. On a Raspberry Pi 4B, have measured
publishing rates for a (320,240) resolution image, compressed to JPEG, at 150+ frames
per second. This is insane, at least for the hardware we're running on and any of the
intended use cases driving this design. For a PiCamera, the hardware chip does not even
collect data faster than about 32 frames/second. Moving data is not free. There is always
a price to pay. Implemented an image publishing throttle for the
Outpostbased on configured frame rate. Better to be kind to such a nice little box as the Raspberry Pi. High stress for no payback? Always say no to such antics.
- Reduced latency between
Outpostand theSpyGlassby moving 0MQ signaling protocol fromtcp://127.0.0.1toipc://name. Had to swap theImageSenderandImageHubendpoints for this, which also provided for a more sensible handshake during initialization.
- Now using an imageZMQ REQ/REP pair to rig the IPC signaling between
Outpostand theSpyGlass. The outpost implements a polling mechanism on the connection to provide for a non-blocking receive until results are ready.
- First working prototype of the datapump module. This is a stand-alone process
intended for running on the same node as a camwatcher. This module services access requests
to the data and image sinks over imageZMQ transport, specifically for use with the
DataFeedclass from a process running on another node, such as the Sentinel itself. - Added example datafeed module implementing
DataFeedrequests to the datapump. This is still evolving. - Fleshed out initial
Outpostfunctionality for the imagenode project, including an early version of theSpyGlassas a multiprocessing vision analysis pipeline.
- Image folder path added as argument to CamData initialization.
- Adopted simplejpeg library in place of using OpenCV for more efficient frame file encoding/decoding.
- Corrected handling for updating the EventID used by an active camwatcher image subscriber.
- Data model for tracking events revised to substitute bounding rectangles for detected objects rather than an object centroid. Classification also added for those events where this can be estimated in real time.
- Added camdata module defining the new
CamDataclass. Encapsulates access to CSV tracking data collected by the camwatcher. Providespandas.DataFrameobject references. - Added systemd service definition for camwatcher operation.
- Revert to baseline imagehub module. Camera handoff to the camwatcher is now performed directly from the imagenode outpost detector.
- Complete refactoring of all imagenode changes specific to SentinelCam outpost functionality into a single module.
- Image capture within the camwatcher now includes the frame capture time as a component of the filename. This more accurately associates timestamps with individual frames and improves performance of video replay. Relying on filesystem timestamps for this was a misstep.
- Utilization of PostgreSQL as a component of the camwatcher data layer replaced with data tables mapped onto a set of CSV-format files; a simple and efficient capture method. Also provides the broadest integration support.
- Example event viewer application
video_review.pyrevised to conform to the new camwatcher data model. Functionality fleshed out to include date and event selection. Demonstrates use of theCamDataobject to retrieve event and image data.
- PyImageSearch folder removed from imagenode, and contents merged into a single sentinelcam library.
- First early working draft of camwatcher functionality.
- Includes a trivial viewer example for replaying a captured video event.
- Modified imagenode to implement log and image publishing. Sends a camera startup command to the connected imagehub. Added an experimental object tracker to exercise camwatcher operations.
- Modified imagehub to implement the camera handoff to camwatcher from an imagenode initialization.