Query params for event snapshots not working consistently

[mbronstein1]

I'm exploring the Frigate API and trying to determine the best approach for my specific use case. I receive event MQTT messages and fetch the clean copy of the event image to save in my database for a number of purposes, including training and being able to toggle whether I want the bbox overlayed or not in my extended application. However, I also need a cropped copy of the object temporarily for certain LLM queries. The priority is the clean copy because the bbox in the mqtt message is the bbox that is saved in the db so it should match perfectly with the image. However, this has not been as easy as I thought.

I have three different approaches here:

1. Set snapshots.bounding_box: true and snapshots.crop: true and then make separate API calls to /snapshot.jpg and /snapshot-clean.png. However, I've found inconsistencies w/ getting the clean snapshot, with over 50% of my calls returning 404. More importantly, because the clean snapshot is saved after the original (but while still using opencv to capture the image from the stream), the bbox coordinates for the object that was received initially from the MQTT message most likely has moved since receiving the message. I need to save a clean version in my application outside of Frigate but need to overlay the bbox I receive from the mqtt message on top of that image in my app. This approach has proven unreliable.
2. Set config options for bbox and crop to false and use the /snapshot.jpg route to make 2 calls for both cropped and non-cropped images. There are query parameters for crop and bbox, but the actual behavior is it usually returns the same image regardless. Looking through the source code, it looks like the parameters are only passed to the image if the object has not ended, in which case it gets a snapshot from the stream. However, the data required to crop and overlay the bbox for each event are present in the database for that event. So ideally, even if the event has finished, the query params should still be able to work. This seems like a bug or oversight to me.
3. I've also tried using the frame_time to get the recording snapshot from /{camera}/recordings/{frame_time}/snapshot.jpg, but the box almost always seems to be off even though the timing should be perfectly timed with the frame_time that I received with the bbox and other data from the MQTT message. (It should be noted that I'm also converting the returned Buffer to the correct dimensions that I am expecting from snapshots and/or passing a specific height param).

Also, I've tried cropping the original image on my end, but I have a node backend and openCV is not a great option, and image processing libraries don't crop nearly as well as openCV. I'd prefer to use Frigate to crop.

Option 2 seems to be the best approach for getting clean and cropped versions, especially considering I need the time-specific clean copy as the main priority. There are query params for the snapshot route, so ideally those should work.

If this is a bug, I'm happy to create a discussion in the bug section. If it's expected behavior, I'm curious what the reasoning is and if it would be easy to also allow the query params to work for ended events. The route is already selecting events that match the id. It wouldn't be too hard to determine at that point if the event is ended or not and react accordingly. Either way, I think the query params should work in both situations, not just one.

[NickM-27]

1. You can check has_snapshot to know if the snapshot exists yet.
2. There is a lot of history here that means the information was previously not available. Once the snapshot implementation changed it is available but it is still complicated. Realistically, in many cases the query args can't be respected once the event is ended. For example if you set bbox=0 but you have bounding_box enabled in the snapshot config then there is no way for the bounding box to be removed from the image. and if crop is enabled in the snapshot config then there is no way to get an uncropped image.
3. this is why frigate has the annotation_offset configuration, the recording and sub stream are unlikely to be perfectly in sync, but this value being used to adjust things slightly makes things a lot closer.

[mbronstein1]

Makes sense. You all are very impressive. Quick responses and meaningful changes to the app. Much appreciated.

[mbronstein1]

Last thing. It seems snapshot_time isnt implemented at all in the code, but it is in the docs for event MQTT message. Is there a purpose/reason for that key?

[NickM-27]

I believe that is old and incorrect as now the entire bit of snapshot data is included

[mbronstein1]

I just understood what you meant by "the entire bit of snapshot data is included". I noticed it only recently while logging the MQTT payload for events. This is super helpful and exactly the thing I was looking for to help resolve this issue. However, the Frigate docs are not up-to-date with this new structure. Is this something that we can expect to be kept in future releases?

{
  "after": {
    "id": "1740166345.714482-5gquej",
    "camera": "D4",
    "frame_time": 1740166369.210073,
    "snapshot": {
      "frame_time": 1740166365.712644,
      "box": [
        768,
        494,
        803,
        549
      ],
      "area": 1925,
      "region": [
        630,
        350,
        950,
        670
      ],
      "score": 0.8720703125,
      "attributes": []
    },
    "label": "person",
    "sub_label": null,
    "top_score": 0.91162109375,
    "false_positive": false,
    "start_time": 1740166345.714482,
    "end_time": null,
    "score": 0.662109375,
    "box": [
      763,
      467,
      790,
      500
    ],
    "area": 891,
    "ratio": 0.8181818181818182,
    "region": [
      628,
      350,
      948,
      670
    ],
    "active": true,
    "stationary": false,
    "motionless_count": 1,
    "position_changes": 1,
    "current_zones": [],
    "entered_zones": [],
    "has_clip": true,
    "has_snapshot": true,
    "attributes": {},
    "current_attributes": [],
    "pending_loitering": false,
    "max_severity": "alert"
  },
  "type": "update"
}

[NickM-27]

yes, I will update the docs

[brianravn]

Regarding 1: I have also been getting a lot of 404s when calling the API for snapshot.jpg. When studying the code, in some situations (if the object is currently being tracked) the API calls 'tracked_obj.get_jpg_bytes(...)'. However, that method seems to have been renamed to 'get_img_bytes(...)'. That could be the cause of the 404s.

[NickM-27]

that change comes from after this post was made, though that is an issue