[Detector Support]: 0.15.0 Hailo8L compatible models #15137
-
Describe the problem you are havingI have been using the raspi 5 with Hailo8L to run Frigate. The current model ssd_mobilenet_v1 is ok, but seems to lack, it keeps registering my cat as a person as an example. I tried using the Yolov8s model (pre-trained hef from Hailo Model Zoo) and COCO label map (model trained on coco) and: Good:
The inference time was reasonable at 30 ms, but again, I could not get objects other than person and car to work. My core question: Will there be additional models supported by H8L when Frigate 0.15 launches? Is there anything I can do to help? Thank you! VersionFrigate 0.15.0 Beta1 Frigate config filemqtt:
enabled: true
host: ****
port: 1883
user: ****
password: ****
detectors:
hailo8l:
type: hailo8l
device: PCIe
model:
path: /config/model_cache/h8l_cache/ssd_mobilenet_v1.hef
#labelmap_path: /config/model_cache/h8l_cache/coco_labelmap.txt #used when I ran Yolov8s
model:
width: 300
height: 300
input_tensor: nhwc
input_pixel_format: bgr
model_type: ssd
#labelmap_path: /config/model_cache/h8l_cache/coco.txt #Tried it here as well with no luck
go2rtc:
streams:
doorbell_main:
- rtsp://admin:{FRIGATE_RTSP_PASSWORD}@****:554/h264Preview_01_main
- ffmpeg:rtsp_cam#audio=opus
doorbell_sub:
- rtsp://admin:{FRIGATE_RTSP_PASSWORD}@192.168.1.214:554/h264Preview_01_sub
- ffmpeg:rtsp_cam#audio=opus
bedroom_sub:
- rtsp://tapocam:****@****:554/stream2
- ffmpeg:#audio=aac&input=-rtsp_transport+tcp
livingroom_sub:
- rtsp://tapocam2:****@****:554/stream1
- ffmpeg:#audio=aac&input=-rtsp_transport+tcp
#live:
#stream_name: doorbell_main
#height: 720
#quality: 8
#ffmpeg:
#hwaccel_args: preset-rpi-64-h264
cameras:
doorbell:
ffmpeg:
inputs:
- path: rtsp://localhost:8554/doorbell_main
roles:
- record
- path: rtsp://localhost:8554/doorbell_sub
roles:
- detect
live:
stream_name: doorbell_main
height: 720
quality: 8
record:
enabled: true
retain:
days: 5
mode: motion
alerts:
retain:
days: 15
detections:
retain:
days: 15
snapshots:
enabled: true
retain:
default: 15
detect:
enabled: true
width: 640
height: 480
fps: 5
objects:
track:
- person
- car
- cat
- dog
motion:
threshold: 50
contour_area: 10
improve_contrast: true
bedroom:
ffmpeg:
inputs:
- path: rtsp://localhost:8554/bedroom_sub
roles:
- record
- detect
live:
stream_name: bedroom_sub
height: 720
quality: 8
record:
enabled: true
retain:
days: 5
mode: motion
alerts:
retain:
days: 15
detections:
retain:
days: 15
snapshots:
enabled: false
retain:
default: 15
detect:
enabled: true
width: 1280
height: 720
fps: 5
objects:
track:
- person
- cat
living:
ffmpeg:
inputs:
- path: rtsp://localhost:8554/livingroom_sub
roles:
- record
- detect
live:
stream_name: livingroom_sub
height: 720
quality: 8
record:
enabled: true # Set to true if you want recording; set to false if no recording
retain:
days: 5
mode: motion
alerts:
retain:
days: 15
detections:
retain:
days: 15
snapshots:
enabled: false
retain:
default: 15
detect:
enabled: true
#width: 1280
#height: 720
fps: 5
objects:
track:
- person
- cat
version: 0.15-0docker-compose file or Docker CLI commandservices:
frigate:
container_name: frigate
privileged: true # this may not be necessary for all setups
restart: unless-stopped
image: ghcr.io/blakeblackshear/frigate:0.15.0-beta1-h8l
shm_size: "512mb" # update for your cameras based on calculation above
devices:
- /dev/hailo0:/dev/hailo0
volumes:
- /etc/localtime:/etc/localtime:ro
- /home/pi/frigate/config:/config
- /mnt/data/docker/frigate/media:/media
- type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
target: /tmp/cache
tmpfs:
size: 1000000000
ports:
- "8971:8971"
- "5000:5000" # Internal unauthenticated access. Expose carefully.
- "8554:8554" # RTSP feeds
- "8555:8555/tcp" # WebRTC over tcp
- "8555:8555/udp" # WebRTC over udp
environment:
FRIGATE_RTSP_PASSWORD: "****"Relevant Frigate log outputNo relevant logs available. I never saw anything in the logs showing a reason for the failures.Install methodDocker Compose Object DetectorOther Screenshots of the Frigate UI's System metrics pages
Any other information that may be helpfulNo response |
Beta Was this translation helpful? Give feedback.
Replies: 9 comments 38 replies
-
|
it depends on the detector, some detectors require explicit handling of each model type while others handle that internally. Technically yolov8 is not supported in Frigate, but hailo outputs in the same format. I don't see any reasons the labelmap would not work unless the labelmap for yolov8 is different. |
Beta Was this translation helpful? Give feedback.
-
|
Here is the label map I used, named "coco_labelmap.txt". |
Beta Was this translation helpful? Give feedback.
-
|
You need to hone in object sizes and adjust the default minimum score higher and or the threshold as well. That will help some. You can also play with motion settings. Finding that motion sweet spot will reduce the number of frames the detector is fed and can improve accuracy |
Beta Was this translation helpful? Give feedback.
-
|
Thank you, this is something I have been testing slowly. However, the main issue is still the labelmap. It thought my bed was a donut. While the default model knows it’s a bed. |
Beta Was this translation helpful? Give feedback.
-
|
Did you make any more progress on this @NathanMTL ? After getting my set up dialed-in on the base model and my hw (pi5, hailo8l on pcie, 3 cameras), I just dropped the yolov8s model in (from the hailo model zoo). Will post back my findings. |
Beta Was this translation helpful? Give feedback.
-
|
No I haven’t, I’m struggling to find information about which models / models types are supported in Frigate to start with. But I’ll keep everyone posted if I get it working, let me know how your set up goes! |
Beta Was this translation helpful? Give feedback.
-
|
I've had yolov8s going for 36 hours or so now, and its definitely detecting more objects than just car & person (and it seems more accuratel too). Lots of correct matches on dog, cat, backpack, etc. My inference times have almost doubled though from ~25ms average up to high 40ms. My config for detector and model looks like this (I just inlined the labels): |
Beta Was this translation helpful? Give feedback.
-
|
Cool, thanks I’ll give this a try! Yeah, I noticed yolov8s is way more accurate than the default model. |
Beta Was this translation helpful? Give feedback.
-
|
@alschroder Thank you! This worked great, I didn't realize you could put the label map directly into the config file like this. For Beta 1 on Rpi 5 8gb and Hailo8L chip I am getting 31ms inference and Yolov8s is way more accurate than the ssdmobilenet model. It hasn't said my cat was a person once and identifies all objects! |
Beta Was this translation helpful? Give feedback.
-
|
Small note: I think (I could be wrong) that Yolo uses "rgb" pixel input format, that is what I have selected. |
Beta Was this translation helpful? Give feedback.
-
|
@alschroder, @NathanMTL: I am really a Beginner in this so it might be a simple Question for your, but how did you Compile/Convert this ZIP File obtained from Hailo Github Repository (containing I found Several Threads on Hailo Community Forums but no conclusive Answer. With the latest Dataflow Compiler 3.29.0 running on Python 3.10 (I created a Dockerfile to build a Docker Image with a Python 3.10 Environment since Python 3.11/3.12 are not supported). However I'm now fully stuck because I don't know which Commands I need to issue. There is a bit of a Split between: a. 2-Step Approach
b. Just Compile directly and train the Model by Specifying Furthermore it seems some posts reference Furthermore I'm not sure what I need to do in order to train it (if I need to train it myself, I guess I need to feed it some Pictures but not clear how to do and go from there). Do you have some more Details on how you managed to get this Thanks 😃 |
Beta Was this translation helpful? Give feedback.
-
|
Just a general comment for those with Pi's and Hailo boards (and since I missed it when I first set up my card) make sure you move to PCIe Gen 3. My inference times went from high 40s (as mentioned above) to around 30ms, which seems in line with others here. |
Beta Was this translation helpful? Give feedback.
-
|
😅was wondering why my stats look different to others. Looks like I'll have to do some more fault finding. |
Beta Was this translation helpful? Give feedback.
-
|
I think indentation is my problem |
Beta Was this translation helpful? Give feedback.
-
|
yes |
Beta Was this translation helpful? Give feedback.
-
|
Thank you. |
Beta Was this translation helpful? Give feedback.
-
|
I believe your |
Beta Was this translation helpful? Give feedback.
-
|
I am still not sure why I can't get it to detect cars. Same model (Yolo), Label Map, etc. It worked before without the label map, but then couldn't identify anything except persons and cars. |
Beta Was this translation helpful? Give feedback.
-
|
Hello! I did try this, but frigate show me "person" everywhere, the label map doesn't seem to be used correctly... any idea? |
Beta Was this translation helpful? Give feedback.
-
|
@alschroder , @NathanMTL or @justindhillon I am confused by the statement here that the newest version of frigate (probably 0.15) does not allow loading any model for the Hailo 8L other than the default /config/model_cache/h8l_cache/ssd_mobilenet_v1.hef , which is also in line with me not being able to load another model.
Could you provide your config and confirm that you're using the yolo model on the newest frigate (0.15) on hailo8l? |
Beta Was this translation helpful? Give feedback.
-
|
Hey @Boldfor, sorry for the delay I have been under the weather. You can run other models here is a breakdown:
Example: You have 5 camera feeds and you want detection for each at 5 frames per second. To keep detection working, you need to be able to run 5x5 (25) frames per second through your detector. You also don't want something too slow.
Using your terminal you can run something like "scp ~/downloads/yolov11m.hef user@192.168.1.xxx [ Your device running frigate ] :/config/model_cache/h8l_cache/" to transfer a model from your computer to lets say your raspberry pi.
I hope that helps! :) |
Beta Was this translation helpful? Give feedback.
-
|
@NickM-27 Thanks for the update on 0.16, it's exciting to hear about the additional model support! Looking forward to the beta! |
Beta Was this translation helpful? Give feedback.
-
OK, I just gotta ask: Is it truly Frames Per Second (camera FPS) or is it Detections(regions with motion) per second? It seems to me that Hailo likes to specify FPS as if a single frame is used for a single inference. However, my poking around in Frgiate seems to be more regions/second. Looking at the period required for inference -- say YoloV11s @ 24mSec yielding approx 40 inferences per second before topping out as the rate limiting factor (assuming not cpu limited). However, my two little cameras (8fps, 5fps; very busy complex scenes) max out and skip 'frames'. I do far better at minimizing the skips with the 640x640 region size opposed to the 320x320. It would be really nice if @OmriAx would provide a few paragraphs on optimizing the Hailo/Frigatesetup. The notice over in the Hailo repo appears to lack consistency with what needs to be specified or what is left to imagination. For example, are we better off with 320x320 region size, or the model's 640x640? What are the recommended settings for pixel format and color map coming from the camera? (Is batching up regions within a single camera frame possible?) BTW, I've been running the 'official?' Hailo integration off of the 0.16 branch and really like the features. |
Beta Was this translation helpful? Give feedback.
-
|
yeah thank the Hailo team, they are the ones that revamped the detector including better performance than 0.15
you can't really compare these, Frigate's regions are not constant, so one frame you might get 3 regions that run detection and another might be 1. I don't think hailo is breaking the frames apart into mulitple regions.
Specifically for Frigate it depends on your use case, cameras, etc. If you are looking to detect very far away / small objects then 320x320 has shown to do better in our testing across many hardware and model variations. Meanwhile, when you have a lot of objects the 640x640 is able to handle larger area while doing fewer inferences, as you described. |
Beta Was this translation helpful? Give feedback.
-
|
@aintgotnoname Cool point, I never thought about it like that. Well, however it works, Yolov11m worked best for me with 3 cameras. |
Beta Was this translation helpful? Give feedback.
-
|
Running the default However, when trying to run I've downloaded the compiled These are my Docker Compose:version: '3'
services:
frigate:
container_name: frigate
privileged: true # this may not be necessary for all setups
restart: unless-stopped
stop_grace_period: 30s # allow enough time to shut down the various services
image: ghcr.io/blakeblackshear/frigate:0.15.1-h8l
shm_size: "512mb" # update for your cameras based on calculation above
devices:
- /dev/hailo0:/dev/hailo0
volumes:
- /etc/localtime:/etc/localtime:ro
- /home/frigate/config:/config
- /home/frigate/storage:/media/frigate
- type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
target: /tmp/cache
tmpfs:
size: 1000000000
ports:
- "8971:8971"
- "5000:5000" # Internal unauthenticated access. Expose carefully.
- "8554:8554" # RTSP feeds
- "8555:8555/tcp" # WebRTC over tcp
- "8555:8555/udp" # WebRTC over udp
environment:
FRIGATE_RTSP_PASSWORD: "password"Frigate
|
Beta Was this translation helpful? Give feedback.
-
|
You have to set the path on the model itself but under the detector |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, @NickM-27 ! I modified the yaml file to the following and it worked: detectors:
hailo8l:
type: hailo8l
device: PCIe
model:
width: 640
height: 640
input_tensor: nhwc
input_pixel_format: rgb
path: /config/model_cache/h8l_cache/yolov8s.hef
model_type: yolox
labelmap:
0: person
1: bicycle
2: car
3: motorcycle
4: airplane
5: bus
6: train
7: truck
2: car
...I had some trouble getting the model working, though. The compiled model in the model zoo is a I solved this by downloading the following model from AWS: Also, I had to make the model file readable from the container by either changing ownership of the file from within the container ( and then restart Frigate. By making these changes, it all works, and the detections are more accurate than with the mobilenet model. |
Beta Was this translation helpful? Give feedback.
I've had yolov8s going for 36 hours or so now, and its definitely detecting more objects than just car & person (and it seems more accuratel too). Lots of correct matches on dog, cat, backpack, etc. My inference times have almost doubled though from ~25ms average up to high 40ms.
My config for detector and model looks like this (I just inlined the labels):