[Bug]: go2rtc process disappears from stats if it restarts itself #16130
Replies: 2 comments 7 replies
-
|
Thanks, this has never been reported before but is something that will be fixed by a larger process refactor in a future version of Frigate.
The more interesting tidbit here is that go2rtc is actually being killed - perhaps by your operating system's OOM killer? This is something you should be able to mitigate. |
Beta Was this translation helpful? Give feedback.
-
|
For sure. With all of the other support requests from users I've handled, I don't think I've ever seen go2rtc killed like that before. Though I have certainly seen all sorts of oddities with Proxmox. |
Beta Was this translation helpful? Give feedback.
-
|
You might consider trying the |
Beta Was this translation helpful? Give feedback.
-
Turns out you were right. I happened to catch it pretty much when it happened. Logged in to the server and noticed RAM usage was rather high. It eventually CPU spiked and that's right when go2rtc logged that it wasn't responding and RAM spiked a bit as well (I have 4 cameras and it was consuming 7gb), then killed itself and RAM dropped back down to 1gb. Interestingly, at this low RAM usage, everything is still working, so something seems to have a memory leak of sorts, and I would most likely suspect go2rtc (just a hunch, no other reason). I realize Frigate 0.15 is close to being released, so I'm going to just wait for that and hope the issue goes away. If not, I'll try to put some effort in to at least figuring out what process is gobbling up memory. |
Beta Was this translation helpful? Give feedback.
-
|
If I had to guess, it's likely due to camera connection / spotty wifi / bad data from the camera. Proxmox in the mix also adds another layer of complexity. As I mentioned before, you might want to try using the |
Beta Was this translation helpful? Give feedback.
-
|
Ok, I will give that a shot and see how it goes (while waiting for 0.15). My doorbell is a wifi camera and I can confirm it's connection is not 100%. |
Beta Was this translation helpful? Give feedback.
-
|
I was having this same issue, where Go2RTC would be killed as the health check couldn't reach the api. I increased the SHM value in my docker compose and it seems to have stopped. |
Beta Was this translation helpful? Give feedback.
-
|
In my situation, passing the wifi camera through ffmpeg as suggested significantly reduced errors. Unfortunately, it's also my doorbell which has an audio backchannel for 2-way audio and can only have one consumer for it and when consuming it prevents the doorbell from working, so I can't pass that stream to ffmpeg (but on the up side, that stream isn't consumed all the time, just on demand). All of this doesn't resolve the issue mentioned, but it's not a pressing issue for me any more (btw thank you @hawkeye217). |
Beta Was this translation helpful? Give feedback.
-
Checklist
Describe the problem you are having
Sometimes go2rtc will log a message saying it wasn't responding to ping and will restart (likely issues with flaky cameras).
When go2rtc restarts, Frigate handles almost everything fine - but it doesn't seem to track the new go2rtc process and still reports the old PID in stats, but of course that process doesn't exist any more. Also, when you go to look at Frigate's system metrics, there's no report of the go2rtc process at all under Other processes. It's still running however and everything else is working - but it's like Frigate has lost track/sight of the process.
Steps to reproduce
...
Version
0.14.1
In which browser(s) are you experiencing the issue with?
No response
Frigate config file
docker-compose file or Docker CLI command
Relevant Frigate log output
Relevant go2rtc log output
Operating system
Proxmox
Install method
Docker Compose
Network connection
Wired
Camera make and model
Lorex
Screenshots of the Frigate UI's System metrics pages
Any other information that may be helpful
I added the non-monotonous DTS errors I get (just one line, but there will be a lot in the logs) in case it's at all relevant - happens right around when the camera is about to flake out and then come back ok. Also, my Front_doorbell camera is wifi and does suffer from packet loss once in a while, so it also throws errors fairly often, crashes ffmpeg (used to convert audio) and then everything eventually comes back ok.
As well, since I added a Coral USB to my setup, I've noticed that it can sometimes hang and the watchdog will restart it - not sure if that's at all related (in the logs it happens a while later).
Beta Was this translation helpful? Give feedback.
All reactions