Introduce Python Celery Workers in birdnet_analysis.py

[tomtom215]

Hello to the awesome maintainers of this fork!

Just wanted to toss an idea out there as I've seen some people trying to run this on lower spec hardware than RPi 4's -

Adding python celery workers to scripts/birdnet_analysis.py

As of right now I think the current threading and queue works well enough, but enhancing it with celery workers may make it a bit more stable on lower spec hardware or if there is a ton of simultaneous birds heard in a short segment.

Yes, it would probably add a redis dependency , but perhaps it could easily be swapped with Valkey, especially since Ubuntu plans to bundle it https://ubuntu.com/blog/valkey-ubuntu

[Nachtzuster]

Thanks for the idea.

On eg RPI zero 2w the limited memory is usually the problem. Adding redis/valkey likely would make thigs worse.
Could you explain the 'instability with tons of simultaneous birds' bit? I'm not aware of that.

[tomtom215]

I get your point about limited memory, but I think it might be able to help with memory use over all due to better thread scheduling.

I have not run into the instability with simultaneous calls as I'm running a RPi5 8Gb, but in reading in how the code for the analysis runs, it is using threads and queue from python - so if you're running on a RPI zero2w with 4 slow threads, if there is a lot of bird songs from multiple birds over multiple analysis segments, and the end user is also trying to browse the Ui, it will lead to competing threads as well as memory. This is where redis and celery workers may be more efficient in the thread scheduling and therefore the memory use is improved as it is designed for this type of scenario compared to manually configuring it with python's threads and queues, and is why I wanted to propose the idea.

There's an updated fork of a semi-abandoned project called BirdCage which uses birdnet for the analysis, but focuses on RTSP audio sources. Here's their analysis functionality that uses celery for a reference:
https://github.com/seajaytea/BirdCAGE/blob/main/birdcage_backend/app/stream_processing.py