Using AI and Low-Cost Camera to Detect Harmful Algae

Overview

This project aims to provide a practical, convenient, and efficient tool to monitor water quality and mitigate / prevent harmful algal blooms in real-time by:

• Fine-tuning pre-trained AI models to detect harmful algae
• Leveraging the portability of smartphones and low-cost cameras
  • Tested with ESP32-CAM AI Thinker and ESP32-S3-EYE, but compatible with a large variety of other boards/SoCs
  • See 'Boards' section in appendix.md for full list of compatible cameras

Since it's designed to be user-friendly and cost-effective, it's also suitable for educational and research purposes.

Project Demo

Detected algae are annotated with a bounding box, predicted class/category, and the AI model's confidence in its prediction(s). Trackbars allow user to configure the AI model's attributes — Confidence, IoU (i.e., Intersection over Union), and Max Detections — in real time.

custom_yolov8n.pt (i.e., custom AI model) validation results

Predict	Correct

Though it may appear so, these aren't duplicates! The left image shows what the model detected, while the right image shows the correct labels. The original images are from one of the batches in the model's validation dataset.

Hardware: Nikon microscope with ESP32-CAM AI Thinker and illuminator

ESP32-CAM AI Thinker inside a custom 3D printed lens attachment atop the microscope's eyepiece.

Requirements

Important

Conda is technically the only hard requirement, though by itself provides a barebones experience and doesn't show the program's full functionality.

Refer to Customization in appendix.md if you don't want to or can't use a(n) ESP32-CAM and/or microscope.

• Any of the boards listed in appendix.md
• Nikon microscope with 3D printed lens attachment and illuminator
• Micro-USB cable (to connect board to computer)
• Roboflow account
• Dataset
  • Google Drive: Original and unedited
  • Roboflow: Includes annotations, pre-processing, and augmentation
• Visual Studio Code
• PlatformIO plugin for Visual Studio Code
• Google Colab account
• Anaconda OR Miniconda

Tip

If you have trouble deciding between Anaconda and Miniconda, please refer to the table below:

Anaconda	Miniconda
New to conda and/or Python	Familiar with conda and/or Python
Not familiar with using terminal and prefer GUI	Comfortable using terminal
Like the convenience of having Python and 1,500+ scientific packages automatically installed at once	Want fast access to Python and the conda commands and plan to sort out the other programs later
Have the time and space (a few minutes and 3 GB)	Don't have the time or space to install 1,500+ packages
Don't want to individually install each package	Don't mind individually installing each package

Typing out entire Conda commands can sometimes be tedious, so I wrote a shell script (conda_shortcuts.sh on GitHub Gist) to define shortcuts for commonly used Conda commands.

Example: Delete/remove a conda environment named test_env

• Shortcut command

  rmenv test_env
  

• Manually typing out the entire command

  conda env remove -n test_env && rm -rf $(conda info --base)/envs/test_env

The shortcut has 80.8% fewer characters!

Installation

1. Verify that conda is installed

   conda --version
   

2. Ensure conda is up to date

   conda update conda
   

3. Enter the directory you want algae-detection to be cloned in

   • POSIX

     cd ~/path/to/directory

   • Windows

     cd C:\Users\user\path\to\directory

4. Clone and enter algae-detection

   git clone https://github.com/lynkos/algae-detection.git && cd algae-detection

5. Create virtual environment from environment.yml

   conda env create -f environment.yml

Quick Start

1. Activate algae_env (i.e., virtual environment)

   conda activate algae_env

2. Confirm algae_env is active
   • algae_env should be in parentheses () or brackets [] before your command prompt, e.g.

     (algae_env) $

   • See which virtual environments are available and/or currently active (active environment denoted with asterisk (*))

     conda info --envs

     OR

     conda env list

3. Run camera.py

Important

Automatically uses computer's default camera (i.e., webcam). To use different cameras:

• ESP32-CAM
  • Refer to ESP32-CAM tutorial in manual.md
• iPhone
  • Requires macOS v13+ and iOS v16+ (see Apple's user guide for further details)
  • Connect iPhone to Mac via USB before following Step #3
  • Run camera.py with argument --cam 1
  • Connected iPhone

    

User Interface

Users can view live footage from the camera. Detected algae are annotated with a bounding box, predicted class/category, and the model's confidence. Trackbars allow user to configure detection model attributes in real time.

See Command Line Arguments table in manual.md for all possible arguments!

• POSIX
  

  python src/detection/camera.py

• Windows
  

  python src\detection\camera.py

4. Press Escape on your keyboard to terminate

Additional Information

• ESP32CAM-RTSP (credit: rzeldent)

Appendix

• Boards
• Diagrams
• Customization
  • Custom Dataset
  • ESP32-CAM
• Inference Deployed Model
• Future Work
• Further Reading
• Glossary

Manual

• Command Line Arguments
• ESP32-CAM
  • Initial Setup
  • Usage
• Train, Validate, and Test Model
• Select Model

Credits

Special thanks to:

• Dr. Antao Chen (product owner) for his mentorship
• rdgbrian (Fall 2023 team lead) for his assistance
• rzeldent for ESP32CAM-RTSP, which has been slightly modified and added as a git subtree in streaming