This program uses the programming language Python for video processing and hand gesture recognition as well as the visual programming environment Pure Data for audio processing. This is based on three main Python libraries:
- OpenCV for video processing
- Mediapipe to build world-class ML solutions and applications
- Open Sound Control for server and client implementations
This uses MediaPipe Hands as a high-fidelity hand and finger tracking solution based on machine learning (ML) to infer 21 3D landmarks of a hand from just a single frame. That way, the user can use the webcam's computer and the index finger to process audio data. Moreover, using Open Sound Control, it possible to send UDP packets from Python to Pure Data. That way, the hand gestures are used as controller for multi-parameter audio effects (pitch, reverb, delay and modulation).
Download Anaconda and prepare your environment using the command line
conda create --name webcam
conda activate webcam
Use the package manager pip to install the required libraires
conda install -c anaconda pip
pip install -r requirements.txt
Download and install Pure Data on your computer and download the repository. Open a terminal in the repository folder and run the following command line.
python gesture_recognition.py