Hand-E

A robotic arm controlled via a Python script running on a microcontroller. Uses OpenCV & Mediapipe. Based on Tony Stark's Dum-E!

History

The project was inspired by Tony Stark's Dum-E, an assistive robotic arm. Originally, the plan was to use OpenCV and Mediapipe in Python with the TVS to relay information to an external microcontroller like an Arduino or RaspberryPi which controls servos. However, I quickly ran into issues with depth perception & the nature of the distance-dependent actuation.

In light of this, I'm currently working on engineering a solution that requires only one camera. I've created a port-based script which sends information gathered by OpenCV on Python to Unity which recreates the nodes & connections of the hand to make a digital hand capable of interacting with the virtual world. This includes full xyz-axis manipulation as well as distance-independent scaling.

Open Palm	Balled Fist

Authors

• @aryan-cs

Contributing

Contributions are always welcome! Please fork the repository and create a pull request to contribute. Feel free to add yourself as an author in your contribution.

Possible Contributions

For beginners, I'd recommend trying to work with the TVS-SCDP script and implementing a feature that accounts for the distance between the hand & camera to determine the distance between the thumb & index fingers (or, for that matter, any two arbitrarily chosen nodes). This can (mathematically) be done using some basic trigonometry. See below.

Where C represents the camera, A represents the index finger, and b represents the thumb.

Other possible contributions include...

• Adding documentation
• Spelling or grammar errors
• Finding a way to implement the TVS in tvs.py
• Optimizing code for efficiency
• Implementing a second camera for depth perception in tvs.py
• Implementing a Unity asset for the hand
• Anything meaningful you believe the project could benefit from!