Kinectus

Kinectus is an image recognition application. It is capable of recognizing the hand-signs rock, paper, scissors within an image. The algorithm is based on an experience/reference model and manual categorisation. The feature-vector is composed of the following geometric measurements:

• circularity : 4pi * surface / perimeter ^ 2
• ellipticity : shortest axis / longest axis
• convexivity : convex surface / effective surface

In order to create a reference database, the application allows to store measurements for each hand-sign. Collectively, these measurements form clouds of points in 3d-space as shown below.

Each cloud's geometric centre (centroid) is used as a reference point for hand-sign recognition. The below gif shows the application in action, recognizing hand-signs by finding the closest match by means of the euclidian distance.

Prerequisites

The application needs the following programs to be installed:

• Kinect for Windows SDK v1.8
• Kinect for Windows Developer Toolkit v1.8
• python 2.7.14 (tick the option "add to PATH")
• Microsoft Visual C++ Compiler for Python 2.7 (in order to compile scikit-image)

Unfortunately, the C++ compiler is missing a header file. For convenience it is included in this repository at assets/stdint.h. Copy it to the directory %USERPROFILE%\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\include\ as suggested by this answer on stackoverflow.

Installation

It is recommended to install the dependencies in a python virtual environnement

pip install virtualenvwrapper-win
mkvirtualenv --python path/to/python-2.7.14/python.exe kinectus
cd path/to/kinectus
pip install numpy # required by scikit-image
pip install -r requirements.txt

The module pykinect@2.1 contains an error. To correct it, change the file Lib/site-packages/pykinect/nui/struct.py on line 213: replace return desc.height.value by return desc.height. An issue has already been created.

Usage

The application has two entrypoints: kinectus.py and evaluate.py.

Create a reference database

In order to record samples the camera Kinect XBOX 360 was used. Connect it and run the following script:

python kinectus.py

A pygame window opens up and shows the current video stream. It is filtered to only show objects at a close distance of about 50cm. In order to store the measurements, use the given keyboard shortcuts. Show the desired hand-sign in front of the camera and hit the corresponding key (the original version is french, hence no good matches):

• p for rock
• c for scissors
• f for paper Data is stored to ./assets/samples.csv.

Evaluation

The second script allows to show the dot-clouds previously stored to disk in a 3d-coordinate system.

python evaluate.py 

In order to load a different dataset, you currently need to modify the the main method or replace the file assets/samples.csv.