Wahl-O-Selfie - v1

Description

Wahl-O-Selfie (v1) is using a self-trained machine learning model which categorizes human faces into (german) political partys similar to "Wahl-O-Mat¹".

Table of Contents

• Description
• Table of Contents
• How to install
  • Installing face_recognition (for Windows)
  • Installing TensorFlow Keras
  • Installing the other dependencies
  • Installing Wahl-O-Selfie
• How to use
• How it works
  • Machine learning model
  • Wahl-O-Selfie program
• Tests and results
  • Test 1
  • Test 2
• Problems
• License and credits
• Postscript

How to install

Wahl-O-Selfie (v1) requires:

• Python 3.10.7
• face_recognition
  • Microsoft Visual Studio 2015 (or newer)
  • CMake
  • dlib
• pillow
• TensorFlow Keras
  • Microsoft Visual C++ Redistributable
• numpy
• matplotlib
• os

Installing face_recognition (for Windows)²

To install face_recognition you have to have Python 3.10.7 and Microsoft Visual Studio 2015 (or newer) with C/C++ Compiler installed. After that you have to install CMake for Windows and add it to your system variables. Then you have to install dlib using pip:

pip install dlib

and finally you can install the face_recognition library, also by using pip:

pip install face_recognition

Installing TensorFlow Keras³

1. System requirements
   • Windows 7 or higher (64-bit)
2. Install Microsoft Visual C++ Redistributable
   • Got to the Microsoft Visual C++ downloads
   • Scroll down the page to the Visual Studio 2015, 2017 and 2019 section.
   • Download and install the Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017 and 2019 for your platform. Make sure long paths are enabled on Windows
3. Install TensorFlow
   • First, run the command:

   pip install --upgrade pip

   to ensure that you have the most recent version of pip installed on your machine.
   • Then, install TensorFlow with pip:

   pip install tensorflow

   • And finally, install keras with pip:

   pip install keras

Installing the other dependencies

Finally, you can install the other dependencies with pip:

pip install pillow numpy matplotlib

The library os is pre-installed with python.

Installing Wahl-O-Selfie

Download the project as a .zip file and unzip it on your machine. Then open main.py with a code editor, preferably with Microsoft Visual Studio Code, but make sure that you have the Python extension installed.

How to use

A few seconds after running the program a window should open up showing you a picture of a guy with a graph at the bottom, which gives you the results. In addition to that there are a few new files in the program folder, most importantly result.jpg and result2.jpg. result.jpg is just a saved version of the picture you already saw in the window that opened up earlier and result2.jpg is a different way of showing the results. Be aware that these files will be overwritten everytime you run the program, so if you see something you like copy-and-paste it to somewhere else.

If you want to use your own photo, you have to replace testimage.jpg in the program folder. Make sure, that you use the JPG format and that it is called "testimage", otherwise the program will not recognize it. Please be aware, that Wahl-O-Selfie v1 can only process images that contain one face, more or less faces will result in an error.

How it works

Machine learning model

Wahl-O-Selfie v1 is using a pre-trained Keras model which was trained with 385 pictures of 79 different german politicians from 6 political partys. The pictures used for training the machine learning model were first run through another program which cut-out only the face to make sure that the training data is as accurate as possible without any other data influencing the results.

This means that this photo:

(Picture of Felix Banaszak - Die Grünen)

is now this photo:

which is way better training data.

All of the processed pictures were then used in Teachable Machine picture model with the different party names as class names and the corresponding politicians as training data for each class. After trainig the model you can download the trained keras model. The Keras model can be found in "trained_model/keras_model.h5" with a "lables.txt", which contains all the class names and was also downloaded from TeachableMachine. "lables.txt" is not used in Wahl-O-Selfie v1, instead it was just used to get the political partys in the program in the right order as I wanted to associate every party with their corresponding color.

Every political party has around 65 pictures of 13 different politicians meaning that there are around 5 different photos per politician which was supposed to ensure that the machine learning model focuses only on the face data and not on other parts of the image, like the background color.

Wahl-O-Selfie program

After loading all the necessary librarys the program continues with loading the pre-trained keras model

model = load_model('./trained_model/keras_model.h5')

and the picture

picture = face_recognition.load_image_file('testimage.jpg')

The default-picture is a photograph made by This person does not exist.

(Picture by "https://thispersondoesnotexist.com")

The loaded picture then gets analyzed by the face_recognition library and all recognized face locations get stored in a list called "face_locations":

face_locations = face_recognition.face_locations(picture)

By using len() the program can get the amount of elements in a list, which is used to get the amount of faces in the picture. Wahl-O-Selfie v1 can only process one face per picture which is why it checks if "face_locations" has more or less elements than one. If one of these if-statements is True it prints out the corresponding error in the terminal. There are a few commented out lines in the more than one face case, which are purely for debugging:

   # for(top, right, bottom, left) in face_locations:
   #     draw = ImageDraw.Draw(pil_image)
   #     draw.rectangle(((left, top), (right, bottom)), outline = (255, 0, 0), width = 4)
   #     pil_image.save("error.jpg")
   #     del draw

When these lines are not commented out they draw red rectangles around all the recognized faces and save the edited version as "error.jpg" so that the user can see all places where the program identified a face and verify that the program works properly.

However, if there is only one face in the picture it crops out the part where the face is located, scales it to 224x224 pixel and runs it through the machine learning model to get the prediction of which party the face most likely belongs to. The predictions are small numbers which first get stored in a "prediction" variable and right after they are safed in their own variables which I named after the political partys, in the order they stand in "lables.txt".

    prediction = model.predict(data)

    AFD = prediction[0, 0]
    CDU_CSU = prediction[0, 1]
    Gruenen = prediction[0, 2]
    Linke = prediction[0, 3]
    FDP = prediction[0, 4]
    SPD = prediction[0, 5]

The prediction data then gets converted into percentages for easier handling and the coordinates of the face are used to draw a red rectangle around that area.

The program also gets the biggest value and to which party it belongs to identify which party is the most likeliest.

max = max(AFD, CDU_CSU, Gruenen, Linke, FDP, SPD)

Afterwards it prints out the predictions in percent to the terminal followed by the most likeliest party.

e.g.

AFD: 2.8%
CDU/CSU: 2.1%
Die Grünen: 0.0%
Linke: 94.5%
FDP: 0.4%
SPD: 0.1%
----------------------------------------
Die Linke

At the end it generates two forms of presenting the data in image form.

For the first version ("result.jpg") the program generates a graph to put under the photo.

    partys = [
    'AFD - ' + str(AFD_percent) + '%', 
    'CDU/CSU - ' + str(CDU_CSU_percent) + '%', 
    'Die Grünen - ' + str(Gruenen_percent) + '%', 
    'Die Linke - ' + str(Linke_percent) + '%', 
    'FDP - ' + str(FDP_percent) + '%',
    'SPD - ' + str(SPD_percent) + '%']
    colors = ("blue", "black", "green", "red", "yellow", "red")
    data = [AFD_percent, CDU_CSU_percent, Gruenen_percent, Linke_percent, FDP_percent, SPD_percent]
    fig, ax = plt.subplots()
    y_pos = np.arange(len(partys))
    ax.barh(y_pos, data, align="center", color=colors)
    ax.set_yticks(y_pos, labels=partys)
    
    ax.invert_yaxis()
    ax.set_xlabel('Procent')
    ax.set_title('Wahl-O-selfie')
    plt.savefig('graph.png', bbox_inches = "tight")
    
    w, h = pil_image.size
    if w < 653:
        img = Image.new(mode = 'RGB', size = (653, h + 455), color="white")
    else:
        img = Image.new(mode = 'RGB', size = (w, h + 455), color="white")
    img_w, img_h = img.size
    img.paste(pil_image, (round((img_w - w) / 2), 0))
    graph = Image.open('graph.png')
    img.paste(graph, (round((img_w - 653) / 2), h))
    img.save('result.jpg')

This code handles generating a graph with all the percentage values for each party and formatting it so that every bar graph has the right color corresponding to the party it is associated with, printing the corresponding party names besides each graph and putting additional text above and below the bar graph. Furthermore, there is some code which handles what should happen if the testimage is thinner or thicker than the bar graph, so that the graph is always centered.

For the second version ("result2.jpg") the program is just printing the party names with their percentages under the red square.

    text_width, text_height = draw.textsize('Die Grünen - 100.0%')
    draw.rectangle(((left,(bottom + 5) + (text_height * 5 + 5)), (right, bottom)), fill = (0, 0, 0), outline = (0, 0, 0))
    draw.text((left + 6, (bottom + 5) + (text_height * 6 - 70)), 'AFD - ' + str(AFD_percent) + '%', fill=(255, 255, 255))
    draw.text((left + 6, (bottom + 5) + (text_height * 6 - 60)), 'CDU/CSU - ' + str(CDU_CSU_percent) + '%', fill=(255, 255, 255))
    draw.text((left + 6, (bottom + 5) + (text_height * 6 - 50)), 'Die Grünen - ' + str(Gruenen_percent) + '%', fill=(255, 255, 255))
    draw.text((left + 6, (bottom + 5) + (text_height * 6 - 40)), 'Die Linke - ' + str(Linke_percent) + '%', fill=(255, 255, 255))
    draw.text((left + 6, (bottom + 5) + (text_height * 6 - 30)), 'FDP - ' + str(FDP_percent) + '%', fill=(255, 255, 255))
    draw.text((left + 6, (bottom + 5) + (text_height * 6 - 20)), 'SPD - ' + str(SPD_percent) + '%', fill=(255, 255, 255))
    pil_image.save('result2.jpg')

    os.remove('graph.png')
    img.show()

Finally, it is using the os library to delete a temporarily created "graph.png" file which contained the previously generated graph for "result.jpg" and it is showing said result.jpg.

Tests and results

Program tested in Python 3.10.7.

Test 1

For Test #1 I used different pictures from people that are in the machine learning model to test if the model learned how these people look like and if it is able to recognize them in other pictures.

Test #1	Results
Combined results
AFD
CDU/CSU
Die Grünen
Die Linke
FDP
SPD

Test 1 results:

• AFD: 5 Tested; 2 True, 3 False

  AFD: 2
  CDU/CSU: 1
  Die Grünen: 2
  Die Linke: 0
  FDP: 0
  SPD: 0

• CDU/CSU: 5 Tested; 3 True, 2 False

  AFD: 0
  CDU/CSU: 3
  Die Grünen: 1
  Die Linke: 1
  FDP: 0
  SPD: 0

• Die Grünen: 5 Tested; 4 True, 1 False

  AFD: 0
  CDU/CSU: 0
  Die Grünen: 4
  Die Linke: 0
  FDP: 1
  SPD: 0

• Die Linke: 5 Tested; 3 True, 2 False

  AFD: 0
  CDU/CSU: 1
  Die Grünen: 0
  Die Linke: 3
  FDP: 1
  SPD: 0

• FDP: 5 Tested; 5 True, 0 False

  AFD: 0
  CDU/CSU: 0
  Die Grünen: 0
  Die Linke: 0
  FDP: 5
  SPD: 0

• SPD: 5 Tested; 3 True, 2 False

  AFD: 1
  CDU/CSU: 1
  Die Grünen: 0
  Die Linke: 0
  FDP: 0
  SPD: 3

Test 1 had an overall accuracy of 66.6667% which shows that the machine learning model has a good understanding of how the people in the training data looked like and it was mostly able to sort them in their corresponding party.

Test 2

For Test #2 I used random pictures of other people in the political partys to see if the machine learning model learned which aspects were important to confidently sort faces into a political party.

Test #2	Results
Combined results
AFD
CDU/CSU
Die Grünen
Die Linke
FDP
SPD

Test 2 results:

• AFD: 5 Tested; 1 True, 4 False

  AFD: 1
  CDU/CSU: 2
  Die Grünen: 0
  Die Linke: 1
  FDP: 1
  SPD: 0

• CDU/CSU: 5 Tested; 2 True, 3 False

  AFD: 1
  CDU/CSU: 2
  Die Grünen: 0
  Die Linke: 2
  FDP: 0
  SPD: 0

• Die Grünen: 5 Tested; 2 True, 3 False

  AFD: 0
  CDU/CSU: 1
  Die Grünen: 2
  Die Linke: 1
  FDP: 1
  SPD: 0

• Die Linke: 5 Tested; 0 True, 5 False

  AFD: 1
  CDU/CSU: 4
  Die Grünen: 0
  Die Linke: 0
  FDP: 0
  SPD: 0

• FDP: 5 Tested; 0 True, 5 False

  AFD: 2
  CDU/CSU: 2
  Die Grünen: 1
  Die Linke: 0
  FDP: 0
  SPD: 0

• SPD: 5 Tested; 1 True, 4 False

  AFD: 1
  CDU/CSU: 2
  Die Grünen: 1
  Die Linke: 0
  FDP: 0
  SPD: 1

Test #2 had only an accuracy of 20% which shows that the machine learning model could not identify the reason why someone is in a party instead it could just remember that someone is in a party. This means that the machine learning model is way too small to be able to learn which face should be in which party.

Problems

Wahl-O-Selfie v1 is just a proof of concept, meaning that the dataset used to determin the political party of a face is rather small with an insufficient diversity, resulting in wrong and/or inaccurate results.

License and credits

This work is made available under the GNU Affero General Public License v3.0.

Project made by Benjamin Hupf.

Postscript

While writing this readme file I found out that something similar was actually build with a much bigger dataset which contained the self-reported political orientation of over a million participants from Facebook and dating website accounts from three countries (the U.S, the UK and Canada). This facial recognition model achieved an accuracy of around 72%. If you want to read the article yourself you can find it HERE.

Footnotes

1. The "Wahl-O-Mat" is a website by the german bpb with questions which you can fill out and afterwards you can see which political partys have similar interests. This is supposed to help people make up their mind about who they should vote for. ↩

2. https://github.com/ageitgey/face_recognition/issues/175#issue-257710508 ↩

3. https://www.tensorflow.org/install/pip ↩