Home
Welcome to the Semantic-Parsing-of-picture-descriptions wiki!
In this interactive language learning game, you will teach the computer a language of your choice.
You will see pictures displaying simple geometric forms of different colours and shapes. Your task is to describe a part of the picture and the computer has to guess which of the objects of the picture you describe. However, at the beginning the computer doesn't know any words of your language (or any language at all). Therefore, you have to teach it your language by providing feedback such that the computer can learn from you.
Your goal for this game is that the computer becomes better and better in guessing the correct answer during the progress of the game.
When starting the game (= running gui_simple_floating.py) you will be asked to enter a name for your session and the program will then create a new folder with your chosen name in the source code directory. In this folder the result data from your game will be stored.
After choosing the name for your result folder the basic instructions will be presented to you and you can start the actual game with the first picture.
You will always see one picture at a time, with an increasing number of objects, and enter your description in the field below the picture.
Please describe the picture in only one sentence and don't use negation.
In the first level, please describe the picture only in a very short sentence mentioning only the shape of an object or the total number of objects / number of objects of the same shape.
In Level 2 you can extend your description by referring to the color of the blocks.
In Level 3 on you can (and should) increase the length and complexity of your descriptions and describe blocks based on their relation to each other and using conjunction but without colors.
In the last Level 4, you can freely choose how to describe the pictures based on number of objects, shapes, colours, relative positions and conjunctions.
Note: As the computer knows nothing about language at the beginning it will sometimes need some time to process your description. Please be patient, it is still learning :)
Example descriptions:
- a triangle
- a blue rectangle
- two green blocks
- a triangle over a circle
- a green circle under a triangle
- a yellow square and a blue rectangle
- a red circle to the right of two green forms
Note: Although all example sentences here are given in English in order to illustrate what kind of descriptions the model is able to handle, you can in principle teach it any language you like. However, please don't change the language within one game.
After entering your description and clicking Enter the computer will guess which block(s) you described most likely given what it has learnt so far and mark them with a black frame. If the guessed blocks match your description give the computer positive feedback by clicking on the YES button. If it guessed wrongly, choose the NEXT button and the computer will make another guess until making the right one. If you like to see the previous guess again, you can click BACK
In the case that you accidentally entered a sentence that does not describe the picture you can use the SKIP button to go on with a new picture instead.
Note: it is possible that your description matches more blocks then the ones you intended to describe. For example, if your description is a red triangle and a yellow circle and the picture displays two red triangles and two yellow circles. If the computer then marks all red triangles and all yellow circles you should accept this guess as correct because it matches your description.
Correct guess for a blue circle over a yellow circle
Correct guess for a blue circle over one yellow circle
Correct guess for a blue circle over two yellow circles
