What is this?

PsiTurk is an open platform for conducting custom behvioral experiments on Amazon's Mechanical Turk.

It is intended to provide most of the backend machinery necessary to run your experiment. It uses AMT's External Question HIT type, meaning that you can collect data using any website. As long as you can turn your experiment into a website, you can run it with PsiTurk!

You can direct questions to our Q&A Google group.

Dependencies

You will need to use a relatively recent version of Python 2 with the following modules installed:

• Flask – A lightweight web framework.
• SQLAlchemy – A powerful SQL abstraction layer.
• Boto – A library for interfacing with Amazon services, including MTurk.

You can install these with the following commands:

easy_install Flask-SQLAlchemy
easy_install boto

To serve your experiment to participants online, you will need to run this code from a web server connected to the internet.

Quick Start

Just follow these directions to get started:

1. Check out this repository in git, or download the whole thing using the 'ZIP' button near the top of this page.

2. Install the dependencies.

3. Sign up for an AWS account, available here.

4. Sign up for a Mechanical Turk requester account, available here.

5. Rename the config file from config.txt.example to config.txt. Update it with your secret AWS code.

6. Making sure that the configuration file is set up to use the Amazon sandbox, issue the following commands from the PsiTurk root folder:

    python mturk/createHIT.py    # To post a HIT to the sandbox
    python app.py                # To start the debugging server
   

7. You should be ready to go! Point your browser to the worker sandbox and try to find your HIT.

Note: If you are just testing the server without posting your HIT to Amazon, you can see the experiment at the following link: http://localhost:5001/mturk?assignmentId=debug&hitId=debug&workerId=debug

Experiment design

We have provided an example stroop experiment that could form the basis of your own experiment. It is a Javascript experiment, with task logic inside the participant's browser using Javascript code in static/task.js. This Javascript code works by dynamically changing the html document served to participants in templates/exp.html. PsiTurk assigns a condition and counterbalance to each participant. These are fed into JavaScript by plugging them into templates/exp.html. PsiTurk actively manages the condition and counterbalance subjects are assigned to, helping you fill them in evenly. To tell PsiTurk how many conditions and counterbalance identities are possible in your experiment, adust num_conds and num_counters in config.txt.

Deployment

Configuration

To make your experiment available on the internet, you will need to make the following changes to the configuration file:

host: 0.0.0.0
question_url: http://yoururl:yourport/mturk

replacing yoururl with the url to your surver, and yourport with the port you have configured in config.txt (by default, 5001).

Server

We strongly recommend you not deploy your experiment using the debugging server (the one you start using python app.py). It is not robust to failures, which can leave your participants stranded without a way of submitting their completed HITs. Additionally, if you accidentally leave debug mode on, you will expose yourself to major security holes.

An alternative we have set up is gunicorn. You can install gunicorn using the following command:

easy_install gunicorn

Then simply run using:

sh run_gunicorn.sh

You can configure gunicorn in config.txt under Server Parameters.

Flask apps like PsiTurk can be deployed as a CGI, fastCGI, or WSGI app on any server system, so there are many alternative options for deployment. Additional options for deploying Flask can be found here.

Database

We recommend using a deployment-robust database solution such as MySQL or PostgreSQL. SQLite does not allow concurrent access to the database, so if the locks work properly, simultaneous access (say, from multiple users submitting their data at the same time) could destabilize your database. In the worst (unlikely) scenario, the database could become corrupted, resulting in data loss.

Instructions for setting up a MySQL server on a Mac can be found in the wiki. Other platforms, check out instructions at mysql.org.

Copyright

You are welcome to use this code for personal or academic uses. If you fork, please cite the authors (Todd Gureckis and John McDonnell).