Getting up and running with the basic Stroop task (deprecated)
Perhaps the best way to learn about psiTurk is to go through the steps of configuring and running an experiment. This tutorial will take you through the steps required to run the basic Stroop experiment that ships default with psiTurk. This project can be a great starting place for developing your own experiment.
This guide assumes that you have already successfully installed psiTurk.
The Stroop effect is the finding that people show interference from reading while naming the font color of words. The task is used to suggest that reading has become a highly "automatic" cognitive skill. You can read more about the Stroop task here. This guide won't comment much on the psychology of it, rather focusing on the technical aspect of running such an experiment online that consists of a sequence of trials and which records response time and keypresses.
The first step is to obtain the archive of code and resources specific to the Stroop demo. For most experiments you can download an archive from the psiTurk experiment exchange. However, the stroop demo comes bundled within the psiturk command line tool.
First create a new folder on your computer to hold the files:
$ mkdir stroop
$ cd stroopNext use the psiturk-setup-example command to place fresh copies of the files into this folder:
$ psiturk-setup-exampleafterward you should have the following listing of files in your project's directory:
$ ls -la
total 16
drwxr-xr-x 7 gureckis staff 238 Oct 18 22:05 .
drwx------ 47 gureckis staff 1598 Oct 18 22:05 ..
-rw-r--r-- 1 gureckis staff 748 Oct 18 22:05 config.txt
drwxr-xr-x 7 gureckis staff 238 Oct 18 22:05 static
drwxr-xr-x 14 gureckis staff 476 Oct 18 22:05 templatesNext, following the psiTurk philosophy, begin to "replay" the experiment by running psiturk in the current folder:
$ psiturk
This should launch the default web browser on your computer and open the dashboard.
If this is your first time running this experiment, you will be asked to enter your AWS credentials (i.e., your personal access key and secret access key):
If you are not planning to run this experiment with actual AMT workers, you can skip this step for now by pressing "Proceed without login." However, before you try to run the experiment or test the code on Amazon Mechanical Turk you will need to correctly fill out this information.
The dashboard provide many useful configuration options and provides information about your current account balance with Amazon, your currently posted hits, etc... See our overview of the psiTurk dashboard for a general introduction to the dashboard features and how it helps you. Mostly the way you configure things is intuitive (simply edit a field and press save).
In terms of configuration, all options are also directly editable in plain text via the config.txt file which was created when you first ran psiturk-setup-example. If you decide to edit this way, it is good practice to shutdown the dashboard by pressing cntl-c in the terminal before editing the config file to avoid conflicts.
The default config.txt file is already mostly configured to help you test the Stoop demo. The only things you should adjust to begin with are:
- Enter your AWS info on the "AWS Info" tab if you didn't when first launching the dashboard
- On the "Server" tab ensure that the port listed is one that is available on your computer (answer is usually yes unless you have particular firewall software running)
The psiTurk dashboard provides a graphical interface to monitor how your experiment/data collection is going. In addition, it allows you to start and stop the "experiment server." This is the process which actually provides your experiment to Internet users (i.e., AMT workers). You can start and stop this server process at any time within the dashboard. The indicator in the top right of the dashboard lets you know when the server is running and allows you to start/stop it. Typically you want to have the server running when you are testing code, testing on the AMT "sandbox", or running your actual experiment. If the server stops when running your actual experiment, Internet users will no longer be able to participate in your experiment even if you still have HITs posted on AMT's website. Thus, you should think of the experiment server as meaning you experiment is current "live."
Important When you make changes to the configuration options, you need to restart the experiment server so those changes take effect. This applies to changes you make either using the dashboard or if you are editing directly using config.txt.
Just to test, try starting/stopping the experiment server a few times in the dashboard to get the hang of it.
If you start the experiment server, then exit out of the dashboard (e.g., 'cntl-c' to kill the psiturk command in your terminal, or close the browser window, etc...) this will NOT kill the experiment server process. You must manually do that by pressing the stop button in the dashboard. This is good because it means you can leave things running in the background of your computer even if the dashboard isn't open.
After you have experimented with this, leave the experiment server running and continue.
Frequently you would like to test your experiment in your browser locally without involving Amazon's servers at all. To do so, ensure that the experiment server is running (green indicator light in the dashboard). Then click on the "Dashboard" tab. In the upper right corner of that tab is a button that says "Test". If you click this button, a new window will open in your browser pointing at your current experiment.
The URL string for this will look something like this:
http://localhost:22362/mturk?assignmentId=debug1382500678820&hitId=debug1382500678820&workerId=debug1382500678820
The http://localhost:22362/mturk part is set in the configuration options under "HIT Config" in the field "Ad URL". The default value, http://localhost:22362/mturk is a special term that refer to your own computer. If you wanted to run this experiment publically you would want to change this option to the public hostname or ip address of your computer. For example, if your laptop or desktop computer had a static ip address of research.uni.edu, you'd want to change the "Ad URL" to http://research.uni.edu/mturk.
The remaining part of the URL created random (i.e., fake) identifiers which stand-in for the values that Amazon provides identifying the user, hit, etc... Since by default psiTurk does not allow individuals to take the same experiment more than once (it checks for you to see if the worker has already completed the task or read too far into the instructions) these random values are helpful during debugging.
The first page that you see in the experiment looks something like this:
This is the page the AMT worker would see when they first accept the hit. When you click the link on that page to being, a full screen window will open up which will run the experiment. You can test it now if you like just to get a sense of things. If you want to stop midway through that is no problem. Just close that browser window. Press the "Test" button again will open a new browser window and let you repeat the process.
- Launch in sandbox (for debugging)
- Check HIT at workersandbox.mturk.com
- Run self in experiment
- Export data in dashboard
- Show how to pay self
- Finally, show how to run live on AMT