-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathMaking-MTurk-experiments.tex
159 lines (119 loc) · 9.81 KB
/
Making-MTurk-experiments.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
\documentclass{article}
\usepackage[margin=1in]{geometry}
\usepackage{xcolor}
\usepackage{adjustbox}
\usepackage{verbatim}
\definecolor{shadecolor}{rgb}{.9, .9, .9}
\usepackage{tcolorbox}
\usepackage[colorlinks = true,
linkcolor = blue,
urlcolor = blue,
citecolor = blue,
anchorcolor = blue]{hyperref}
\usepackage{fvextra}
\DefineVerbatimEnvironment{codewrap}{Verbatim}{breaklines,commandchars=\\\{\}}
\newenvironment{code}%
{\par\noindent\adjustbox{margin=1ex,bgcolor=shadecolor,margin=0ex \medskipamount}\bgroup\minipage\linewidth\codewrap}%
{\endcodewrap\endminipage\egroup}
%\DeclareUrlCommand{\burl}{\def\UrlFont{\ttfamily\color{blue}}}
\newcommand{\burl}[3][blue]{\href{#1}{\color{#2}{#3}}}%
\title{Programming MTurk experiments via {\em JSEXP}}
\author{Florian Jaeger with help from Linda Liu, Zach Burchill, Wednesday Bushong, and Xin Xie}
\begin{document}
\maketitle
\tableofcontents
\section{Overview}
JSEXP is a collection of Javascript programs to facilitate psychological experiments over the web. It works both for MTurk and for Prolific (via Proliferate). JSEXP let's you define blocks of different tasks (e.g., lexical decision, picture naming, transcription, similarity judgments, etc. and records participants keyboard and mouse responses and reaction times. JSEXP was originally written by Dave Kleinschmidt and then extended and modified by Zach Burchill, Wednesday Bushong, Esteban Buz, Florian Jaeger, Linda Liu, Xin Xie, and others.
Make sure you have completed the tutorials on {\em Setting up Mechanical Turk} and {\em Conducting Mechanical Turk experiments}. This tutorial walks you through how to program your own experiment using {\em \href{https://github.com/hlplab/JSEXP/}{JSEXP}}.
\section{Setup JSEXP}
Clone \href{https://github.com/hlplab/JSEXP}{JSEXP}. We recommend using \href{https://git-scm.com/book/en/v2/Git-Tools-Submodules}{git submodules} to add JSEXP to your project repository. For example, in your git repo for our projects we typically have a folder called experiments/ with subfolders for each series of experiments, e.g., experiments/series-A/. Consider cloning JSEXP into each of those subfolders.
This will allow you to pull/push the most recent updates of this package while continuing to develop your experiment-specific code. When you clone a
project that contains a submodule, remember that you need to also clone the submodule (by default the submodule folder will be cloned but
not its content). Use:
\begin{code}
git clone --recurse-submodules YOUR-PROJECT-URL
\end{code}
during the cloning to also automatically clone all submodules contained in the project. Once cloned, you can update all submodules in your project by using
\begin{code}
git submodule update --remote
\end{code}
Once updates to a submodule are committed and pushed to your project, you and your collaborators can pull these updates. Make sure to use
`git pull --recurse-submodules' if you want changes to the submodules not only to be fetched but also to be pulled.
\section{Making your own experiment}
A JSEXP experiment consist of:
\begin{itemize}
\item An HTML file that loads all the necessary Javascript (inc. JSEXP)
\item Your experiment-specific Javascript that uses JSEXP functionality.
\item the JSEXP/ repo
\item Optionally:
\begin{itemize}
\item stimulus files
\item list files
\item style (css) files
\item survey files (for post-experiment surveys)
\end{itemize}
\end{itemize}
%current idea for workflow:
%develop script and test on laptop (localhost; student): select condition, list, etc. through URL parameters
%test on slate (florian upload: test on students)
%sandbox from slate (florian)
% deploy from slate (florian)
\subsection{What needs to be in the HTML file?}
\begin{itemize}
\item A header with an optional link to a style sheet. In case you plan to run your experiment over Prolific (via Proliferate), the header also needs to load the proliferate javascript.
\begin{code}
<head>
<meta charset="UTF-8" />
<link rel="stylesheet" type="text/css" href="JSEXP/styles/style.css" />
<title>Speech perception experiment</title>
<script src="https://proliferate.alps.science/static/js/proliferate.js" type="text/javascript"></script>
</head>
\end{code}
\item A form element that posts the results to MTurk when the experiment is done. JSEXP will populate this form with hidden form elements based on a) the URL parameters you provide and b) data collected from the participants.
\begin{code}
<form id="mturk_form" method="POST" action="https://www.mturk.com/mturk/externalSubmit">
<input id="submitButton" type="submit" name="Submit" value="Submit" />
</form>
\end{code}
\item A link to whatever Javascript code you're using. See the example experiments provides as part of this repo.
\begin{code}
<!-- general javascript, incl. general definitions of objects
for experiment, block, and stimulus lists //-->
<script src="js-adapt/jquery-1.10.1.min.js" type="text/javascript" ></script>
<script src="js-adapt/modernizr.min.js" type="text/javascript"></script>
<script src="js-adapt/stimuli.js" type="text/javascript" ></script>
<script src="js-adapt/labelingBlock.js" type="text/javascript" ></script>
<script src="js-adapt/exposureBlock.js" type="text/javascript" ></script>
<script src="js-adapt/experimentControl2.js" type="text/javascript" ></script>
<script src="js-adapt/soundcheckBlock.js" type="text/javascript"></script>
<script src="js-adapt/mturk_helpers.js" type="text/javascript"></script>
<script src="js-adapt/progressBar.js" type="text/javascript"></script>
<script src="js-adapt/logreg.js" type="text/javascript"></script>
<script src="js-adapt/utilities.js" type="text/javascript"></script>
<script src="get_stim.js" type="text/javascript"></script>
<!-- Here is where your experiment javascript file is specified -->
<script src="experiment-A.js" type="text/javascript"></script>
\end{code}
\item Any HTML objects references in the Javascript code. This includes, for example, various <div>s that the Javascript code will
fill with content at different points during the experiment.
\end{itemize}
\section{Disabling CORS errors when testing locally}
If you Javascript is loading external sources like stimulus files or surveys, local testing will result in a CORS error (unless you're running a web server). You can \href{https://medium.com/swlh/avoiding-cors-errors-on-localhost-in-2020-5a656ed8cefa}{avoid CORS Errors on localhost} (in 2020) by starting Chrome the following way (it's important that the last argument contains the full path to the page you'd like to open). Note, however, that {\bf this will not actually make Javascript {\em load} those external resources.} It seems to only suppress the error.
\begin{code}
open -n -a "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" --args --user-data-dir="/tmp/chrome_dev_test" --disable-web-security /Users/tiflo/Documents/GitHub/Project1/experiments/series-A/experiment-A.html
\end{code}
\section{Calling your own experiment}
JSEXP reads in the URL parameters in the {\em search string} of the URL handed to the browser. This makes it possible to have one HTML file for many different conditions of your experiment. As soon as the URL parameters are read in, JSEXP removes them from the URL search string (except for the experiment\_id and participant\_id required for Prolific experiments through Proliferate). This avoids that the URL parameters that might, e.g., specify conditions affect participants' behavior.
\subsection{Reserved URL parameters}
The following is a list of reserved URL parameters that have a fixed interpretation in JSEXP. The values in curly parenthesis list allowed values.
\begin{itemize}
\item platform=\{mturk, prolific, proliferate\} Determines the submit method chosen at the end of the experiment. For Prolific and Proliferate, the output is converted into a JSON object that is submitted to Prolific. For Mechanical Turk, the mturk\_form in the HTML file is submitted as is. (DEFAULT: mturk)
\item debug=\{T,TRUE\}. Enter debug mode. This activates all messages, warnings, and errors that JSEXP would otherwise hide, and prints them to the Javascript console of the browser. The debug mode also suppressed the removal of the URL parameters from the URL search string, so that the experimenter can check whether those parameters look correct. Finally, the debug mode causes the script to pause (through a pop-up alert) right before the results are submitted. This allows the experimenter to read exactly what information would be submitted to the crowdsourcing platform. (DEFAULT: no debugging)
\end{itemize}
\section{Example experiments using {\em JSEXP}}
\begin{itemize}
\item The code for the unsupervised and supervised learning experiment in Kleinschmidt et. al (2015) \footnote{\href{https://mindmodeling.org/cogsci2015/papers/0200/paper0200.pdf}{https://mindmodeling.org/cogsci2015/papers/0200/paper0200.pdf}} can be cloned from \url{https://bitbucket.org/hlplab/nrtmodule/src/master/}. If you'd like to get a sense of a full version of the experiment, you can test it out at \url{https://www.hlp.rochester.edu/mturk/mtadapt/sup-unsup/}.
\item The code for a web-based version of the priming experiment from Xie \& Myers (2017) can be found at \url{https://github.com/xinxie-cogsci/Online-experiments}. If you'd like to get a sense of a full version of the experiment, you can test it out at, for example, \url{https://www.hlp.rochester.edu/mturk/xxie/io_perception/exp1/exp1_transcription.html?condition=experimental&speaker=M15&order=2&visual=1&block=2} (for information on the URL parameters, see the readme.md file in the github repository).
\item And here is a demo of a whole number of different paradigms available on request: \url{https://www.hlp.rochester.edu/mturk/lliu/demos/}.
\end{itemize}
\end{document}