Running experiments on Amazon Mechanical Turk
Journal Title: Judgment and Decision Making - Year 2010, Vol 5, Issue 5
Abstract
Although Mechanical Turk has recently become popular among social scientists as a source of experimental data, doubts may linger about the quality of data provided by subjects recruited from online labor markets. We address these potential concerns by presenting new demographic data about the Mechanical Turk subject population, reviewing the strengths of Mechanical Turk relative to other online and offline methods of recruiting subjects, and comparing the magnitude of effects obtained using Mechanical Turk and traditional subject pools. We further discuss some additional benefits such as the possibility of longitudinal, cross cultural and prescreening designs, and offer some advice on how to best manage a common subject pool.
Authors and Affiliations
Gabriele Paolacci, Jesse Chandler and Panagiotis G. Ipeirotis
Keywords
Related Articles
- EP ID EP677756
- DOI -
- Views 127
- Downloads 0
The role of representation in experience-based choice
Recently it has been observed that different choices can be made about structurally identical risky decisions depending on whether information about outcomes and their probabilities is learned by description or from expe...
Post-decision consolidation and distortion of facts
Participants decided whom of two patients to prioritize for surgery in three studies. The factual quantitative information about the patients (e.g., probability of surviving surgery) was given in vignette form with case...
Age-related differences in adaptive decision making: Sensitivity to expected value in risky choice
While previous research has found that children make more risky decisions than their parents, little is known about the developmental trajectory for the ability to make advantageous decisions. In a sample of children, 5-...
The Drift Diffusion Model can account for the accuracy and reaction time of value-based choices under high and low time pressure
An important open problem is how values are compared to make simple choices. A natural hypothesis is that the brain carries out the computations associated with the value comparisons in a manner consistent with the Drift...
Measuring the relative contributions of rule-based and exemplar-based processes in judgment: Validation of a simple model
Judgments and decisions can rely on rules to integrate cue information or on the retrieval of similar exemplars from memory. Research on exemplar-based processes in judgment has discovered several task variables influenc...