The collective intelligence of random small crowds: A partial replication of Kosinski et al. (2012)
Journal Title: Judgment and Decision Making - Year 2019, Vol 14, Issue 1
Abstract
We examined the trade-off between the cost of response redundancy and the gain in output quality on the popular crowdsourcing platform Mechanical Turk, as a partial replication of Kosinski et al. (2012) who demonstrated a significant improvement in performance by aggregating multiple responses through majority vote. We submitted single items from a validated intelligence test as Human Intelligence Tasks (HITs) and aggregated the responses from “virtual groups” consisting of 1 to 24 workers. While the original study relied on resampling from a relatively small number of responses across a range of experimental conditions, we randomly and independently sampled from a large number of HITs, focusing only on the main effect of group size. We found that – on average – a group of six MTurkers has a collective IQ one standard deviation above the mean for the general population, thus demonstrating a “wisdom of the crowd” effect. The relationship between group size and collective IQ was characterised by diminishing returns, suggesting moderately sized groups provide the best return on investment. We also analysed performance of a smaller subset of workers who had each completed all 60 test items, allowing for a direct comparison between a group’s collective IQ and the individual IQ of its members. This demonstrated that randomly selected groups collectively equalled the performance of the best-performing individual within the group. Our findings support the idea that substantial intellectual capacity can be gained through crowdsourcing, contingent on moderate redundancy built into the task request.
Authors and Affiliations
Ans Vercammen, Yan Ji and Mark Burgman
Keywords
Related Articles
- EP ID EP678400
- DOI -
- Views 145
- Downloads 0
Biased calculations: Numeric anchors influence answers to math equations
People must often perform calculations in order to produce a numeric estimate (e.g., a grocery-store shopper estimating the total price of his or her shopping cart contents). The current studies were designed to test whe...
Modeling sequential context effects in judgment analysis:
In this article a broad perspective incorporating elements of time series theory is presented for conceptualizing the data obtained in multi-trial judgment experiments. Recent evidence suggests that sequential context ef...
Partner selection supported by opaque reputation promotes cooperative behavior
Reputation plays a major role in human societies, and it has been proposed as an explanation for the evolution of cooperation. While the majority of previous studies equates reputation with a transparent and complete his...
Glad to be sad, and other examples of benign masochism
We provide systematic evidence for the range and importance of hedonic reversals as a major source of pleasure, and incorporate these findings into the theory of benign masochism. Twenty-nine different initially aversive...
Aging and choice: Applications to Medicare Part D
We examined choice behavior in younger versus older adults using a medical decision-making task similar to Medicare Part D. The study was designed to assess age differences in choice processes in general and specifically...