Testing the ability of the surprisingly popular method to predict NFL games
Journal Title: Judgment and Decision Making - Year 2018, Vol 13, Issue 4
Abstract
We consider the recently-developed “surprisingly popular” method for aggregating decisions across a group of people (Prelec, Seung and McCoy, 2017). The method has shown impressive performance in a range of decision-making situations, but typically for situations in which the correct answer is already established. We consider the ability of the surprisingly popular method to make predictions in a situation where the correct answer does not exist at the time people are asked to make decisions. Specifically, we tested its ability to predict the winners of the 256 US National Football League (NFL) games in the 2017–2018 season. Each of these predictions used participants who self-rated as “extremely knowledgeable” about the NFL, drawn from a set of 100 participants recruited through Amazon Mechanical Turk (AMT). We compare the accuracy and calibration of the surprisingly popular method to a variety of alternatives: the mode and confidence-weighted predictions of the expert AMT participants, the individual and aggregated predictions of media experts, and a statistical Elo method based on the performance histories of the NFL teams. Our results are exploratory, and need replication, but we find that the surprisingly popular method outperforms all of these alternatives, and has reasonable calibration properties relating the confidence of its predictions to the accuracy of those predictions.
Authors and Affiliations
Michael D. Lee, Irina Danileiko and Julie Vi
Keywords
Related Articles
- EP ID EP678364
- DOI -
- Views 120
- Downloads 0
Bracketing effects on risk tolerance: Generalizability and underlying mechanisms
Research has shown that risk tolerance increases when multiple decisions and associated outcomes are presented together in a broader “bracket” rather than one at a time. The present studies disentangle the influence of p...
A statistical test of independence in choice data with small samples
This paper develops tests of independence and stationarity in choice data collected with small samples. The method builds on the approach of Smith and Batchelder (2008). The technique is intended to distinguish cases whe...
Subjective but not objective numeracy influences willingness to pay for BRCA1/2 genetic testing
A positive test result for BRCA1/2 gene mutation is a substantial risk factor for breast and ovarian cancer. However, testing is not always covered by insurance, even for high risk women. Variables affecting willingness...
Image Theory’s counting rule in clinical decision making: Does it describe how clinicians make patient-specific forecasts?
The field of clinical decision making is polarized by two predominate views. One holds that treatment recommendations should conform with guidelines; the other emphasizes clinical expertise in reaching case-specific judg...
What have I just done? Anchoring, self-knowledge, and judgments of recent behavior
Can numerical anchors influence people’s judgments of their own recent behavior? We investigate this question in a series of six studies. In Study 1, subjects’ judgments of how many anagrams they were given assimilated t...