Testing the ability of the surprisingly popular method to predict NFL games
Journal Title: Judgment and Decision Making - Year 2018, Vol 13, Issue 4
Abstract
We consider the recently-developed “surprisingly popular” method for aggregating decisions across a group of people (Prelec, Seung and McCoy, 2017). The method has shown impressive performance in a range of decision-making situations, but typically for situations in which the correct answer is already established. We consider the ability of the surprisingly popular method to make predictions in a situation where the correct answer does not exist at the time people are asked to make decisions. Specifically, we tested its ability to predict the winners of the 256 US National Football League (NFL) games in the 2017–2018 season. Each of these predictions used participants who self-rated as “extremely knowledgeable” about the NFL, drawn from a set of 100 participants recruited through Amazon Mechanical Turk (AMT). We compare the accuracy and calibration of the surprisingly popular method to a variety of alternatives: the mode and confidence-weighted predictions of the expert AMT participants, the individual and aggregated predictions of media experts, and a statistical Elo method based on the performance histories of the NFL teams. Our results are exploratory, and need replication, but we find that the surprisingly popular method outperforms all of these alternatives, and has reasonable calibration properties relating the confidence of its predictions to the accuracy of those predictions.
Authors and Affiliations
Michael D. Lee, Irina Danileiko and Julie Vi
Keywords
Related Articles
- EP ID EP678364
- DOI -
- Views 176
- Downloads 0
Maximizing as a predictor of job satisfaction and performance: A tale of three scales
Research on individual differences in maximizing (versus satisficing) has recently proliferated in the Judgment and Decision Making literature, and high scores on this construct have been linked to lower life satisfactio...
Risk communication with pictographs: The role of numeracy and graph processing
We conducted three studies to investigate how well pictographs communicate medical screening information to persons with higher and lower numeracy skills. In Study 1, we conducted a 2 (probability level: higher vs. lower...
Proportion dominance in valuing lives: The role of deliberative thinking
Proportion responding (PR) is the preference for proportionally higher gains, such that the same absolute quantity is valued more as the reference group decreases. This research investigated this kind of proportion PR in...
The effects of behavioral and outcome feedback on prudent decision-making under conditions of present and future uncertainty
One of the largest reasons decision-makers make bad decisions (act imprudently) is that the world is full of uncertainty, we feel uncertain about the consequences of our actions. Participants played a repeated game in wh...
Commitment-enhancing tools in Centipede games: Evidencing European–Japanese differences in trust and cooperation
Theories of trust distinguish general trust in situations of social uncertainty from assurance-based trust in committed, long-term relationships. This study investigates European-Japanese differences using the Centipede...