Too soon to tell if the US intelligence community prediction market is more accurate than intelligence reports: Commentary on Stastny and Lehner (2018)
Journal Title: Judgment and Decision Making - Year 2019, Vol 14, Issue 3
Abstract
Stastny and Lehner (2018) reported a study comparing the forecast accuracy of a US intelligence community prediction market (ICPM) to traditionally produced intelligence reports. Five analysts unaffiliated with the intelligence reports imputed forecasts from the reports after stating their personal forecasts on the same forecasting questions. The authors claimed that the accuracy of the ICPM was significantly greater than that of the intelligence reports and suggest this may have been due to methods that harness crowd wisdom. However, additional analyses conducted here show that the imputer’s personal forecasts, which were made individually, were as accurate as ICPM forecasts. In fact, their updated personal forecasts (made after reading the intelligence reports) were marginally more accurate than ICPM forecasts. Imputed forecasts are also strongly correlated with the imputers’ personal forecasts, casting doubt on the degree to which the imputation was in fact a reliably inter-subjective assessment of what intelligence reports implied about the forecasting questions. Alternative methods for comparing intelligence community forecasting methods are discussed.
Authors and Affiliations
David R. Mandel
Keywords
Related Articles
- EP ID EP678416
- DOI -
- Views 135
- Downloads 0
It pays to be nice, but not really nice: Asymmetric reputations from prosociality across 7 countries
Cultures differ in many important ways, but one trait appears to be universally valued: prosociality. For one’s reputation, around the world, it pays to be nice to others. However, recent research with American participa...
“Leaving it to chance”—Passive risk taking in everyday life
While risk research focuses on actions that put people at risk, this paper introduces the concept of “passive risk”—risk brought on or magnified by inaction. We developed a scale measuring personal tendency for passive r...
Risky Decision Making: Testing for Violations of Transitivity Predicted by an Editing Mechanism
Transitivity is the assumption that if a person prefers A to B and B to C, then that person should prefer A to C. This article explores a paradigm in which Birnbaum, Patton and Lott (1999) thought people might be systema...
Maximizing and customer loyalty: Are maximizers less loyal?
Despite their efforts to choose the best of all available solutions, maximizers seem to be more inclined than satisficers to regret their choices and to experience post-decisional dissonance. Maximizers may therefore be...
Cognitive reflection predicts the acceptance of unfair ultimatum game offers
In the ultimatum game, one player proposes a split of money between him- or herself and another player, who can accept the offer (and both players keep the allocated money) or reject the offer (and both players get nothi...