A response to Mandel’s (2019) commentary on Stastny and Lehner (2018)
Journal Title: Judgment and Decision Making - Year 2019, Vol 14, Issue 5
Abstract
Stastny and Lehner (2018) compared the accuracy of forecasts in an intelligence community prediction market to comparable forecasts in analysis reports prepared by groups of professional intelligence analysts. To obtain quantitative probabilities from the analysis reports experienced analysts were asked to read the reports and state what probability they thought the reports implied for each forecast question. These were called imputed probabilities. Stastny and Lehner found that the prediction market was more accurate than the imputed probabilities and concluded that this was evidence that the prediction market was more accurate than the analysis reports. In a commentary, Mandel (2019) took exception to this interpretation. In a re-analysis of the data, Mandel found a very strong correlation between readers’ personal and imputed probabilities. From this Mandel builds a case that the imputed probabilities are little more than a reflection of the readers’ personal views; that they do not fairly reflect the contents of the analysis reports; and therefore, any accuracy results are spurious. This paper argues two points. First, the high correlation between imputed and personal probabilities was not evidence of substantial imputation bias. Rather it was the natural by-product of the fact that the imputed and personal probabilities were both forecasts of the same events. An additional analysis shows a much lower level of imputation bias that is consistent with the original results and interpretation. Second, the focus of Stastny and Lehner (2018) was on the reports as understood by readers. In this context, even if there was substantial imputation bias it would not invalidate accuracy results; it would instead provide a possible causal explanation of those results.
Authors and Affiliations
Paul Lehner and Bradley Stastny
Keywords
Related Articles
- EP ID EP678442
- DOI -
- Views 114
- Downloads 0
Gambler’s fallacy, hot hand belief, and the time of patterns
The gambler’s fallacy and the hot hand belief have been classified as two exemplars of human misperceptions of random sequential events. This article examines the times of pattern occurrences where a fair or biased coin...
Belief in the unstructured interview: The persistence of an illusion
Unstructured interviews are a ubiquitous tool for making screening decisions despite a vast literature suggesting that they have little validity. We sought to establish reasons why people might persist in the illusion th...
Herbert Simon’s spell on judgment and decision making
How many judgment and decision making (JDM) researchers have not claimed to be building on Herbert Simon’s work? We identify two of Simon’s goals for JDM research: He sought to understand people’s decision processes—the...
The impact of regret and worry on the threshold level of concern for flood insurance demand: Evidence from Dutch homeowners
It has been argued that individuals behave according to a threshold level of concern decision rule when considering protection against risk: if the perceived probability of the risk is below a threshold level, then the l...
Don’t stop thinking about tomorrow: Individual differences in future self-continuity account for saving
Some people find it more difficult to delay rewards than others. In three experiments, we tested a “future self-continuity” hypothesis that individual differences in the perception of one’s present self as continuous wit...