Boosting intelligence analysts’ judgment accuracy: What works, what fails?
Journal Title: Judgment and Decision Making - Year 2018, Vol 13, Issue 6
Abstract
A routine part of intelligence analysis is judging the probability of alternative hypotheses given available evidence. Intelligence organizations advise analysts to use intelligence-tradecraft methods such as Analysis of Competing Hypotheses (ACH) to improve judgment, but such methods have not been rigorously tested. We compared the evidence evaluation and judgment accuracy of a group of intelligence analysts who were recently trained in ACH and then used it on a probability judgment task to another group of analysts from the same cohort that were neither trained in ACH nor asked to use any specific method. Although the ACH group assessed information usefulness better than the control group, the control group was a little more accurate (and coherent) than the ACH group. Both groups, however, exhibited suboptimal judgment and were susceptible to unpacking effects. Although ACH failed to improve accuracy, we found that recalibration and aggregation methods substantially improved accuracy. Specifically, mean absolute error (MAE) in analysts’ probability judgments decreased by 61% after first coherentizing their judgments (a process that ensures judgments respect the unitarity axiom) and then aggregating their judgments. The findings cast doubt on the efficacy of ACH, and show the promise of statistical methods for boosting judgment quality in intelligence and other organizations that routinely produce expert judgments.
Authors and Affiliations
David R. Mandel, Christopher W. Karvetski and Mandeep K. Dhami
Keywords
Related Articles
- EP ID EP678388
- DOI -
- Views 166
- Downloads 0
High-stakes hedges are misunderstood too. A commentary on: “Valuing bets and hedges: Implications for the construct of risk preference”
Frederick, Levis, Malliaris & Meyer (2018) report a package of laboratory studies where participants underestimate the value of “hedges”: Risky bets which cancel out the risk of another presently-held bet. However, it mi...
Psychometric characteristics of two forms of the Slovak version of the Indecisiveness Scale
The study investigates the psychometric characteristics of the Slovak version of the original and short form of the Indecisiveness Scale on three samples of university students and one general population sample. An explo...
To trade or not to trade: The moderating role of vividness when exchanging gambles
Individuals are generally reluctant to trade goods—a phenomenon identified as the endowment effect. This paper focuses on consumers’ puzzling reluctance to exchange gambles, and in particular lottery tickets with identic...
Sense of control matters: a long spatial distance leads to a short-term investment preference
Increasingly, we can invest in projects that are distributed around the world through online investment platforms. Will the spatial distance between these projects and ourselves affect our investment preferences? The pre...
The power of touch: An examination of the effect of duration of physical contact on the valuation of objects
The duration of ownership has been shown to increase the valuation of items that people currently own as well as items they have owned in the past, a phenomenon termed the “length-of-ownership effect.” We hypothesize tha...