Boosting intelligence analysts’ judgment accuracy: What works, what fails?
Journal Title: Judgment and Decision Making - Year 2018, Vol 13, Issue 6
Abstract
A routine part of intelligence analysis is judging the probability of alternative hypotheses given available evidence. Intelligence organizations advise analysts to use intelligence-tradecraft methods such as Analysis of Competing Hypotheses (ACH) to improve judgment, but such methods have not been rigorously tested. We compared the evidence evaluation and judgment accuracy of a group of intelligence analysts who were recently trained in ACH and then used it on a probability judgment task to another group of analysts from the same cohort that were neither trained in ACH nor asked to use any specific method. Although the ACH group assessed information usefulness better than the control group, the control group was a little more accurate (and coherent) than the ACH group. Both groups, however, exhibited suboptimal judgment and were susceptible to unpacking effects. Although ACH failed to improve accuracy, we found that recalibration and aggregation methods substantially improved accuracy. Specifically, mean absolute error (MAE) in analysts’ probability judgments decreased by 61% after first coherentizing their judgments (a process that ensures judgments respect the unitarity axiom) and then aggregating their judgments. The findings cast doubt on the efficacy of ACH, and show the promise of statistical methods for boosting judgment quality in intelligence and other organizations that routinely produce expert judgments.
Authors and Affiliations
David R. Mandel, Christopher W. Karvetski and Mandeep K. Dhami
Keywords
Related Articles
- EP ID EP678388
- DOI -
- Views 152
- Downloads 0
A fine-grained analysis of the jumping-to-conclusions bias in schizophrenia: Data-gathering, response confidence, and information integration
Impaired decision behavior has been repeatedly observed in schizophrenia patients. We investigated several cognitive mechanisms that might contribute to the jumping-to-conclusions bias (JTC) seen in schizophrenia patient...
It must be awful for them: Perspective and task context affects ratings for health conditions
When survey respondents rate the quality of life (QoL) associated with a health condition, they must not only evaluate the health condition itself, but must also interpret the meaning of the rating scale in order to assi...
Charting the internal landscape: Affect associated with thoughts about major life domains explains life satisfaction
Studies of happiness have examined the impact of demographics, personality and emotions accompanying daily activities on life satisfaction. We suggest that how people feel while contemplating aspects of their lives, incl...
The impact of actively open-minded thinking on social media communication
Online, social media communication is often ambiguous, and it can encourage speed and inattentiveness. We investigated whether Actively Open Minded Thinking (AOT), a dispositional willingness to seek out new or potential...
Are good reasoners more incest-friendly? Trait cognitive reflection predicts selective moralization in a sample of American adults
Two studies examined the relationship between individual differences in cognitive reflection (CRT) and the tendency to accord genuinely moral (non-conventional) status to a range of counter-normative acts — that is, to t...