How generalizable is good judgment? A multi-task, multi-benchmark study
Journal Title: Judgment and Decision Making - Year 2017, Vol 12, Issue 4
Abstract
Good judgment is often gauged against two gold standards – coherence and correspondence. Judgments are coherent if they demonstrate consistency with the axioms of probability theory or propositional logic. Judgments are correspondent if they agree with ground truth. When gold standards are unavailable, silver standards such as consistency and discrimination can be used to evaluate judgment quality. Individuals are consistent if they assign similar judgments to comparable stimuli, and they discriminate if they assign different judgments to dissimilar stimuli. We ask whether “superforecasters”, individuals with noteworthy correspondence skills (see Mellers et al., 2014) show superior performance on laboratory tasks assessing other standards of good judgment. Results showed that superforecasters either tied or out-performed less correspondent forecasters and undergraduates with no forecasting experience on tests of consistency, discrimination, and coherence. While multifaceted, good judgment may be a more unified than concept than previously thought.
Authors and Affiliations
Barbara A. Mellers, Joshua D. Baker, Eva Chen, David R. Mandel and Philip E. Tetlock
Keywords
Related Articles
- EP ID EP678300
- DOI -
- Views 154
- Downloads 0
Cognition in the woods: Biases in probability judgments by search and rescue planners
A type of emergency decision-making which has not received research attention is the police search for a lost person in a rural or wilderness area. For many such incidents, decisions concerning where to search for the lo...
Too soon to tell if the US intelligence community prediction market is more accurate than intelligence reports: Commentary on Stastny and Lehner (2018)
Stastny and Lehner (2018) reported a study comparing the forecast accuracy of a US intelligence community prediction market (ICPM) to traditionally produced intelligence reports. Five analysts unaffiliated with the intel...
Two sides of the same coin: Information processing style and reverse biases
This paper examines the effect of information processing styles (indexed by the Rational-Experiential Inventory of Pacini & Epstein, 1999) on adherence to bias judgments, and particularly to reverse biases; i.e., when tw...
Making good cider out of bad apples — Signaling expectations boosts cooperation among would-be free riders
The present study investigates how group-cooperation heuristics boost voluntary contributions in a repeated public goods game. We manipulate two separate factors in a two-person public goods game: i) group composition (S...
Measuring Risk Literacy: The Berlin Numeracy Test
We introduce the Berlin Numeracy Test, a new psychometrically sound instrument that quickly assesses statistical numeracy and risk literacy. We present 21 studies (n=5336) showing robust psychometric discriminability acr...