Using cognitive models to combine probability estimates
Journal Title: Judgment and Decision Making - Year 2014, Vol 9, Issue 3
Abstract
We demonstrate the usefulness of cognitive models for combining human estimates of probabilities in two experiments. The first experiment involves people’s estimates of probabilities for general knowledge questions such as “What percentage of the world’s population speaks English as a first language?” The second experiment involves people’s estimates of probabilities in football (soccer) games, such as “What is the probability a team leading 1–0 at half time will win the game?”, with ground truths based on analysis of large corpus of games played in the past decade. In both experiments, we collect people’s probability estimates, and develop a cognitive model of the estimation process, including assumptions about the calibration of probabilities and individual differences. We show that the cognitive model approach outperforms standard statistical aggregation methods like the mean and the median for both experiments and, unlike most previous related work, is able to make good predictions in a fully unsupervised setting. We also show that the parameters inferred as part of the cognitive modeling, involving calibration and expertise, provide useful measures of the cognitive characteristics of individuals. We argue that the cognitive approach has the advantage of aggregating over latent human knowledge rather than observed estimates, and emphasize that it can be applied in predictive settings where answers are not yet available.
Authors and Affiliations
Michael D. Lee and Irina Danileiko
Keywords
Related Articles
- EP ID EP678127
- DOI -
- Views 164
- Downloads 0
Assessing the sensitivity of information distortion to four potential influences in studies of risky choice
The emergence of a leading alternative during the course of a decision is known to bias the evaluation of new information in a manner that favors that alternative. We report 3 studies that address the sensitivity of pred...
Biased calculations: Numeric anchors influence answers to math equations
People must often perform calculations in order to produce a numeric estimate (e.g., a grocery-store shopper estimating the total price of his or her shopping cart contents). The current studies were designed to test whe...
Numerate decision makers don’t use more effortful strategies unless it pays: A process tracing investigation of skilled and adaptive strategy selection in risky decision making
The present study investigated skilled and adaptive strategy selection in risky decision making. We proposed that people with high objective numeracy, a strong predictor of general decision making skill, would have a bro...
Pair-wise comparisons of multiple models
Often research in judgment and decision making requires comparison of multiple competing models. Researchers invoke global measures such as the rate of correct predictions or the sum of squared (or absolute) deviations o...
Stepwise training supports strategic second-order theory of mind
People model other people’s mental states in order to understand and predict their behavior. Sometimes they model what others think about them as well: “He thinks that I intend to stop.” Such second-order theory of mind...