A universal method for evaluating the quality of aggregators
Journal Title: Judgment and Decision Making - Year 2019, Vol 14, Issue 4
Abstract
We propose a new method to facilitate comparison of aggregated forecasts based on different aggregation, elicitation and calibration methods. Aggregates are evaluated by their relative position on the cumulative distribution of the corresponding individual scores. This allows one to compare methods using different measures of quality that use different scales. We illustrate the use of the method by re-analyzing various estimates from Budescu and Du (Management Science, 2007).
Authors and Affiliations
Ying Han and David Budescu
Keywords
Related Articles
- EP ID EP678426
- DOI -
- Views 149
- Downloads 0
The wisdom of ignorant crowds: Predicting sport outcomes by mere recognition
The collective recognition heuristic is a simple forecasting heuristic that bets on the fact that people’s recognition knowledge of names is a proxy for their competitiveness: In sports, it predicts that the better-known...
The attraction effect in motor planning decisions
In motor lotteries the probability of success is inherent in a person’s ability to make a speeded pointing movement. By contrast, in traditional economic lotteries, the probability of success is explicitly stated. Decisi...
Limited resources or limited luck? Why people perceive an illusory negative correlation between the outcomes of choice options despite unequivocal evidence for independence
When people learn of the outcome of an option they did not choose (the alternative outcome) before they know their own outcome, they see an illusory negative correlation between the two outcomes, the Alternative Omen Eff...
A model-based approach for the analysis of the calibration of probability judgments
The calibration of probability or confidence judgments concerns the association between the judgments and some estimate of the correct probabilities of events. Researchers rely on estimates using relative frequencies com...
When being wasteful appears better than feeling wasteful
“Waste not want not” expresses our culture’s aversion to waste. “I could have gotten the same thing for less” is a sentiment that can diminish pleasure in a transaction. We study people’s willingness to “pay” to avoid th...