Weighted Brier score decompositions for topically heterogenous forecasting tournaments
Journal Title: Judgment and Decision Making - Year 2018, Vol 13, Issue 2
Abstract
Brier score decompositions, including those attributed to Murphy and to Yates, provide popular metrics for estimating forecast performance attributes like calibration and discrimination. However, the decompositions are generally limited to situations where forecasters make successive forecast judgments against the same class of substantive event (e.g., rain vs. no rain). They do not readily translate to common situations where: forecasts are weighted unequally; forecasts can be made against a range of heterogeneous topics and events over varying time horizons; forecasts can be updated over time until an event occurs or an event deadline is reached; or outcome alternatives can vary in number and nature (e.g., ordered vs. unordered outcomes) across forecast questions. In this paper, we propose extensions of the Murphy and Yates decompositions to address these features. The extensions involve new analytic expressions for the decompositions of weighted Brier scores, along with proposed resampling methods. We use data from a recent forecasting tournament to illustrate the methods.
Authors and Affiliations
Edgar C. Merkle and Robert Hartman
Keywords
Related Articles
- EP ID EP678336
- DOI -
- Views 172
- Downloads 0
Who helps more? How self-other discrepancies influence decisions in helping situations
Research has shown that people perceive themselves as less biased than others, and as better than average in many favorable characteristics. We suggest that these types of biased perceptions regarding intentions and beha...
Psychological aspects of the rejection of recycled water: Contamination, purification and disgust
There is a worldwide and increasing shortage of potable fresh water. Modern water reclamation technologies can alleviate much of the problem by converting wastewater directly into drinking water, but there is public resi...
The irrational hungry judge effect revisited: Simulations reveal that the magnitude of the effect is overestimated
Danziger, Levav and Avnaim-Pesso (2011) analyzed legal rulings of Israeli parole boards concerning the effect of serial order in which cases are presented within ruling sessions. They found that the probability of a favo...
On the use of recognition in inferential decision making: An overview of the debate
I describe and discuss the sometimes heated controversy surrounding the recognition heuristic (RH) as a model of inferential decision making. After briefly recapitulating the history of the RH up to its current version,...
Reluctant altruism and peer pressure in charitable giving
Subjects donate individually (control group) or in pairs (treatment group). Those in pairs reveal their donation decision to each other. Average donations in the treatment group are significantly higher than in the contr...