Weighted Brier score decompositions for topically heterogenous forecasting tournaments
Journal Title: Judgment and Decision Making - Year 2018, Vol 13, Issue 2
Abstract
Brier score decompositions, including those attributed to Murphy and to Yates, provide popular metrics for estimating forecast performance attributes like calibration and discrimination. However, the decompositions are generally limited to situations where forecasters make successive forecast judgments against the same class of substantive event (e.g., rain vs. no rain). They do not readily translate to common situations where: forecasts are weighted unequally; forecasts can be made against a range of heterogeneous topics and events over varying time horizons; forecasts can be updated over time until an event occurs or an event deadline is reached; or outcome alternatives can vary in number and nature (e.g., ordered vs. unordered outcomes) across forecast questions. In this paper, we propose extensions of the Murphy and Yates decompositions to address these features. The extensions involve new analytic expressions for the decompositions of weighted Brier scores, along with proposed resampling methods. We use data from a recent forecasting tournament to illustrate the methods.
Authors and Affiliations
Edgar C. Merkle and Robert Hartman
Keywords
Related Articles
- EP ID EP678336
- DOI -
- Views 152
- Downloads 0
Are buyers of apartments superstitious? Evidence from the Russian real estate market
We study the influence of numerological superstitions on people’s buying behavior in the apartment market using unique actual sales data. Based on the dataset from Saint-Petersburg primary real estate market we compare t...
American’s desire for less wealth inequality does not depend on how you ask them
A large body of survey research offers evidence that citizens are not always fully aware of the economic and political realities in their respective countries. Norton and Ariely (2011) extended this research to the domai...
On the psychology of self-prediction: Consideration of situational barriers to intended actions
When people predict their future behavior, they tend to place too much weight on their current intentions, which produces an optimistic bias for behaviors associated with currently strong intentions. More realistic self-...
Gender differences in the endowment effect: Women pay less, but won’t accept less
We explore different contexts and mechanisms that might promote or alleviate the gender effect in risk aversion. Our main result is that we do not find gender differences in risk aversion when the choice is framed as a w...
Energy conservation goals: What people adopt, what they recommend, and why
Failures to reduce greenhouse gas emissions by adopting policies, technologies, and lifestyle changes have led the world to the brink of crisis, or likely beyond. Here we use Internet surveys to attempt to understand the...