An IRT forecasting model: linking proper scoring rules to item response theory
Journal Title: Judgment and Decision Making - Year 2017, Vol 12, Issue 2
Abstract
This article proposes an Item Response Theoretical (IRT) forecasting model that incorporates proper scoring rules and provides evaluations of forecasters’ expertise in relation to the features of the specific questions they answer. We illustrate the model using geopolitical forecasts obtained by the Good Judgment Project (GJP) (see Mellers, Ungar, Baron, Ramos, Gurcay, Fincher, Scott, Moore, Atanasov, Swift, Murray, Stone & Tetlock, 2014). The expertise estimates from the IRT model, which take into account variation in the difficulty and discrimination power of the events, capture the underlying construct being measured and are highly correlated with the forecasters’ Brier scores. Furthermore, our expertise estimates based on the first three years of the GJP data are better predictors of both the forecasters’ fourth year Brier scores and their activity level than the overall Brier scores obtained and Merkle’s (2016) predictions, based on the same period. Lastly, we discuss the benefits of using event-characteristic information in forecasting.
Authors and Affiliations
Yuanchao Emily Bo, David V. Budescu, Charles Lewis, Philip E. Tetlock and Barbara Mellers
Keywords
Related Articles
- EP ID EP678279
- DOI -
- Views 156
- Downloads 0
Simple eye movement metrics can predict future decision making performance: The case of financial choices
Decisions are often delegated to experts chosen based on their past performance record which may be subject to noise. For instance, a person with little skill could still make a lucky decision that proves correct ex-post...
Cynicism in negotiation: When communication increases buyers’ skepticism
The economic literature on negotiation shows that strategic concerns can be a barrier to agreement, even when the buyer values the good more than the seller. Yet behavioral research demonstrates that human interaction ca...
Lay understanding of probability distributions
How accurate are laypeople’s intuitions about probability distributions of events? The economic and psychological literatures provide opposing answers. A classical economic view assumes that ordinary decision makers cons...
Are complex decisions better left to the unconscious? Further failed replications of the deliberation-without-attention effect
The deliberation-without-attention effect occurs when better decisions are made when people experience a period of distraction before a decision than when they make decisions immediately or when they spend time reflectin...
Attachment to land: The case of the land of Israel for American and Israeli Jews and the role of contagion
This is a first study on attachment to national and sacred land and land as a protected value. A measure of attachment to the land of Israel is developed and administered to two groups, Jewish college students in Israel...