An IRT forecasting model: linking proper scoring rules to item response theory
Journal Title: Judgment and Decision Making - Year 2017, Vol 12, Issue 2
Abstract
This article proposes an Item Response Theoretical (IRT) forecasting model that incorporates proper scoring rules and provides evaluations of forecasters’ expertise in relation to the features of the specific questions they answer. We illustrate the model using geopolitical forecasts obtained by the Good Judgment Project (GJP) (see Mellers, Ungar, Baron, Ramos, Gurcay, Fincher, Scott, Moore, Atanasov, Swift, Murray, Stone & Tetlock, 2014). The expertise estimates from the IRT model, which take into account variation in the difficulty and discrimination power of the events, capture the underlying construct being measured and are highly correlated with the forecasters’ Brier scores. Furthermore, our expertise estimates based on the first three years of the GJP data are better predictors of both the forecasters’ fourth year Brier scores and their activity level than the overall Brier scores obtained and Merkle’s (2016) predictions, based on the same period. Lastly, we discuss the benefits of using event-characteristic information in forecasting.
Authors and Affiliations
Yuanchao Emily Bo, David V. Budescu, Charles Lewis, Philip E. Tetlock and Barbara Mellers
Keywords
Related Articles
- EP ID EP678279
- DOI -
- Views 144
- Downloads 0
Moderators of framing effects in variations of the Asian Disease problem: Time constraint, need, and disease type
This study examined framing effects in decisions concerning public health. Tversky and Kahneman’s famous Asian Disease Problem served as experimental paradigm. Subjects chose between a sure and a risky option either pres...
How do defendants choose their trial court? Evidence for a heuristic processing account
In jurisdictions with two or more tiers of criminal courts, some defendants can choose the type of trial court to be tried in. This may involve a trade-off between the probability of acquittal/conviction and the estimate...
Introduction to the special issue: Coherence and correspondence in judgment and decision making
In 2007, Kenneth Hammond published Beyond Rationality: The search for wisdom in a troubled time. In this book and in prior works, Hammond (1990; 1996) promotes the recognition and equal acceptance of two different classe...
On making the right choice: A meta-analysis and large-scale replication attempt of the unconscious thought advantage
Are difficult decisions best made after a momentary diversion of thought? Previous research addressing this important question has yielded dozens of experiments in which participants were asked to choose the best of seve...
Energy conservation goals: What people adopt, what they recommend, and why
Failures to reduce greenhouse gas emissions by adopting policies, technologies, and lifestyle changes have led the world to the brink of crisis, or likely beyond. Here we use Internet surveys to attempt to understand the...