Using inferred probabilities to measure the accuracy of imprecise forecasts
Journal Title: Judgment and Decision Making - Year 2012, Vol 7, Issue 6
Abstract
Research on forecasting is effectively limited to forecasts that are expressed with clarity; which is to say that the forecasted event must be sufficiently well-defined so that it can be clearly resolved whether or not the event occurred and forecasts certainties are expressed as quantitative probabilities. When forecasts are expressed with clarity, then quantitative measures (scoring rules, calibration, discrimination, etc.) can be used to measure forecast accuracy, which in turn can be used to measure the comparative accuracy of different forecasting methods. Unfortunately most real world forecasts are not expressed clearly. This lack of clarity extends to both the description of the forecast event and to the use of vague language to express forecast certainty. It is thus difficult to assess the accuracy of most real world forecasts, and consequently the accuracy the methods used to generate real world forecasts. This paper addresses this deficiency by presenting an approach to measuring the accuracy of imprecise real world forecasts using the same quantitative metrics routinely used to measure the accuracy of well-defined forecasts. To demonstrate applicability, the inferred probability method is applied to measure the accuracy of forecasts in fourteen documents examining complex political domains.
Authors and Affiliations
Paul Lehner, Avra Michelson, Leonard Adelman and Anna Goodman
Cynicism in negotiation: When communication increases buyers’ skepticism
The economic literature on negotiation shows that strategic concerns can be a barrier to agreement, even when the buyer values the good more than the seller. Yet behavioral research demonstrates that human interaction ca...
‘Tis better to choose and lose than to never choose at all
When decisions involve opting in or out of competition many decision makers will opt-in even when doing so leads to losses on average. In the current paper, we examine the generality of this effect in risky choices not i...
Gender differences in the endowment effect: Women pay less, but won’t accept less
We explore different contexts and mechanisms that might promote or alleviate the gender effect in risk aversion. Our main result is that we do not find gender differences in risk aversion when the choice is framed as a w...
The default pull: An experimental demonstration of subtle default effects on preferences
The impact of default options on choice is a reliable, well-established behavioral finding. However, several different effects may lend to choosing defaults in an often indistinguishable manner, including loss aversion,...
Response dynamics: A new window on the decision process
The history of judgment and decision making is defined by a trend toward increasingly nuanced explanations of the decision making process. Recently, process models have become incredibly sophisticated, yet the tools avai...