Pair-wise comparisons of multiple models
Journal Title: Judgment and Decision Making - Year 2011, Vol 6, Issue 8
Abstract
Often research in judgment and decision making requires comparison of multiple competing models. Researchers invoke global measures such as the rate of correct predictions or the sum of squared (or absolute) deviations of the various models as part of this evaluation process. Reliance on such measures hides the (often very high) level of agreement between the predictions of the various models and does not highlight properly the relative performance of the competing models in those critical cases where they make distinct predictions. To address this important problem we propose the use of pair-wise comparisons of models to produce more informative and targeted comparisons of their performance, and we illustrate this procedure with data from two recently published papers. We use Multidimensional Scaling of these comparisons to map the competing models. We also demonstrate how intransitive cycles of pair-wise model performance can signal that certain models perform better for a given subset of decision problems.
Authors and Affiliations
Stephen B. Broomell, Budescu, David V. and Han-Hui Por
Keywords
Related Articles
- EP ID EP677865
- DOI -
- Views 156
- Downloads 0
Professionally biased: Evidence for misestimations of driving speed, journey time and time-savings among taxi and car drivers
People make systematic and predictable mistakes regarding estimations of average speed and journey time. In addition, people have been shown to commit a time-saving bias by underestimating the time that can be saved when...
Learning to communicate risk information in groups
Despite vigorous research on risk communication, little is known about the social forces that drive these choices. Erev, Wallsten, & Neal (1991) showed that forecasters learn to select verbal or numerical probability est...
Perceptions of water systems
Public understanding of the water system is vital in confronting contemporary water challenges, as public support is necessary for implementing measures to address shortages and repair infrastructure. In this study, univ...
From group diffusion to ratio bias: Effects of denominator and numerator salience on intuitive risk and likelihood judgments
The group-diffusion effect is the tendency for people to judge themselves to be less likely to experience a negative outcome as the total number of people exposed to the threat increases — even when the probability of th...
Information search with situation-specific reward functions
The goal of obtaining information to improve classification accuracy can strongly conflict with the goal of obtaining information for improving payoffs. Two environments with such a conflict were identified through compu...