Pair-wise comparisons of multiple models
Journal Title: Judgment and Decision Making - Year 2011, Vol 6, Issue 8
Abstract
Often research in judgment and decision making requires comparison of multiple competing models. Researchers invoke global measures such as the rate of correct predictions or the sum of squared (or absolute) deviations of the various models as part of this evaluation process. Reliance on such measures hides the (often very high) level of agreement between the predictions of the various models and does not highlight properly the relative performance of the competing models in those critical cases where they make distinct predictions. To address this important problem we propose the use of pair-wise comparisons of models to produce more informative and targeted comparisons of their performance, and we illustrate this procedure with data from two recently published papers. We use Multidimensional Scaling of these comparisons to map the competing models. We also demonstrate how intransitive cycles of pair-wise model performance can signal that certain models perform better for a given subset of decision problems.
Authors and Affiliations
Stephen B. Broomell, Budescu, David V. and Han-Hui Por
Keywords
Related Articles
- EP ID EP677865
- DOI -
- Views 163
- Downloads 0
Randomization and serial dependence in professional tennis matches: Do strategic considerations, player rankings and match characteristics matter?
In many sports contests, the equilibrium requires players to randomize across repeated rounds, i.e., exhibit no temporal predictability. Such sports data present a window into the (in)efficiency of random sequence genera...
Imagine being a nice guy: A note on hypothetical vs. incentivized social preferences
We conducted an experimental study on social preferences using dictator games similar to Fehr et al. (2008). Our results show that social preferences differ between subjects who receive low-stakes monetary rewards for th...
Beyond reach: Do symmetric changes in motor costs affect decision making? A registered report
Executing an important decision can be as easy as moving a mouse cursor or reaching towards the preferred option with a hand. But would we decide differently if choosing required walking a few steps towards an option? Mo...
Elicitation of normative and fairness judgments: Do incentives matter?
Krupka and Weber (2013) introduce an incentive-compatible coordination game as an alternative method for elicitation of normative judgments. I show, however, that people provide virtually the same responses in incentiviz...
A shocking experiment: New evidence on probability weighting and common ratio violations
We study whether probability weighting is observed when individuals are presented with a series of choices between lotteries consisting of real non-monetary adverse outcomes, electric shocks. Our estimation of the parame...