Pair-wise comparisons of multiple models
Journal Title: Judgment and Decision Making - Year 2011, Vol 6, Issue 8
Abstract
Often research in judgment and decision making requires comparison of multiple competing models. Researchers invoke global measures such as the rate of correct predictions or the sum of squared (or absolute) deviations of the various models as part of this evaluation process. Reliance on such measures hides the (often very high) level of agreement between the predictions of the various models and does not highlight properly the relative performance of the competing models in those critical cases where they make distinct predictions. To address this important problem we propose the use of pair-wise comparisons of models to produce more informative and targeted comparisons of their performance, and we illustrate this procedure with data from two recently published papers. We use Multidimensional Scaling of these comparisons to map the competing models. We also demonstrate how intransitive cycles of pair-wise model performance can signal that certain models perform better for a given subset of decision problems.
Authors and Affiliations
Stephen B. Broomell, Budescu, David V. and Han-Hui Por
Keywords
Related Articles
- EP ID EP677865
- DOI -
- Views 138
- Downloads 0
Moral judgments of risky choices: A moral echoing effect
Two experiments examined moral judgments about a decision-maker’s choices when he chose a sure-thing, 400 out of 600 people will be saved, or a risk, a two-thirds probability to save everyone and a one-thirds probability...
Are additives unnatural? Generality and mechanisms of additivity dominance
Naturalness is important and valued by most lay Western individuals. Yet, little is known about the lay meaning of “natural”. We examine the phenomenon of additivity dominance: adding something to a natural product (addi...
Response mode, compatibility, and dual-processes in the evaluation of simple gambles: An eye-tracking investigation
We employed simple gambles to investigate information processing in relation to the compatibility effect. Subjects should be more likely to engage in a deliberative thinking strategy when completing a pricing task rather...
The Short Maximization Inventory
We developed the Short Maximization Inventory (SMI) by shortening the Maximization Inventory (Turner, Rim, Betz & Nygren, 2012) from 34 items to 15 items. Using the Item Response Theory framework, we identified and remov...
Tailored proper scoring rules elicit decision weights
Proper scoring rules are scoring methods that incentivize honest reporting of subjective probabilities, where an agent strictly maximizes his expected score by reporting his true belief. The implicit assumption behind pr...