Pair-wise comparisons of multiple models

Journal Title: Judgment and Decision Making - Year 2011, Vol 6, Issue 8

Abstract

Often research in judgment and decision making requires comparison of multiple competing models. Researchers invoke global measures such as the rate of correct predictions or the sum of squared (or absolute) deviations of the various models as part of this evaluation process. Reliance on such measures hides the (often very high) level of agreement between the predictions of the various models and does not highlight properly the relative performance of the competing models in those critical cases where they make distinct predictions. To address this important problem we propose the use of pair-wise comparisons of models to produce more informative and targeted comparisons of their performance, and we illustrate this procedure with data from two recently published papers. We use Multidimensional Scaling of these comparisons to map the competing models. We also demonstrate how intransitive cycles of pair-wise model performance can signal that certain models perform better for a given subset of decision problems.

Authors and Affiliations

Stephen B. Broomell, Budescu, David V. and Han-Hui Por

Keywords

Related Articles

Investigating an alternate form of the cognitive reflection test

Much research in cognitive psychology has focused on the tendency to conserve limited cognitive resources. The CRT is the predominant measure of such miserly information processing, and also predicts a number of frequent...

The wisdom of crowds: Predicting a weather and climate-related event

Environmental uncertainty is at the core of much of human activity, ranging from daily decisions by individuals to long-term policy planning by governments. Yet, there is little quantitative evidence on the ability of no...

The benefits of global scaling in multi-criteria decision analysis

When there are multiple competing objectives in a decision-making process, Multi-Attribute Choice scoring models are excellent tools, permitting the incorporation of both subjective and objective attributes. However, the...

A method to elicit beliefs as most likely intervals

We show how to elicit the beliefs of an expert in the form of a “most likely interval”, a set of future outcomes that are deemed more likely than any other outcome. Our method, called the Most Likely Interval elicitation...

A short form of the Maximization Scale: Factor structure, reliability and validity studies

We conducted an analysis of the 13-item Maximization Scale (Schwartz et al., 2002) with the goal of establishing its factor structure, reliability and validity. We also investigated the psychometric properties of several...

Download PDF file
  • EP ID EP677865
  • DOI -
  • Views 160
  • Downloads 0

How To Cite

Stephen B. Broomell, Budescu, David V. and Han-Hui Por (2011). Pair-wise comparisons of multiple models. Judgment and Decision Making, 6(8), -. https://europub.co.uk/articles/-A-677865