Methodological notes on model comparisons and strategy classification: A falsificationist proposition
Journal Title: Judgment and Decision Making - Year 2011, Vol 6, Issue 8
Abstract
Taking a falsificationist perspective, the present paper identifies two major shortcomings of existing approaches to comparative model evaluations in general and strategy classifications in particular. These are (1) failure to consider systematic error and (2) neglect of global model fit. Using adherence measures to evaluate competing models implicitly makes the unrealistic assumption that the error associated with the model predictions is entirely random. By means of simple schematic examples, we show that failure to discriminate between systematic and random error seriously undermines this approach to model evaluation. Second, approaches that treat random versus systematic error appropriately usually rely on relative model fit to infer which model or strategy most likely generated the data. However, the model comparatively yielding the best fit may still be invalid. We demonstrate that taking for granted the vital requirement that a model by itself should adequately describe the data can easily lead to flawed conclusions. Thus, prior to considering the relative discrepancy of competing models, it is necessary to assess their absolute fit and thus, again, attempt falsification. Finally, the scientific value of model fit is discussed from a broader perspective.
Authors and Affiliations
Morten Moshagen and Benjamin E. Hilbig
Keywords
Related Articles
- EP ID EP677864
- DOI -
- Views 148
- Downloads 0
Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments
The heuristics-and-biases research program highlights reasons for expecting people to be poor intuitive forecasters. This article tests the power of a cognitive-debiasing training module (“CHAMPS KNOW”) to improve probab...
Liberal-conservative differences in inclusion-exclusion strategy choice
Inclusion and exclusion strategies for allocation of scarce goods involve different processes. The conditions under which one strategy is chosen in favor of the other, however, have not been fully explicated. In the pres...
A simple remedy for overprecision in judgment
Overprecision is the most robust type of overconfidence. We present a new method that significantly reduces this bias and offers insight into its underlying cause. In three experiments, overprecision was significantly re...
How the number of options and perceived variety influence choice satisfaction: An experiment with prescription drug plans
This study measures the perceived costs, perceived benefits, choice outcome satisfaction, and choice process satisfaction from consumers making hypothetical choices amongst prescription drug plans. I juxtapose the number...
Dishonestly increasing the likelihood of winning
People not only seek to avoid losses or secure gains; they also attempt to create opportunities for obtaining positive outcomes. When distributing money between gambles with equal probabilities, people often invest in tu...