Methodological notes on model comparisons and strategy classification: A falsificationist proposition
Journal Title: Judgment and Decision Making - Year 2011, Vol 6, Issue 8
Abstract
Taking a falsificationist perspective, the present paper identifies two major shortcomings of existing approaches to comparative model evaluations in general and strategy classifications in particular. These are (1) failure to consider systematic error and (2) neglect of global model fit. Using adherence measures to evaluate competing models implicitly makes the unrealistic assumption that the error associated with the model predictions is entirely random. By means of simple schematic examples, we show that failure to discriminate between systematic and random error seriously undermines this approach to model evaluation. Second, approaches that treat random versus systematic error appropriately usually rely on relative model fit to infer which model or strategy most likely generated the data. However, the model comparatively yielding the best fit may still be invalid. We demonstrate that taking for granted the vital requirement that a model by itself should adequately describe the data can easily lead to flawed conclusions. Thus, prior to considering the relative discrepancy of competing models, it is necessary to assess their absolute fit and thus, again, attempt falsification. Finally, the scientific value of model fit is discussed from a broader perspective.
Authors and Affiliations
Morten Moshagen and Benjamin E. Hilbig
Keywords
Related Articles
- EP ID EP677864
- DOI -
- Views 142
- Downloads 0
The effects of total sleep deprivation on bayesian updating
Subjects performed a decision task (Grether, 1980) in both a well-rested and experimentally sleep-deprived state. We found two main results: 1) final choice accuracy was unaffected by sleep deprivation, and yet 2) the es...
Subjective integration of probabilistic information from experience and description
I report a new judgment task designed to investigate the subjective weights allotted to experience and description when integrating information from the two sources. Subjects estimated the percentage of red balls in a ba...
An ecological perspective to cognitive limits: Modeling environment-mind interactions with ACT-R
Contrary to the common belief that more information is always better, Gigerenzer et al. (1999) showed that simple decision strategies which rely on little information can be quite successful. The success of simple strate...
Prefer a cash slap in your face over credit for halva
We investigated how frequency and amount of punishment affect the decision making of Iranian subjects. In our first experiment, performing a computer-based Persian version of the Iowa Gambling Task (IGT), our subjects sc...
A novel approach to studying strategic decisions with eye-tracking and machine learning
We propose a novel method of using eye-tracking to study strategic decisions. The conventional approach is to hypothesize what eye-patterns should be observed if a given model of decision-making was accurate, and then pr...