Criteria for performance evaluation
Journal Title: Judgment and Decision Making - Year 2009, Vol 4, Issue 2
Abstract
Using a cognitive task (mental calculation) and a perceptual-motor task (stylized golf putting), we examined differential proficiency using the CWS index and several other quantitative measures of performance. The CWS index (Weiss & Shanteau, 2003) is a coherence criterion that looks only at internal properties of the data without incorporating an external standard. In Experiment 1, college students (n = 20) carried out 2- and 3-digit addition and multiplication problems under time pressure. In Experiment 2, experienced golfers (n = 12), also college students, putted toward a target from nine different locations. Within each experiment, we analyzed the same responses using different methods. For the arithmetic tasks, accuracy information (mean absolute deviation from the correct answer, MAD) using a coherence criterion was available; for golf, accuracy information using a correspondence criterion (mean deviation from the target, also MAD) was available. We ranked the performances of the participants according to each measure, then compared the orders using Spearman’s rs. For mental calculation, the CWS order correlated moderately (rs =.46) with that of MAD. However, a different coherence criterion, degree of model fit, did not correlate with either CWS or accuracy. For putting, the ranking generated by CWS correlated .68 with that generated by MAD. Consensual answers were also available for both experiments, and the rankings they generated correlated highly with those of MAD. The coherence vs. correspondence distinction did not map well onto criteria for performance evaluation.
Authors and Affiliations
David J. Weiss, Kristin Brennan, Rick Thomas, Alex Kirlik and Sarah M. Miller
Keywords
Related Articles
- EP ID EP677672
- DOI -
- Views 190
- Downloads 0
Unconscious intuition or conscious analysis? Critical questions for the Deliberation-Without-Attention paradigm
The Deliberation without Attention (DWA) effect refers to apparent improvements in decision-making following a period of distraction. It has been presented as evidence for beneficial unconscious cognitive processes. We i...
Modeling sequential context effects in judgment analysis:
In this article a broad perspective incorporating elements of time series theory is presented for conceptualizing the data obtained in multi-trial judgment experiments. Recent evidence suggests that sequential context ef...
A statistical test of independence in choice data with small samples
This paper develops tests of independence and stationarity in choice data collected with small samples. The method builds on the approach of Smith and Batchelder (2008). The technique is intended to distinguish cases whe...
The impact of regret and worry on the threshold level of concern for flood insurance demand: Evidence from Dutch homeowners
It has been argued that individuals behave according to a threshold level of concern decision rule when considering protection against risk: if the perceived probability of the risk is below a threshold level, then the l...
Intuition speed as a predictor of choice and confidence in point spread predictions
Previous research has revealed that intuitive confidence is an important predictor of how people choose between an intuitive and non-intuitive alternative when faced with information that opposes the intuitive response....