Reply: Birnbaum’s (2012) statistical tests of independence have unknown Type-I error rates and do not replicate within participant
Journal Title: Judgment and Decision Making - Year 2013, Vol 8, Issue 1
Abstract
Birnbaum (2011, 2012) questioned the iid (independent and identically distributed) sampling assumptions used by state-of-the-art statistical tests in Regenwetter, Dana and Davis-Stober’s (2010, 2011) analysis of the “linear order model”. Birnbaum (2012) cited, but did not use, a test of iid by Smith and Batchelder (2008) with analytically known properties. Instead, he created two new test statistics with unknown sampling distributions. Our rebuttal has five components: 1) We demonstrate that the Regenwetter et al. data pass Smith and Batchelder’s test of iid with flying colors. 2) We provide evidence from Monte Carlo simulations that Birnbaum’s (2012) proposed tests have unknown Type-I error rates, which depend on the actual choice probabilities and on how data are coded as well as on the null hypothesis of iid sampling. 3) Birnbaum analyzed only a third of Regenwetter et al.’s data. We show that his two new tests fail to replicate on the other two-thirds of the data, within participants. 4) Birnbaum selectively picked data of one respondent to suggest that choice probabilities may have changed partway into the experiment. Such nonstationarity could potentially cause a seemingly good fit to be a Type-II error. We show that the linear order model fits equally well if we allow for warm-up effects. 5) Using hypothetical data, Birnbaum (2012) claimed to show that “true-and-error” models for binary pattern probabilities overcome the alleged short-comings of Regenwetter et al.’s approach. We disprove this claim on the same data.
Authors and Affiliations
Yun-shil Cha, Michelle Choi, Ying Guo, Michel Regenwetter and Chris Zwilling
Keywords
Related Articles
- EP ID EP678043
- DOI -
- Views 122
- Downloads 0
A universal method for evaluating the quality of aggregators
We propose a new method to facilitate comparison of aggregated forecasts based on different aggregation, elicitation and calibration methods. Aggregates are evaluated by their relative position on the cumulative distribu...
Energy conservation goals: What people adopt, what they recommend, and why
Failures to reduce greenhouse gas emissions by adopting policies, technologies, and lifestyle changes have led the world to the brink of crisis, or likely beyond. Here we use Internet surveys to attempt to understand the...
A marketing science perspective on recognition-based heuristics (and the fast-and-frugal paradigm)
Marketing science seeks to prescribe better marketing strategies (advertising, product development, pricing, etc.). To do so we rely on models of consumer decisions grounded in empirical observations. Field experience su...
"Decisions from experience" = sampling error + prospect theory: Reconsidering Hertwig, Barron, Weber & Erev (2004)
According to prospect theory, people overweight low probability events and underweight high probability events. Several recent papers (notably, Hertwig, Barron, Weber & Erev, 2004) have argued that although this pattern...
Exemplar-based inference in multi-attribute decision making: Contingent, not automatic, strategy shifts?
Several studies propose that exemplar retrieval contributes to multi-attribute decisions. The authors have proposed a process theory enabling a priori predictions of what cognitive representations people use as input to...