Statistical models and shoe leather

An article in last Friday’s New York Times profiles Carmen Reinhart and Kenneth Rogoff, authors of This Time Is Different, which I reviewed earlier on this blog. The article highlights the divide within economics between “theoretical coherence and elegance” and “investigating data” and notes that “over the last few decades . . . economists [have] glorified financial papers that were theory-rich and data-poor.” The quants made the empiricists seem mundane, perhaps even outmoded. Reinhart and Rogoff, of course, are the quintessential data collectors and maintain that although data gathering may not save us, ignorance of data is lethal.

In a series of papers collected and edited by colleagues after his death (Statistical Models and Causal Inference: A Dialogue with the Social Sciences, Cambridge University Press, 2010) David A. Freedman, a mathematical statistician and professor of statistics at the University of California, Berkeley, also defends the empiricists. Relying heavily on epidemiological examples, he argues for a “shoe-leather” methodology against increasingly technical approaches to statistical modeling.

Put in the simplest terms, Freedman claims that without “careful empirical work tailored to the subject and the research question, informed both by subject-matter knowledge and statistical principles” the opportunity for error in statistical modeling is huge. (p. xiv) The “’desire to substitute intellectual capital for labor’ by using statistical models to avoid the hard work of examining problems in their full specificity and complexity” is both pervasive and perverse. (p. xvi)

Or, as he writes in the abstract to his paper “On Types of Scientific Inquiry: The Role of Qualitative Reasoning,” “One type of scientific inquiry involves the analysis of large data sets, often using statistical models and formal tests of hypotheses. Large observational studies have, for example, led to important progress in health science. However, in fields ranging from epidemiology to political science, other types of scientific inquiry are also productive. Informal reasoning, qualitative insights, and the creation of novel data sets that require deep substantive knowledge and a great expenditure of effort and shoe leather have pivotal roles. Many breakthroughs came from recognizing anomalies and capitalizing on accidents, which require immersion in the subject. Progress means refuting old ideas if they are wrong, developing new ideas that are better, and testing both. Qualitative insights can play a key role in all three tasks. Combining the qualitative and the quantitative—and a healthy dose of skepticism—may provide the most secure results.”
(p. 337)

Hear, hear!