One-study wonders, research, and the minimum wage

Megan McArdle provides a nice overview of what we know about minimum wage hikes. What really struck me was her comments on “one-study wonders”:

Alas, if only economic analysis were so easy. In fact, it’s very hard to study what happens when we raise the minimum wage. The people confidently proclaiming their ability to see the future are often what I like to call “one-study wonders”: people who have gotten their hands on a single study that confirms what they already believe (or would very much like to) and then proceed to wave it around while ignoring the rest of the vast, conflicting, suggestive but hardly definitive economic literature on the subject.

One-study wonders are always dangerous. A single study, no matter how well done, is never proof of anything. It’s too easy for something to accidentally go wrong. Imagine, if you will, a bunch of German researchers constructing a broad representative sample of U.S. retail establishments to study average store traffic on weekdays — and unwittingly scheduling it for the day after Thanksgiving. Reality is complicated and full of spurious correlations, and even brilliant researchers are unable to anticipate all the ways that things will go wrong, all the hidden variables that confound their results. That’s why replication is one of the most important tasks in science; it’s how we check to see whether results are solid or a fluke.