Stephen Ziliak and D. N. McCloskey have sharply criticized the prevailing use of significance tests. Their work has, in turn, come under vigorous attack. The vehemence of the debate may induce readers to wrongly dismiss it as a “he said-she said” debate, or else to take sides in an unbending way that does not do justice to valid points raised by the other side. This paper aims at a more balanced reading. While Ziliak and McCloskey claim that a substantial majority of economists who use significance tests confuse statistical with substantive significance, or commit the logical error of the transposed conditional, I argue that such errors are much less frequent than they claim, though still much too pervasive. They also argue that since significance tests focus on the existence of an effect rather than on its size, the tests do not answer scientific questions. I respond with counter-examples. Ziliak and McCloskey also complain that significance tests ignore loss functions. I argue that loss functions should be introduced only at a later stage. Ziliak and McCloskey are correct, however, that confidence intervals deserve much more emphasis. The most valuable message of their work is that significance tests should be treated less mechanically.

Econometricians have been claiming proudly since World War II that significance testing is the empirical side of economics. In fact today most young economists think that the word “empirical” simply means “collect enough data to do a significance test”. Tjalling Koopmans’s influential book of 1957, Three Essays on the State of Economic Science, solidified the claim. A century of evidence after Student’s t-test points strongly to the opposite conclusion. Against conventional econometrics we argue that statistical significance is neither necessary nor sufficient for proving commercial, human, or scientific importance. A recent comment by Thomas Mayer, though in parts insightful, does nothing to alter conclusions about the logic and evidence which we and others have assembled against significance testing. Let’s bury it, and get on to empirical work that actually changes minds.

Sciblogs Archive

Sciblogs is the biggest blog network of scientists in New Zealand, an online forum for discussion of everything from clinical health to climate change. Our Scibloggers are either practising scientists or have been writing on science-related issues for some time. They welcome your feedback!

Sciblogs was created by the Science Media Centre and is independently funded