13 well-worn criticisms of significance tests (and how to avoid them)

2013 is right around the corner, and here are 13 well-known criticisms of statistical significance tests, and how they are addressed within the error statistical philosophy, as discussed in Mayo, D. G. and Spanos, A. (2011) “Error Statistics“.

(#1) error statistical tools forbid using any background knowledge.

(#2) All statistically signiﬁcant results are treated the same.

(#3) The p-value does not tell us how large a discrepancy is found.

(#4) With large enough sample size even a trivially small discrepancy from the null can be detected.

(#5) Whether there is a statistically signiﬁcant diﬀerence from the null depends on which is the null and which is the alternative.

(#6) Statistically insigniﬁcant results are taken as evidence that the null hypothesis is true.