In his October 31st post, Marty stated “The statistical significance test simply assesses the likelihood of the rival hypothesis of “chance.” ” I would like to elaborate a little on this statement because it makes a very important point about statistical hypothesis testing. As both Bonnie and Marty have indicated, there will always be error in any data that we collect– sampling error, measurement error, and experimenter-procedural error. Unfortunately, humans are not well prepared to assess the extent of this error in data from a mere observational basis. Too often we are wont to see relationships in nature where none exist. Statistical hypothesis testing offers a relatively simple (although students often don’t initially perceive it to be simple) solution for this problem.

A statistical hypothesis test is a dispassionate method of making a decision of whether “chance” most reasonably explains the relationship observed in the data or is there something else we should search for in explanation. It is important to remember that the statistical test simply tests a null hypothesis assuming that certain conditions apply in the data being tested. It is up to the experimenter to insure that those conditions are met by his or her data. And, if the hypothesis test indicates that the hypothesis of chance is an unlikely explanation of the relationship observed, it does not provide any evidence that the research hypothesis is a plausible explanation for the results. An experiment can be confounded or a third variable may be responsible for an observed correlation. The statistical test cannot assess the likelihood of such occurrences in the data, only a careful analysis of the design of the research can provide that assessment. This important point is sometimes misrepresented by text authors with statements similar to “the alternative hypothesis states that the independent variable does affect the dependent variable.” But the alternative hypothesis of a statistical tests states no such such relationship. For a parametric test, it simply indicates that the sample means were drawn from different populations, but not the reason why those populations may differ. And this alternative hypothesis remains essentially the same regardless of the experimenter’s research hypothesis. On the other hand, if the hypothesis test indicates that chance is the most plausible explanation for the results obtained, then again it cannot indicate whether the result was from a poor design or inadequate measurement of the variables in the research.

Hypothesis testing thus simply provides an objective way of deciding that given the data we have obtained, is chance a plausible explanation? But hypothesis testing is simply the start of the explanatory process, not the end of that process.

2 responses to “What Does a Statistical Hypothesis Test Test?”

Hi Hal,
The idea of chance differences and sampling error are so critical for students to understand if they are to be able to properly use and interpret Null Hypothesis Testing.
Though I typically don’t think of my classes in this fashion, I was once asked to estimate the percentage of time I spend on specific topics for one of the classes I teach. As I read your post, it occurred to me, that I spend an extreme amount of time talking about “chance” differences due to sampling error. Even during my class yesterday, where I felt I had so much to cover and not enough time, I still took about 5 minutes to have students generate individual/chance differences present in the study we were discussing.

Another way I help students to focus on the role of chance/individual differences in Null Hypothesis Testing is to provide a multi-step homework assignment. First, the students generate a list of 3 hypotheses, then they are asked to generate a list of potential “chance differences” (extraneous variables that could add to sampling error) for each hypothesis.

Yes, chance differences is a critical concept for students to grasp, one that if absent, then the ability to fully use and appreciate what Null Hypothesis Testing truly becomes compromised.