An
Overview: Choosing the Correct Statistical Test

The correct statistical test for an experiment largely
depends on the nature of the independent and dependent variables analyzed. For
the purpose of choosing a statistical test, variables fall into two classes:
Categorical and Continuous. Categorical variable values cannot be sequentially
ordered or differentiated from each other using a mathematical method.

Examples include:

·gender

·ethnicity

·software user interfaces

Continuous variables are numeric values that can be ordered
sequentially, and that do not naturally fall into discrete ranges.

Examples include:

·weight

·number of seconds it takes to perform a task

·number of words on a user interface

These concepts can be combined to make a simple model for
choosing the correct statistical testi

Dependent Variable

Categorical

Continuous

Independent

Variable

Categorical

Chi Square

t-test, ANOVA

Continuous

LDA, QDA

Regression

The model is straightforward,
illustrating how the nature of the independent and dependent variables drive
the choice of a statistical test.

With understanding of the basic
model for choosing a statistical test, we can add relevant details to the
model. First, we need to address two additional types of variables, ordinal and
interval.

First, ordinal variables are similar
to continuous variables; they can be ordered sequentially. They are also
similar to categorical variables because they (perhaps) cannot be
differentiated from each other using a mathematical method. For example,
education level is an ordinal variable. The levels of educational achievement
(high school, some college, undergraduate degree, etc.) can be sequenced in the
order in which they are achieved, and when defined as such, cannot be
differentiated from each other mathematically. So the question is, using the
simple model for choosing a statistical test, is an ordinal variable
Categorical or Continuous? The answer depends on how the researcher defines the
variable. When education levels are defined as high school, some college,
undergraduate degree, etc., the levels are categorical, and the researcher
should choose a test for categorical data. The researcher could, however,
define education level in a slightly different way. If the researcher instead
defined education level as years of full-time education, then the variable
takes on the characteristics of a Continuous variable, and the researcher
should choose a statistical test for a Continuous variable.

Interval variables also exhibit
characteristics of Categorical and Continuous variables. Interval variables
fall into equally spaced ranges. For example, an experimenter collects salary
levels using the following ranges:

·$10,000 – 20,000

·$20, 000 – 30,000

·$30,0000 – 40,000, etc.

The values can be numerically sequenced,
so they are similar to Continuous variables. Because the ranges are equally
spaced, though, an unnatural restriction is placed on the values, and thus they
are similar to Categorical values. When it comes to choosing a statistical
test, there is no hard and fast rule for defining interval data as Categorical
or Continuous, and the researcher should use his/her discretion in making the
choice. Granularity of ranges is a reasonable guide for deciding how to define
the data. For example, when intervals are granular, the researcher may decide
to define the variable as Continuous, and for coarser intervals, Categorical.

Number of variables

The number of independent and
dependent variable in the experiment also affect which statistical test to
choose. For example, linear regression applies when the researcher compares 1
continuous dependent variable and 1 continuous independent variable. Multiple
regression applies when the researcher compares 2 or more continuous
independent variables against 1 continuous dependent variable.

The number of levels of a
categorical variable can also drive which statistical test to use. For example,
a researcher wants to compare whether gender affects the amount of time to
perform a task using a given user interface. Gender serves as a 2 level
categorical independent variable because it has 2 possible values: male and
female. Time to complete the task would serve as the continuous dependent
variable. In this example, a 2-sample t-test would be the correct statistical
test. If the categorical independent variable has more than 2 values, however,
one-way ANOVA should be applied. Throughout this guide, the number of
independent and dependent variables needed to run the statistical test are
included right after the section heading.

Normality

For a given set of independent and
dependent variables, often there are two statistical tests available: one
parametric and one non-parametric. Parametric tests are appropriate when
continuous variables follow a normal distribution, and non-parametric tests are
appropriate when they do not. Throughout this guide, the numeric distribution
requirements are included right after the section heading.