Friday, October 30, 2015

There are two types of psychological researchers: those who
are acutely aware of what is widely referred to as the replication crisis,
and those who have heard about it. The factum of this crisis is simple: it is
becoming increasingly obvious that some proportion of the published literature reports
empirical results that cannot be reproduced and that are likely to be false.
The implication is clear: if scientists believe in results that are false, they
will build theories based on erroneous assumptions, which in turn has
detrimental effects on instances when these theories are applied to practical
domains (e.g., treatment of psychological disorders, evidence-based teaching
methods, hiring decisions by organisational psychologists). Clearly, this is an
outcome that needs to be avoided at all costs.

Luckily, it is relatively clear what needs to be done to
improve the way science works. Three issues that drive the crisis are generally
discussed: (1) Publication bias (e.g., Ioannidis, 2005; Rosenthal, 1979; Simonsohn, Nelson, & Simmons,
2014),
(2) underpowered studies (e.g., Button et al., 2013; Christley, 2010; Cohen, 1962; Royall, 1986),
and (3) questionable research practices (e.g., John, Loewenstein, & Prelec, 2012; Wagenmakers, Wetzels,
Borsboom, van der Maas, & Kievit, 2012). If all three of these issues
were addressed, the replication crisis would be – if not eliminated altogether
– strongly reduced. Addressing Point (1) requires changes in the incentive
system of academia (see, e.g., Lakens & Evers, 2014, for a discussion), and in the way journals
decide whether or not to publish a given submission. While this is possibly the
most important issue to solve, an individual researcher cannot take an easy set
of routine steps to achieve this desired end state. Point (2) requires
researchers to carefully think about the expected or minimal practically
meaningful effect size and to determine the number of participants, a priori, to be able to make meaningful
conclusions about the presence or absence of such an effect. While this is not
too tricky conceptually, it often requires sample sizes that appear unrealistic
to one who is used to test 16 participants per condition. In some instances –
for example, when working with a clinical population – collecting a large
number of participants can indeed be unrealistic. Thus, addressing this issue
also requires some deeper changes in the research structure, such as
encouraging collaborations between many labs to share the work associated with
collecting sufficient data, and making available the raw data of all smaller
studies (regardless of their outcome), to allow for meta-analyses across these
smaller studies.

In this blog post, I focus on Point (3). The prevalence of
questionable research practices in existing studies is easiest to address, in
the sense that each individual researcher could contribute to resolving this
issue. In practical terms, this would require spreading awareness of how not to
p-hack. The term (and concept of) p-hacking is something I became aware of
relatively late during my PhD. At the department, we had a workshop by Zoltan Dienes,
who talked, among other things, about optional stopping. For the first time, I
heard that it was not good practice to collect some data, look at the results,
and then to decide whether or not to collect more data. There were many
surprised reactions to this bit of information, including by senior researchers,
showing that researchers in general are not necessarily aware of what
constitutes questionable research practices.

In the majority of cases, questionable research practices,
such as optional stopping in a frequentist framework, are not a result of
malevolence or dishonesty, but rather stem from a lack of awareness. In
addition to this workshop, I have encountered many other examples of senior
researchers who did not seem to be aware that some common research practices
are considered to be questionable. As some additional examples, here is some
advice that I got from various researchers throughout my studies:

At a student pre-conference
workshop, a senior professor said: “Always check your data after you’ve tested,
say, five participants. If there is not even a hint of an effect, there’s no
point wasting your time by collecting more data.”

In response to some data which
did not show a hypothesised significant correlation between two variables,
another senior scientist advised to remove some “obvious outliers” and look at
a subgroup of participants, because “clearly, there’s something there!”

Another senior researcher told me:
“For any effect that you want to test, repeat the analysis with at least five
different trimming methods and dependent variable transformations, and only believe in the
effect if the results of all analysis methods converge.”

All of these researchers genuinely wanted to give good and
helpful advice to a student. However, all of these practices could be
considered questionable. Again, this demonstrates the lack of awareness of what
passes as good research practices and what doesn’t. Addressing the issue of
questionable research practices, therefore, would require, first and foremost, spreading
awareness of what constitutes p­-hacking.
To this end, I attempt to make a list of common practices that are not
acceptable in a frequentist framework:

(1)Optional
stopping: Collect some data, and then decide whether or not more data should be
collected based on the obtained results. Note that things change if we plan the
multiple analyses in advance and adjust the alpha-level accordingly, as
described, e.g., in this
blogpost by Daniël Lakens.

(2)Creative
trimming: If the expected effect is not significant, have a closer look at the
individual subject data (and/or item-level data, in a psycholinguistic
experiment), and exclude any subjects (and/or items) that could be considered
outliers, until the required effect becomes significant.

(3)Adding
covariates: If the expected effect is not significant, perhaps there are some
confounding variables which we can include as covariates in various
constellations until the effect becomes significant?

(4)When
an independent variable has more than two levels and the overall main effect is
not significant, do follow-up tests anyway, to see if the effect emerges at any
level.

(5)Looking
at subgroups: If the expected effect is not significant, maybe it shows up only
in women? Or in participants aged 40 and over? Or those with particularly fast
reaction times?

(6)Cramming
as many independent variables as possible into a multi-way ANOVA: The more
contrasts you have, the greater the chance that at least one of them will be
statistically significant.

All of these issues are related to the problem of multiple
testing. In the frequentist framework, multiple testing is a problem, because
the p-value does not do what many
people seem to think it does: it does not tell us anything about the likelihood
of an effect. Instead, it tells us the likelihood of obtaining the data, if we
assume that there is no effect. The usefulness of a p-value, therefore, lies not in telling you whether or not there is
an effect, but as a tool to ensure that, in the long run, your Type-I error
rate (i.e., probability of erroneously concluding that there is an effect) is no greater than the
alpha-level (i.e., 5% for a cut-off of p
< 0.05). If one calculates multiple p-values
from the same data, the probability that at least one of these is smaller than
0.05 increases. This compromises the desirable property of the p-value, that in the long run, 5% of all
studies will be false positives under the null hypothesis: in other words, the
probability of a false positive under the null increases substantially with
multiple comparisons. This is described in detail in multiple publications (e.g., Cramer et al., 2015; Dienes, 2011; Forstmeier & Schielzeth,
2011)
and blog posts (e.g., by Hilda
Bastian and Dorothy
Bishop).

What can we do?

If we want to avoid questionable research practices (as we
should), it is important to consider alternatives that would allow us to do
sound and honest analyses. There are three possibilities, which have been
discussed in the previous literature:

(1)Alpha-level
corrections (e.g,. see Cramer et al., 2015).

(2)Pre-registration
(e.g., Cramer et al., 2015; Wagenmakers et al., 2012): Before collecting data, a
pre-registration report can be uploaded, e.g., on the Open Science Framework
(OSF) website (osf.io). The OSF website allows you to
create a non-modifiable record of your report, which can be submitted along
with the paper. Such a pre-registration report should contain details about the
independent and dependent variables, the comparison of interest (including an
explicit hypothesis), the number of participants that will be tested, and the
trimming procedure. Any post-hoc decisions
(e.g., about including an unexpected confound in the final analyses) can still
be added, assuming there is a sound justification, because it will be clear
both to the authors and readers that this was a post-hoc decision. Pre-registration is possibly the quickest and
most efficient way to do good science – not to mention that deciding on these
analysis issues a priori will save a
lot of time during the analyses!

(3)Changing
to Bayesian analysis methods (e.g., Dienes, 2011; Rouder, 2014; Wagenmakers, 2007):
For reasons explained in these papers, Bayesian analyses are not susceptible to
multiple testing problems. The Bayesian philosophical approach is different from that of the
frequentists: here, we are not concerned with long-term error rates,
but rather about quantifying the degree of support for one hypothesis over
another (which, incidentally, seems to be what some researchers think they are
doing when they are calculating a p-value).
Bayes Factors scare some researchers because they are more difficult to
calculate than p-values, but this is
changing. R users can download the BayesFactor
package, which allows them to do Bayesian model comparisons using the same
models they would use in a frequentist analysis (ANOVAs, LMEs, regressions).
SPSS users can use the newly developed software JASP,
which is user-friendly and similar in layout to SPSS – and it’s open source!

Conclusion

Thanks to some excellent work by various researchers (see
the reference list below), we have a pretty good idea of what needs to be done
to address the replication crisis. While two possible solutions require deeper
changes to the way in which research is conducted and disseminated – namely
combatting the practice of conducting under-powered studies and publication
bias – the prevalence of questionable research practices can (and should) be
addressed by each individual researcher. As shown by my anecdotes above, some
are genuinely unaware about the drawbacks of multiple tests in frequentist
frameworks – although this is based, of course, on my anecdotal evidence, and
is hopefully changing with the recent publicity associated with the replication
crisis. My guess is that once researchers become aware of the practices that
are considered dishonest, and learn that there are viable alternatives, the
majority will start to use honest and sound methods, because it’s a
win-win situation: Overall, science will become more reproducible, and
researchers will have a warm, fuzzy feeling, knowing that they can be more
confident in their exciting new findings.