Abstract: Standard neuroimaging data analysis based on traditional principles of
experimental design, modelling, and statistical inference is increasingly
complemented by novel analysis methods, driven e.g. by machine learning
methods. While these novel approaches provide new insights into neuroimaging
data, they often have unexpected properties, generating a growing literature on
possible pitfalls. We propose to meet this challenge by adopting a habit of
systematic testing of experimental design, analysis procedures, and statistical
inference. Specifically, we suggest to apply the analysis method used for
experimental data also to aspects of the experimental design, simulated
confounds, simulated null data, and control data. We stress the importance of
keeping the analysis method the same in main and test analyses, because only
this way possible confounds and unexpected properties can be reliably detected
and avoided. We describe and discuss this Same Analysis Approach in detail, and
demonstrate it in two worked examples using multivariate decoding. With these
examples, we reveal two previously unknown sources of error: A mismatch between
counterbalancing (crossover designs) and cross-validation which leads to
systematic below-chance accuracies, and linear decoding of a nonlinear effect,
a difference in variance.