The paper describes a way of presenting research papers that allows readers to interact with the paper to explore alternative analysis options.

This new approach to statistical reporting draws from two recent ideas: Multiverse analysis and explorable explanations.

Multiverse analysis is a philosophy of statistical reporting where paper authors report the outcomes of many different statistical analyses in order to show how fragile or robust their findings are. Explorable explanations are narratives that can be read as normal explanations but where the reader can also interact with the narrative by dynamically changing some elements of the explanation.

Kay says the original motivation behind this work is a problem called “the garden of forking paths,” which illustrates how each decision a researcher makes in the data analysis – removing statistical outliers, choosing different analysis models – can lead to different paths. The result is that the conclusion based on these different choices may be different.

“One of the ideas behind multiverse analysis is to try to find a way of reporting this entire tree of possible analyses instead of just the one that you like at the end, as a way of acknowledging our uncertainty in what we should have done to the data and the impact of that on the results,” says Kay.

Rather than presenting these analyses as overwhelming static visualizations, Kay and his fellow researchers have proposed adding an interactive element that allows the reader to explore the multiverse of possible analyses that could have been done to the dataset.

“A lot of this project was us sitting down and brainstorming ideas of how we could construct these interactive reports,” says Kay. “Ditch the stale old PDF. That’s not how we should be reporting academic papers. We have the capability of making interactive papers – what should those look like?”

In this paper, Kay and team propose a way to interleave text and interactive visualizations in a way that readers have the opportunity to change things in the text and have it update the visualization – and vice versa – as a way to help people better understand the analyses. See examples here.

Best Paper awards go to the top one percent of accepted papers at CHI, the top international conference on Human-Computer Interaction. The authors will present their paper on Monday, May 6, at 4 pm at the 2019 CHI conference in Glasgow, Scotland.