Take a fantastic 3D voyage through the brain with immersive VR system

November 23, 2017

What happens when you combine access to unprecedented huge amounts of anatomical data of brain structures with the ability to display billions of voxels (3D pixels) in real time, using high-speed graphics cards?

Answer: An awesome new immersive virtual reality (VR) experience for visualizing and interacting with up to 10 terabytes (trillions of bytes) of anatomical brain data.

Developed by researchers from the Wyss Center for Bio and Neuroengineering and the University of Geneva, the system is intended to allow neuroscientists to highlight, select, slice, and zoom on down to individual neurons at the micrometer cellular level.

This 2-D brain image of a mouse brain injected with a fluorescent retrograde virus in the brain stem — captured with a lightsheet microscope — represents the kind of rich, detailed visual data that can be explored with a new VR system. (credit: Courtine Lab/EPFL/Leonie Asboth, Elodie Rey)

The new VR system grew out of a problem with using the Wyss Center’s lightsheet microscope (one of only three in the world): how can you navigate and make sense out the immense volume of neuroanatomical data?

“The system provides a practical solution to experience, analyze and quickly understand these exquisite, high-resolution images,” said Stéphane Pages, PhD, Staff Scientist at the Wyss Center and Senior Research Associate at the University of Geneva, senior author of a dynamic poster presented November 15 at the annual meeting of the Society for Neuroscience 2017.

For example, using “mini-brains,” researchers will be able to see how new microelectrode probes behave in brain tissue, and how tissue reacts to them.

Journey to the center of the cell: VR movies

A team of researchers in Australia has taken the next step: allowing scientists, students, and members of the public to explore these kinds of images — even interact with cells and manipulate models of molecules.

As described in a paper published in the journal Traffic, the researchers built a 3D virtual model of a cell, combining lightsheet microscope images (for super-resolution, real-time, single-molecule detection of fluorescent proteins in cells and tissues) with scanning electron microscope imaging data (for a more complete view of the cellular architecture).

To demonstrate this, they created VR movies (shown below) of the surface of a breast cancer cell. The movies can be played on a Samsung Gear VR or Google cardboard device or using the built-in YouTube 360 player with Chrome, Firefox, MS Edge, or Opera browsers. The movies will also play on a conventional smartphone (but without 3D immersion).

UNSW 3D Visualisation Aesthetics Lab | The cell “paddock” view puts the user on the surface of the cell and demonstrates different mechanisms by which nanoparticles can be internalized into cells.

Recent advances in high-resolution 3D imaging techniques allow researchers to access unprecedented amounts of anatomical data of brain structures. In parallel, the computational power of commodity graphics cards has made rendering billions of voxels in real-time possible. Combining these technologies in an immersive virtual reality system creates a novel tool wherein observers can physically interact with the data. We present here the possibilities and demonstrate the value of this approach for reconstructing neuroanatomical data. We use a custom built digitally scanned light-sheet microscope (adapted from Tomer et al., Cell, 2015), to image rodent clarified whole brains and spinal cords in which various subpopulations of neurons are fluorescently labeled. Improvements of existing microscope designs allow us to achieve an in-plane submicronic resolution in tissue that is immersed in a variety of media (e. g. organic solvents, Histodenz). In addition, our setup allows fast switching between different objectives and thus changes image resolution within seconds. Here we show how the large amount of data generated by this approach can be rapidly reconstructed in a virtual reality environment for further analyses. Direct rendering of raw 3D volumetric data is achieved by voxel-based algorithms (e.g. ray marching), thus avoiding the classical step of data segmentation and meshing along with its inevitable artifacts. Visualization in a virtual reality headset together with interactive hand-held pointers allows the user with to interact rapidly and flexibly with the data (highlighting, selecting, slicing, zooming etc.). This natural interface can be combined with semi-automatic data analysis tools to accelerate and simplify the identification of relevant anatomical structures that are otherwise difficult to recognize using screen-based visualization. Practical examples of this approach are presented from several research projects using the lightsheet microscope, as well as other imaging techniques (e.g., EM and 2-photon).

Abstract of Journey to the centre of the cell: Virtual reality immersion into scientific data

Visualization of scientific data is crucial not only for scientific discovery but also to communicate science and medicine to both experts and a general audience. Until recently, we have been limited to visualizing the three-dimensional (3D) world of biology in 2 dimensions. Renderings of 3D cells are still traditionally displayed using two-dimensional (2D) media, such as on a computer screen or paper. However, the advent of consumer grade virtual reality (VR) headsets such as Oculus Rift and HTC Vive means it is now possible to visualize and interact with scientific data in a 3D virtual world. In addition, new microscopic methods provide an unprecedented opportunity to obtain new 3D data sets. In this perspective article, we highlight how we have used cutting edge imaging techniques to build a 3D virtual model of a cell from serial block-face scanning electron microscope (SBEM) imaging data. This model allows scientists, students and members of the public to explore and interact with a “real” cell. Early testing of this immersive environment indicates a significant improvement in students’ understanding of cellular processes and points to a new future of learning and public engagement. In addition, we speculate that VR can become a new tool for researchers studying cellular architecture and processes by populating VR models with molecular data.