Virtual reality comes to medicine

From $20 Google Cardboard to high tech $Millions

by TILKE ELKINS  February 2017

Is 2017 the year of virtual reality (VR)? A plethora of techie commentators and journalists insist that it is. That goes double for the medical field, where the many uses of VR — surgical training, trauma treatment, corpse dissection, patient education, and more — have been widely touted. But how much VR are doctors actually using? What most people imagine when they hear mention of the technology is a high-end version: a fully immersive experience involving headsets, haptic sensors that give the wearer a variety of touch sensations, and a fully computer-programmed set of graphics that can be explored in three dimensions. Given its hefty price tag, this form of VR has yet to be used outside large institutions. But lower-tech versions of VR, like Google Cardboard, which attaches to a smartphone, are available for as little as $20. Simple devices like this have already made it possible to perform medical breakthroughs. In a recent case, a team of surgeons was able to save the life of a baby with a serious heart defect by examining the organ with the Google device.

To get a sense of how the technology is evolving from the inside, DR spoke with Meredith Osborn (meredithosborn.com), a professional medical illustrator who creates both 2D and 3D animation.

DR: What’s your understanding of the difference between augmented reality (AR) and virtual reality (VR)? Are physicians using one more than the other?

Ms. Osborn: In the case of augmented reality you’re interacting with the real world, but with some digital enhancements. Pokémon Go is a great example of augmented reality: looking at your phone screen you see what is really in front of you but augmented with a little digital Pokémon bouncing on the screen. Virtual reality is different. It is a completely immersive alternate reality. You’re “in” the game, the simulator, the whatever (often by wearing a headset and holding controllers). Physicians are using both AR and VR.

DR: How well-informed do you think most doctors are about the uses of AR and VR tools in medicine?

Ms. Osborn: Like any new technology, it will take time for everyone to become fully informed. As for AR and VR, it appears to me we’re still in the very early stages of applying it to medicine. There is some truly fascinating work being done, but mostly in artists' studios, engineers' labs, and any medical school or program that has groups lucky enough to be in testing and early adoption.

DR: Based on your personal experience in medical contexts, how do you see doctors actually using these technologies today?

Ms. Osborn: In my experience, very few are actually using it. Those that do are mostly using it in medical education as opposed to patient practice. For example, having a teaching mannequin, but with the internal anatomy accurately projected onto the skin so that students learning a procedure can understand the relationship between what they’re doing and the underlying anatomy that’s being affected. Or virtual reality where you are performing a procedure and the controls give haptic responses appropriate to the textures of the different tissue types -- if you hit bone, you feel it. Or you feel the slight give when you’ve broken through the virtual skin. And since it's virtual reality, you can choose to display whatever underlying anatomy you want — or don’t want — depending on your level of mastery.

DR: How much about AR/VR is just hype? In other words, do you think that the industry has made claims about the technologies that will not reach fruition?

Ms. Osborn: AR/VR is very much the cool kid on the block right now. There is a lot of genuine enthusiasm about potential that will be realized in the next decade or so. A particular point where I feel it crosses into unsubstantiated hype is any claim to completely replace unaugmented boring reality. A great example is cadaver dissection. Digital cadavers exist, and there are probably a few programs that have completely replaced physical cadavers with them — and saved huge amounts of money by doing so. Digital cadavers can be re-used again and again. And digital cadavers with haptic feedback can be programmed to ‘feel’ like living tissue as opposed to the preserved texture of actual tissue. So they are useful. But I have played with digital cadavers, and I have spent hours (and hours and hours) in the anatomy lab elbow deep in actual cadavers… and there is something about the real experience that is irreplaceable to me.

To get a little philosophical about it, medical knowledge will always be tactile and physical. It’s not the sort of knowledge that just lives in your head. It’s also in your fingers and nose and eyes. Though AR and VR are sensory experiences, digital sensation and real sensation are not the same thing. Both can provide valuable learning opportunities, but I do not think any claims that AR/VR will completely replace reality are substantiated.

DR: When it comes to benefits to doctors and patients, what technology do you think has the most potential to impact the practice of medicine?

Ms. Osborn: Clearly the training tools available through AR and VR are powerful, but what I think has the most potential is a completely different approach -- using VR to teach empathy. For example, Carrie Shaw with Embodied Labs created a VR experience that simulates the experiences of the elderly. Using the simulation, you become Alfred, with his poor eyesight and hearing, his occasional wandering into memory-land, and you see and feel what that is like (https://www.youtube.com/watch?v=pOW7oG6bIFI). One scene in particular, you are sitting in your doctor’s office having been given a cognition test. But you couldn’t really hear the instructions and can’t read the paper. You fail the test. Now, perhaps Alfred really is experiencing cognitive decline but his physician doesn’t know because he didn’t realize that Alfred could not properly hear the questions or read the instructions. I appreciate that scene in the VR experience because it demonstrates how empathy is more than just better bedside manner; it can affect outcome of assessments which, in turn, affects treatment decisions. The power of stepping into another’s shoes is impossible to hype, and VR can literally put you there.

DR: Do you think that 2D illustration will continue to remain relevant in medicine, or do you see 3D imaging supplanting it entirely?

Ms. Osborn: Great question, and the answer is no. 2D won’t ever go away. In addition to the fact that I don’t think print media will ever go away completely, 2D is simply much faster to produce. So if a 2D illustration will get the job done, why waste precious time and money on going 3D with it?

DR: As an artist, do you think that there’s as much context for personal expression in 3D formats as there is in 2D? How exactly does that manifest? (i.e. choice of colours, shapes, movement, etc.). Do you have a preference, 2D or 3D?

Ms. Osborn: I’m a commercial artist (as opposed to a fine artist) so I care less about personal expression in my work. My work is intended to communicate a message or convey information that meets the needs of the audience. That said, I understand what you’re getting at, and I do think the answer is yes, 3D can hold just as much personal expression as 2D, the same way as working with clay can hold as much personal expression as working with paint.

DR: Have you met with any creative challenges in learning to use advanced imaging technologies?

Ms. Osborn: One of the beauties of art is that anyone can pick up a pencil and draw. But that is definitely not true for these advanced imaging technologies. The learning curve is steep and you really should pay someone to teach you — whether that’s a traditional for-credit academic environment or less traditional online experience. Like any art form, you have to learn the tools before you have an opportunity to be creative in your use of them. That’s the biggest challenge for 3D, just learning the tools.

DR: What would you like to see in the future in how AR/VR are used?

Ms. Osborn: I’m most excited about what will happen once patients get their hands on this tech. Imagine a clinician holding up an iPad and looking through the camera at the patients arm, but with the underlying anatomy showing so the patient can see as s/he explains their condition and treatment. Or — though we all love to imagine a world where clinicians have that kind of time — handing the patient the iPad with a self-guiding educational app including that AR experience. Or imagine a geriatric office having VR headsets for people in the waiting room. Caregivers could go through Carrie’s Alfred simulation just for fun and find themselves with re-invigorated empathy.

This article was accurate when it was published. Please confirm rates and details directly with the companies in question.