Looking Around in 3D

I’m from game development. There’s a lot of focus there on improving graphical fidelity, frame rates, and photorealism. And now with the rise of virtual reality, throw into the mix: head & hand tracking, haptic feedback, and other various devices to imitate reality reality.

I have a few aesthetic objections to photorealism – of all the visual styles possible with a computer screen, a digital canvas totally under your control, you choose… to imitate the look of the world that’s already all around us all the time? That just seems like a lack of imagination and a waste of potential. But above all…

Surface-level “realism” does not last.

I’m reminded of the old, old 1896 film, Train Pulling into a Station, a 50-second silent movie where a train moves towards the camera. Yeah, that’s it. As urban legend has it, when audiences first saw the film, they panicked and fled to the back of the room. But of course, with time, we all got used to it. Same thing once film had sound, color, special effects, HD, and even 3D. We all get used to it. (in the case of 3D movies – for many guys including myself, bad 3D sickens me and undos any immersion I had.)

I can’t help but feel that the current state of virtual reality is like Train Pulling into a Station. Sure, it’s impressive now, but soon the surface-level stuff won’t matter. But think about what film theory is today! It’s not about how to create the most spectacular photorealistic scenes. It’s about the framing, the editing, the everything that’s done in service of visual storytelling.

Surface doesn’t last. Story does.

It’s the classic “form over function” mistake.

I guess it’s a problem all new artists & mediums go through. At first, everyone can only see the surface-level stuff, because, well, it’s the surface. It’s the first thing we notice. And all of us go through this stage – I know I’m always going through it, as I try to learn new art forms and mediums.

I hope this post doesn’t come off as dismissive of Virtual Reality. On the contrary, I’m pretty excited for it, and I’ve experimented with it myself! So while it’s good we have continuing research into graphics performance and motion control precision, I think that’s all surface, and instead, we should be thinking more about how all this new technology lets us better communicate stories.

For example, in VR, the viewer controls the camera. How does this constrain us, and how can we use this fact to enhance storytelling?

When films got color & sound, it took us a while to make full use of these new additions. Sure, first we replicated reality, but then as the medium matured, we learnt to manipulate lighting and do musical scoring to convey moods, to better communicate the stories.

So what’s new in VR and VR games?

Viewer control of camera

Spatial awareness of a scene

Depth perception

Direct manipulation

Kinesthetics

Interactivity

Eye contact

…and more. I’m excited to figure out what new functions these additions can serve. I wonder how they can help us tell stories better, and with luck, tell better stories.

Immersion is a narrative problem.

And in hindsight, it’s kind of obvious, isn’t it? If a skilled writer can make a bunch of ink symbols on dead paper feel immersive, that’s a strong testament to the power of story.

I wonder if all that money and research going into higher resolutions and higher framerates, in an attempt to solve the problem of “presence”, couldn’t just be solved by looking below the surface, and looking for better stories.

Coz if you think “looking around in 3D” will still be impressive in coming few years, I got two words for you: