July 10-12th, my friend Kevin and I participated in a VR Hackathon. The theme was Mythos & Moxie. On the first day we were inspired by Ken Perlin’s talk about thinking 10 years ahead. Trying to do now what you expect in the future. Then we saw work that was being done with 360 video, special hardware and software being used to create 360 video - as well as a new video platform online for viewing this 360 video. It immediately caught our attention and we started asking ‘what is virtual reality?’. What constitutes VR? Can VR be a sphere of video that you’re watching? Or do we want something more?

A bit of background on us: Kevin and I come from a background of 3d, working with games, television, and architecture. So we’re very keen on what is ‘3d’. To us, virtual reality needs to be three dimensional. There needs to be an ability to have depth perception through parallax shift.

Kevin put together a pitch of our idea that we presented at the hackathon. I’ve transcribed it below:

Virtual reality it's in it's infancy. During this time different experiences are emerging that are profoundly different, yet exist under the same label. We want'ed to define for ourselves - 'What is virtual reality?' 'What does that mean?' - If I look at a projector’s projection on a wall, Is that virtual reality? No? If I make the projection wider, is that virtual reality? Higher? What if I enclose my body in a sphere of projections? Is the definition of virtual reality that you simply increase the field of view of the camera and your perception? We contemplated this and we came to the conclusion that for us, the difference between immersive, awesome 360 degree video, and virtual reality, is depth. Depth allows you to be in the space. To have this virtual existence, this virtual reality in which you exist. This is your reality. This is where you are. If you move your head, you change your perspective, you change not just viewing angle of the camera but you change from which side the camera looks at the subject. It's three dimensional. You can walk around subject.

This unfortunately is mathematically impossible with traditional forms of video. The way we capture video currently is always from a single pivot point. But what is video? Video is a series of images we call frames. But these frames are two dimensional, they are flat. So your perspective can not change. Now if you made a frame this is three dimensional, that would allow you to explore that frame from any angle you like, and if you had a series of these frames, you could have live action three dimensional video and thus you have a three dimensional space in which you can exist. To generate three dimensional frames is a bit of a challenge. But ladies and gentlemen, today, Brendan and I have brought you 14 of those three dimensional frames. Looped. Over a time period of about 1 second, in which you can explore this three dimensional space, in which you can move around the subject, look at it from any angle, crouch down, touch the floor, feel the experience and be in the space. We invite you to explore your first virtual reality live action scene.

We tried to emulate how we hope the future of live action capturing will be for VR. We see what we did as having the same feel as very early video recordings.

For our experiment, we manually captured the space with an outside-in approach. Using a camera we took 35-50 pictures of the space to process each ‘frame’ of the animated sequence. This required the motion to hold for each frame as it was captured. Each frame was then processed into a 3d mesh. And then scaled and lined up to step through and view. The viewer has an untethered view into the space, being able to crouch, lean, and even walk around the subjects in the space. Although people have already been exploring photogrammetry for VR - animating it is a crucial next step.