It's the Wild West in the world of virtual reality and the rules are being written and scrapped in real time.

Glixel

The virtual reality game Accounting is exactly the sort of surreal experience you’d expect from a collaboration between one of the creators of Rick and Morty and the guy behind the mind-bending game The Stanley Parable. Strap on the HTC Vive VR headset, and you are transported into drab offices, idyllic forests, and dank dungeons. You interact with objects using a vintage pointer icon straight out of Windows 95. All the while, you’re screamed at (and hit on) by a ghost, a corpulent king, and a xylophone made of human skeletons. It’s just a hint of the potential for literally anything to happen in this immersive new medium.

Amid all of this lunacy, the creators of Accounting were faced with a very mundane problem – how could they make the player look where they want them to look? In traditional games and movies, that’s been easy – you just point the camera in the direction of the thing that you want your audience to see. But if your audience is wearing a head-mounted VR display, they can look in whatever direction they choose. How do you direct the player’s attention to a specific spot in VR?

This is among one of the many fundamental problems that VR content developers are struggling to solve. They are having to learn the rules of a completely new medium, and unlearn some of the rules they’ve carried over from others.

We’ve been here before. Every new art form goes through an initial awkward phase of trial and error. When film was new, most movies were stagy and static, with the camera observing the action from the vantage point of someone sitting front row center in a theater – the best seat in the house. Filmmakers gradually learned to use camera positioning and close-ups and lighting and editing to heighten the drama.

Similarly, when video games first made the jump to 3D, game makers struggled with how best to let players look around and navigate the game world. (The original 1996 Tomb Raider‘s tank-style controls have aged even worse than its crude, lo-fi graphics.) But developers gradually learned how to make moving your avatar and the camera intuitive and satisfying.

That’s the point that a lot of VR game developers are at now, according to Shawn Layden, the chairman of Sony Interactive Entertainment Worldwide Studios. He’s had a front row seat, watching games being created for (and ported to) the PSVR. “Early on, some of the games will seem kind of like a standard console living room experience, but you’re just doing it in a really immersive fashion,” says Layden. “Eventually, there will be games that will only be fully realized in a virtual reality headset, and won’t translate to the TV screen.”

Related

Layden compares the process of figuring out what works and what doesn’t in VR to learning a new language. “We talk about grammar and the syntax a lot,” he says. “We’re starting to create the lexicon of VR to describe these things, because it won’t be the same as the words we use to describe current experiences.”

So how are VR game developers learning the vocabulary of their medium, like making the player look in a certain direction?

In the case of the game Accounting, the answer was surprisingly simple. “When we need players to look somewhere, we just make a phone ring, or we make a baby cry,” says Justin Roilland, co-creator of Rick and Morty, and co-founder of VR studio Squanchtendo.

Timoni West is the principal designer on the Labs VR team at Unity – a code and development platform that powers more and more of today’s games. She thinks that’s smart.

“I see a lot of developers using audio cues,” she says. “You can’t control where people look, but you can use directional audio to whisper in someone’s ear, or trigger a loud noise that seems to come from behind them. That makes people instinctively turn around.”

Unity aims to be the go-to development tool for people designing virtual reality games for computers, consoles, and mobile devices. West’s Labs VR team at Unity is trying to provide everything developers need for the profusion of VR platforms, like a wide variety of input sources – physical controllers with buttons as well as hand gestures and the direction of your gaze. She’s also noting the best practices that seem to be emerging.

West sees all sorts of examples of game developers taking advantage of the potential of VR to create uniquely intense moments, like Insomniac’s Lovecraftian chiller Edge of Nowhere. “At one point, you hear this creepy voice behind your head, and from the corner of your eye you see these, these tendrils coming at you,” she recounts. “I watch people playing that, and they’re like, ‘Nope, nope, nope …'”

But she also sees projects that don’t take full advantage of the medium. For instance, many VR content creators fail to fill the full 360 degree field of view with interesting stuff to look at. “I once saw a VR experience, a sort of trailer for a cop show, that had characters standing around and talking to each other,” says West. “But these were TV actors who were trained to not move or do anything to draw attention to themselves when it’s not their turn to talk. That works in TV, but when you’re watching it in VR, it’s like you’re surrounded by a bunch of inanimate mannequins. The technique just doesn’t transfer.”

Another dilemma: How do you portray the player’s body in VR? If you look down in real life, you see your torso, your arms, your legs. Should you model a body in virtual reality?

Many games simply show the tools that the player uses. For example, if you hold up the controller in front of your face real life, you see a gun hovering in mid-air in the same spot inside of the VR action game you’re playing. Other games show virtual representations of the actual controllers that the player is grasping, or simply a pair of disembodied hands that they use to interact with the game world.

West advises designers to keep things like this simple, as anything that doesn’t match up with what a player expects to see can be very disconcerting. “I was playing this VR game Asunder Earthbound, and I could see my hands moving around in it. But they were modeled to look like the hands of an old person,” says West. “My reptile brain was instantly like, ‘Stop everything! What’s going on?! Are you sick right now?’ It completely pulled me out of the experience.”

This is a particularly tricky issue, because many have suggested that VR games have the potential to be “empathy engines,” something that can literally allow you to see the world through someone else’s eyes. West says that VR developers are discovering that they have to give players time to gradually orient themselves to having a new body. “You have to acclimate to being someone else, someone who’s taller, or is a different gender,” she says. She compares it to the moments spent settling into your seat at the movie theater and watching the lights dim before the show starts.

Simply moving around in VR presents even more conundrums. If your eyes and ears tell you that you are racing across a scorched field in a post-apocalyptic wasteland, but your other senses tell you that you are kicking back on the couch in your living room, it can undermine the illusion. Or worse – it can lead to motion sickness.

“Physics systems are a huge issue,” she says. We take it for granted that gravity doesn’t affect Mario in quite the same way that it does normal humans, but that floatiness can be disorienting when you actually feel as if you’re inside Mario’s world. And what about sci-fi games set in zero-g outer space?

“Moving around freely in VR is not a big deal for me, but I meet people who are opposed to free motion on a philosophical level.”

“Trying to get the physics to work in a way that’s not like what we have on Earth, but that is still predictable and doesn’t lead to crazy glitches, is a big challenge,” West adds. “A big but interesting challenge.”

The potential issues of moving around in the game world multiply for VR games that get you up out of your seat. Even if you’ve cleared a large play area before firing up your game, you can’t walk in a straight line for long without bumping into a wall or a couch or a bemused significant other.

Many VR developers are using a teleportation mechanic as a workaround. For instance, Epic’s Robo Recall has you slaughtering an army of malfunctioning automatons by popping from place to place like Nightcrawler from X-Men. Other games like Obduction, by the creators of the Myst franchise, gives players the option of jumping between “nodes” or moving freely through the environment.

“In the game Land’s End, by the team that made Monument Valley, you can teleport between points by looking in the direction you want to go,” says West. “It’s very beautifully done. Moving around freely in VR is not a big deal for me, but I meet people who are opposed to free motion on a philosophical level.”

One technique being explored by VR researchers is redirected walking. By subtly shifting the point of view, they can make users feel as if they are wandering freely through a large environment when they are actually walking in circles in a circumscribed area. “You still have to be in a pretty big space for that though, something like the size of half a basketball court,” warns West.

What will a mature VR experience look like, once the medium has come into its own? Sony’s Layden suggests that it might evolve into something in which the current challenges will all seem quaint. “In gaming, you’re typically objective driven; you always need to go get that thing, or go to that place, or achieve that goal or destroy that thing,” he says. “But what if I don’t feel compelled to move the way I would be in a traditional game? What is progression like in a virtual reality world where you can just stand there to experience it? VR just completely blows up the whole traditional idea of narrative.”

West, meanwhile, predicts that the evolution of the art form will slam into overdrive once game developers stop using mice and keyboards to make VR titles, and start using using content creation tools that actually utilize VR. “At Unity, we’re working on something that lets you sculpt a 3D world around you,” she says.

She predicts another explosion in creativity when these VR content creation tools filter down from professionals to the general public, citing what people are already making with Google’s TiltBrush, a simple VR app that lets you paint in 3D. “The things people are making with Tiltbrush are just crazy,” says West. “They’re doing stuff that Tiltbrush was never designed to do. It’s like they’re using Microsoft Paint to make a Pixar movie.”