For the past three years, Google Spotlight Stories has been enlisting veteran animators to help evolve storytelling software from Google’s Advanced Technology and Projects group (ATAP) to create interactive 360-degree and virtual reality films for mobile devices, YouTube 360, Google Cardboard, and VR headsets, as well as traditional 2D.

The collaboration enabled animators to experiment with a new form of storytelling while test-driving the ATAP software. Since the first of those films—Windy Day from Spotlight Stories creative director Jan Pinkava—came out in 2013, the program has rolled out an additional eight shorts that work both as single linear and interactive stories. When users move the phone to various scenes, phone sensors enable interactivity by unlocking multiple subplots and endings.

This month welcomes the last and most complicated of those films, Rain or Shine, about a girl with magical glasses, from London-based director Felix Massie and Nexus Studios. With this leg of the collaboration completed, the ATAP team is now looking toward the next phase—continued work with animators to adjust the software with an eye toward eventually releasing it as a production toolkit with user-friendly interfacing. It hopes to present updates at the Google I/O developer conference in May.

Karen Dufilho

"The goal is the create a story development kit that works as a story editing plug-in, stitching scenes and effects for any animation program, not just Maya, and creating movie production values for mobile devices," says Google Spotlight Stories executive producer Karen Dufilho. "This needed the back and forth to learn what the software needed to be. We can’t complete software until artists really put it through the production pipeline and test it, break it, and get frustrated with it, and then we can go back and iterate on it.

"From a conceptual standpoint, it’s constant learning—do we think this is going to work, let’s try it, did it work, what do we have to adjust, how does that change our conceptual thinking moving forward," she adds. "From a technical one, the tools are getting more production ready, but it’s not off-the-shelf software yet, so we’re continuing to put it through these pretty significant projects. This is new real estate for stories, and that’s what continues to motivate us and the technology."

Studios had to rethink every aspect of filmmaking—from scripting, staging, to art direction and lighting—opening the door to new methods of storytelling and future markets utilizing artificial intelligence.

"Our story uses 'gaze-based interactivity.’ We use the data of where you are looking at to adapt and change the experience," says Nexus Studios chief Chris O’Reilly. "Key story events 'wait' for you to witness them before they happen so that we can ensure your engagement in the narrative. Other events are transformed by how much attention you pay to them. For example, if you spend more time watching the folk outside the pub drinking, the more inebriated they become."

O’Reilly expects the technologies of adaptive, interactive visual storytelling to find a marketplace in an increasingly intuitive digital landscape. "These will have applications not just in VR," he says. "We are moving towards an AI era and this will need a face. Who better to create this than animators at the forefront of exploring real time?"

The animators Google enlisted worked in a wide range of styles and approaches to put greater stringent demands on the technology and offer feedback for further development. The challenges worked both ways. The animators were forced to think in continuous 360-degree landscapes to allow viewers to wander before subtly guiding them back to the main story. At the same time, they needed a version of that narrative to work in traditional linear form. The software developers needed to continuously upgrade the technology to achieve what the filmmakers envisioned, regardless of animation style.

"Duet was hand drawn and On Ice is CG, but live under the same roof of conceptual thinking in that it’s 360 and immersive," says Dufilho. "The tools we created, and gave to [On Ice director] Shannon [Tindle], didn’t really exist before we learned from Duet what those tools needed to be."

It was a steep learning curve. "At first we didn’t understand the implications of what it meant to not have scene cuts, or a framework for your design composition, to have that freedom turned loose to anybody," says Duet production designer Max Keane.

"If I was going to make this a traditional short, we’d have a render farm, bake it all in, and you would just play it," says Tindle. "But we’re rendering in real time. That’s why we talk about things that are expensive on the GPU (graphics processing unit). I want a character with a complex facial rig, that starts to eat up resources, and you start to drop frame rates. But you don’t want to do that, because you want the best experience possible."

As the artists began figuring out their narratives and specific looks, they’d request the programmers to achieve that on the tech side. "I would ask for what I needed story-wise and they would develop the story development kit to give us those things," he adds. "So we had a set of tools, which has grown as we’ve been throwing things at it."

The biggest challenge on the technical side was adapting existing phone sensors and graphics to this new task. "It took a lot of work to make it seamless and smooth, so people felt like they were looking through a window into a virtual world," says Spotlight Stories’ technical project lead Rachid El Guerrab. "We were bringing in film people used to high-quality animation. Could we give them the ability to do, in real time, what they’re used to doing in their studio with big render farms and big servers? There’s a lot of technology that we had to write that you would do normally in a real-time game engine. We had phones that we had to put in the fridge, because they were processing too much and would get too hot and drop performance."

"Our mandate was to break stuff and push it forward," says Pearl’s Osborne. "Computers have gotten fast enough to where the artist’s interaction with the artwork in animation can be closer to the final image as they’re working on it. In features, there’s a long delay between what you do and the final product, and the idea of working in a real-time engine is more along the lines of a game engine. Your cognitive connection as an artist to the work is closer. You’re making choices that are on screen and going to be seen. Then being able to move around. It’s a storytelling language that hasn’t really been figured out—how to guide the audience in a way that’s going to be compelling and interesting, not distracting, and have the story make sense and be emotional."

"When you’re playing a game, you want to have a look around a scene, go down a dead end and see what’s there," says Rain or Shine’s Massie. "This is a cool thing where you can look around and be rewarded for exploring, but also maintain the integrity of the story."

The 360-degree format was as vexing musically as it was visually. Without specific edits moving the story along, lacked the necessary timing elements. A traditional narrative form has set times between scenes, instead of the new format’s random amounts that depend upon the viewer.

"You have these different scenes, and it’s the music’s job to connect them," says Spotlight Stories composer Scot Stafford. "How to you make a good experience for people not knowing how long they’ll take or where they’ll look? The first thing you want to know as a composer is, 'How long does it take to get from here to there?’ and the answer is, 'You’ll never know.' Adapting to that and being able to help these guys tell their stories was very challenging."

"Scot didn’t just repeat a cycle of the music, either," notes Duet’s Glen Keane. "He rewarded people going off into other areas, and created something that encouraged exploration.

"I always wanted to just draw; animation came to me," adds Keane. "But as we transitioned into CG, drawing became that much more precious to me. Ironically, leaving Disney to go to Google was a way of rediscovering drawing and expression, and animation happened in the most cutting-edge technology."

Never miss a story.

Sign up for the Co.Create newsletter and get the best stories delivered to your inbox daily.