Challenging Art with Augmented Reality—How Facebook Invaded My Studio

Heather Day

At the F8 conference Tuesday, Mark Zuckerberg shared the world’s first augmented reality artwork for the Facebook Camera—an installation created by me.

Making the installation opened me up to working more closely with technology--and thinking of augmented reality as a new medium. The video below introduces the concept and shows the finished work that Mark presented.

As satisfying as the finished work feels, the process of actually creating this multidimensional piece interests me just as much. It began and ended lightning fast. Facebook approached me about collaborating on a secret project and within days their Applied Machine Learning Team invaded my studio. In one afternoon the team filmed all my mark-making techniques—from spray painting to brush strokes and manual paint pouring.

The Facebook crew set up quite a few cameras to capture every moment of the painting process.

We then worked together to animate the marks and collect them into a digital library. To formulate the installation, I had to work backwards—pulling animated marks and then explaining the composition to Facebook’s animators. They were involved in every step of the creative process.

Video chatting and screen sharing with Andrea and Andreas from the Facebook team. Here we were confronted with the challenge of simplifying what we had created.

Initially, I took iPhone photos of the space where the piece would be projected and then drew on the photos with a Wacom tablet with the digital marks. But to create a living, breathing augmented reality installation, the animators and I had to communicate and work on color, composition and movement together. We shared screens, Skyped and emailed ideas back and forth to make final pieces come together—it was a true collaboration. The finished installation can be seen at Facebook’s headquarters in Menlo Park, CA.

Digital drawings I made on my iPhone with a Wacom tablet prior to the studio shoot. I created 5 of these sketches to help us understand what we were going to film and create.

Creating art through augmented reality represents something that I’ve always imagined doing—but never had the tools to try. Since I was a kid, I’ve imagined marks that moved. I imagined taking colors from around the city and making marks on buildings, streets, houses, signs and fire hydrants. These marks had a rhythm and pulse. I imagined them leaping off walls and following me around—tripping over themselves to stain the sidewalk by my apartment, bouncing off my bathroom walls as I brushed my teeth—they made the entire experience of living a work of art.

Augmented reality makes this possible. It merges tech and art to create new opportunities, new possibilities. For the first time, someone in tech may be considering how to enrich a project through an artist’s work. And artists can begin to imagine how tech can take their work to another dimension—literally. This is how I want to challenge the future of art.

Taking augmented reality as a new medium, we can challenge the definition of making and experiencing art. What will a museum visit be like in the future? Or going to a gallery? What will being an artist look like? My experience with Facebook allows me to begin answering these questions. But it poses so many others.

Special thanks to the talented Facebook team I had the great privilege to work with : Kristen Spilman, Andrea Ho, John Barnett, Andreas Berner, and Joshua C Harris.

Keep scrolling to see more photos.

Another digital drawing I made on my iPhone with a Wacom tablet prior to the studio shoot.

More from our setup. The filming was captureed so seamlessly thanks to that giant crane you see here.

The first batch of paint was poured, and I started to warm up. It was a bit odd having a whole film crew watch me work but it got easier as the day went on.

It was important to film directly under the camera and try to keep my body out o the way. There were a lot of pauses to make sure we captured every angle.

The final product of the painting themselves were actually not very interesting. It was more important to capture my marks in a wide range of movements. The color could later be altered digitally.

Working with animators to bring the marks I created in the studio to life.