WALKPro3D: wearable mobile projection mapping

Jun 9, 2016

What if you put the power of video projection mapping in the hands of a performer? That’s what Shaun Prickimage is doing with Walkabout Projection and WALKPro3D. Shaun and Jacob Stone discussed wearable projection, gesture control, and painting bodies with light in this Skype conversation.

#WALKPro3D is the first of its kind, a small and mobile HD video projection system that can bring digital puppetry and 3D character performances to any location, as featured in ‘Don’t Think’, the film of a live performance by The Chemical Brothers.

Manipulated magically by Leap Motion & Unity 3D gaming engine.

No need for cables or screens, WALKPro3D has been used for various branded marketing and charity events, clubs, live music, and in video within arts and fashion contexts. An ingenious and hypnotic form of puppetry, magic lantern and phantasmagoria for the digital age. These magic digital interventions generate an instant buzz of excitement in the crowd, encouraging selfies & sharing on social media.

TRANSCRIPT OF SKYPE CONVERSATION

Between Shaun Prickimage of Walkabout Projection and WALKPro3D and Jacob Stone of Punch Drunk Productions

Jacob:Jacob Stone here from Punch Drunk Productions. I am sitting here with Shaun Prickimage from Walkabout Projection over in London. Shaun is doing some really interesting stuff with mobile projection, wearable projection, and then injecting that into environments at events. Shaun, I’m wondering if you could just start off in a nutshell, what exactly you’re doing with Walkabout and describe it?

Shaun:Walkabout Projection is taking small projectors and using them for performance in creative ways. Our latest highlighting performance is the WALKPro3D which is using hand gestures. So we’re only using the five fingers with the Leap motion sensor to control characters, very much like a digital puppet, a marionette.

Jacob:Back up for me with the concept a little bit for anybody who isn’t familiar with what’s going on. In a normal event you probably have a LED wall, or a projector on a screen or something with video elements, but Walkabout, you’re completely going away from having a set screen and a set place right?

Shaun:Yeah, absolutely. I mean, we love the idea of taking the image, the character off the standard screen in a static location. Basically, the video game can escape from the bedroom and go to the street. We take a battery powered projector and the little computer that we use and the sensor. It all runs off a battery so we can go anywhere.So the world becomes your canvas.

Jacob: That’s so cool. So rather than being tied to something that’s part of a set piece or part of a regular stage, you can take the projected image put it anywhere, anywhere in the venue. Of course I know a little bit about what you’re doing because I watched some of the videos but talk about some of the actual things you’ve done with this. Some of the events or some of the videos that you’ve been involved in with this. I’d love to hear some of the anecdotes.

Shaun:I guess the first real exposure and really good experience we had was in the Chemical Brother’s film.

That was a few years ago and they spotted me running around a few festivals like Glastonbury. So they really thought it’d be cool to help heighten the senses, the sensory experience of the film. So at Fuji Rocks in Japan they filmed the show and turned it into a big cinematic super experience that was on general release. I saw it at Lester Square, you know, luckily with the crew and stuff like that, so it actually turned into a bit of a party.

I saw it a couple of times. That was really good that we were able to help bring the visual from the stage into the festival, and they had film crews running around capturing all that.

Jacob: In this particular case with the Chemical Brothers, what were some of the things that you’re projecting on? Are you projecting on to people? Are you projecting on walls? Paint the picture for me a little bit.

Shaun: That’s something that they saw me doing originally and they really wanted to take that and use those elements. I’m projecting on the floor, on the ground, in the dirt, projecting on people, projecting on tents, you know, basically any surface becomes your canvas, and you can do it on large scale and you can do it on small scale. They originally used one of my bigger projectors, but now we’ve moved into the more smaller scale ones so now we can actually get really kind of intimate small little pieces happening also.

Jacob: Before when you and I have talked, you were talking about smaller more intimate situations and equipment, and one of the things that you said when we were talking a couple weeks back, or a couple months back, was that you’ve involved performers in the video element rather than just say like a VJ or a video producer. And I thought that was really interesting because traditionally you’re going to have a VJ who’s very familiar with the equipment, and very familiar with kind of nerdy stuff, doing VJ stuff at an event, or a video playback tech, or a graphics tech, but I think you’ve gone a step in a performance direction, right? Where you’ve been bringing people who are actors and dancers and outfitting them with some of the projection gear. Isn’t that right?

Shaun:Yeah, yeah, absolutely. I think very early on we were totally amazed just with the fact of being able to move the projected image around. I guess that’s like the early days of film when people were just amazed to see anything on a screen.

We we kind of quickly wanted to progress things and once I started using characters, the ball starting rolling that we wanted to make them feel more lifelike. We want them to have a life of their own and not just be a VJ or a technician dragging them around the room.

So we wanted to really give them some kind of character, and breath, and gravity. So the best way to do this that is to really use people who are comfortable with their bodies and comfortable with surroundings. So it was kind of a natural progression to to work with dancers, and performers, and even even actors. We did some puppetry workshops and stuff as well just to get people comfortable with how to move and how to control the characters.

Jacob: It seems like this could hypothetically be used in a performance of some sort and go from the stage out and back in again. Bringing animated digital elements outside and diffuse them into the audience.

What’s really interesting, some some of the stuff I’ve seen you’ve done is where you’ve put it out in the street. Not necessarily even in an event, right? I would imagine at a music festival or something where you’ve got all sorts of regular public roads being able to all of a sudden insert animated things happening out there in the public and in normal spaces right?

Shaun:Yeah, absolutely. Both my colleagues, we put together some some great ideas that would work amazing in festivals. A couple of them being having multiple performers, like maybe three performers. With a single character. It could be a sheep, or you know, some something that’s kind of of local or kind of appropriate, or it could be a shark, and they could all just follow each other in kind of like a flocking synchronized kind of movement.One person doing this is kind of interesting. When three people do it, and have some choreography or follow each other, then it becomes a performance.Then it becomes quite powerful. And it can be very fluid, especially if they infuse in performers. Some people are comfortable with surroundings and don’t mind weaving through people, interacting with people, then it can be very fluid in a real performance.

They can be really choreographed, especially if you’re trying to get a message across. Our aim is to really evoke some kind of feeling or convey a message that is appropriate for the surrounding or the event.

Jacob: Talk a little bit about the gesture based control you’ve got now, that’s something that’s newer I think, right? That wasn’t part of the original system? Got the performers who are able to hold in one hand a projector and then with the other hand, without touching any controls, just move their hand around in space and control what the projected image does, right?

Shaun:Yeah. I mean we really pushed this forward. I think it’s almost a year ago now a dance theatrical performance. We managed to get some funding with the Cory Baker Dance Company, and it just happens they were doing a New Zealand traditional story with the Māori myth.

I’m from New Zealand originally, so it felt really nice to be involved in something like this, and we created some indigenous creatures, and we brought them to life. As you were just saying before about working in a theatrical environment, coming from the the stage and into the audience, that’s basically exactly what they did at the beginning of the performance. The dancers were moving amongst the crowd and then moved back on the stage and then the show started.

Immediately as you arrived while people are milling around and fussing, you have the the performers totally immersing the crowd. Right from the very beginning you feel like you’ve been transported to a different world.

It’s something that I wanted to pursue and this was the perfect time to pursue it, we did lots of rehearsals and testing about what works best. Obviously not everyone can work out hand gestures as quickly as we’d like and especially if we’re we’re working with different performers, we had to work out a simple system that works well with the the sensor that we we were using.

So there was also trial and error, but I think we’ve got, we’re getting it to a really good place.

There is going to be another performance of this dance piece this month. Now that it’s June, so it’s I will definitely go there and document it with at least with pictures or some video, because we don’t have proper documentation of it yet.

Jacob: I feel like that’s always the difficulty when you’re creating new systems like this, the eagerness is there to get it out, and to test it, and to try it, and it’s always a second thought to bring a camera crew or a photographer or anything with you to document it. That’s constantly happening with us. And so, I’m sure you’ve done some pretty mind blowing stuff that you don’t even have on your own Walkabout website. I’ve got your portfolio up here that I’m looking through and I’m thinking, “I bet you’ve done all sorts of stuff beyond that…”

Shaun: Well, it’s also that we’re not allowed to. You know, you’re doing things for maybe a private party.

Or you’re doing things for a brand and you’re not allowed to run around with a camera and document it yourself.

One, because of privacy issues, or two, because they have their own documenting people. Like with the project that we did for Adidas, we had two camera guys and two video guys following us around the whole time, but if you look at the actual piece, the final piece that was shown, we’re only in there for like a split second. (laughs).

And we were there for hours being stalked continuously. The whole piece was only like just over, just a minute anyway, and it was like five kilometers of a run that was happening, so there was so much to show.

But I would have loved to have seen more of that. You know, as the runners come past and interacting them, because they were launching a new shoe. But yeah, what other things have we done? You’ve got Walkabout, and you’ve got WALK3D, so you’ve got the gesture based control of wearable projection.

Jacob: Talk a little, just tell me a little bit of how this came to be? Maybe your background in video and live video and how it lead to this point? I’d be really curious to know, what are some of the things you were doing five, ten years ago maybe that caused the light bulb to light up and say, “Ah ha. Let’s make this wearable.”

Shaun:Yeah, I think it all kind of started off that we saw some artist, I think he was a real artist, in Mexico. I forget what his name was, but he was doing something where he had some motorcycle, or some car batteries, strapped to his back somehow, and you know, projecting a very square, TV like image. When I saw that, I thought, “That’s really interesting but you’ve gone to all the trouble of making yourself mobile. I wouldn’t want to just project, uh, a TV experience.”

I didn’t see much but what I saw he was using it in a very static way. I think he kind of missed the trick. From my experience in working in clubs and events doing the VJ thing, I kind of liked the visual not touching the frame, so I already had that aesthetic in my head, keeping at minimal and not touching the frame so that the visual, lives, you know, almost floats from the screen.

So it was just the case of using that aesthetic, doing some tests, getting the whole mobile situation happening, with car, with motorcycle batteries, which are super heavy. And painful.

Jacob: So you started out with using motorcycle batteries?

Shaun:Yeah, two of them.

And they are heavy. And, and then, and then progressed very quickly, well, not very quickly, but then progressed to professional broadcast camera battery packs.

Around the waist. So we went from Ghostbuters, to looking like Batman. (laughs) And and now we’re kind of, what are we? Now I guess we are kind of Batman again, still Batman because, cause there is a kind of wrist projection thing happening from Batman as well.

Jacob: So, the next step maybe is James Bond, I suppose right? That’s about as svelte as you can get equipment wise.

Shaun:Coming, coming from the glasses, yeah. Something like that, coming from the retina.

Jacob: How long has that that development process been going? When was it that you saw the street performer with the car batteries at first? What has been the chronology up to here?

Shaun: It’s mostly been an art, a hobby art project, you could call it I guess. It’s been a slow burning project. And that’s mainly because of technology.

Pretty much from when I met Piklipita, which I can’t remember the year, it’s probably at least five years ago, he was working with 3D characters, his style was these little cutesy 3D characters, which you’ll see, you’ve probably seen the video of as well. He had these 3D characters and he was making his own operating system with these open source Game Boy type machines. So it was a case of him having the product, you know the characters, and myself having the device and making it mobile, that we kind of started working together. He was still walking next to me controlling the characters while I was carrying the equipment. (laughs)

It’s progressed into making a two man wireless thing. And then it was just like, “Okay well let’s just get it all happening with one man control everything.” So it has been a slow process, but but right from the beginning after meeting him, you know, five years ago, that was the first thing I was saying, I was like, “I want to be able to, you know, take game control, game characters in real time.” But the processing power just wasn’t there, and the size, you know, the size of the projection kits was, was quite big, so you’re spending half your time trying to hold the projector, you don’t really have a lot of scope to do anything else.”

But now, everything’s got a lot smaller, um, and the hand gesture has become available, and the mini-computers, that yeah, it’s just the case of keeping an eye on technology, keeping your dreams alive, and we just jumped on it as soon as it all started coming together.

Jacob: So it’s been about a five year process, about 2010, 2011, something like that?

Shaun:Yeah. I mean roughly.

Jacob: I’m just curious to see, you know when when these things…when ideas like this germinate sometimes they can take such a long amount of time, and I mean, frankly, I’m impressed. Going from car batteries, to a tuxedo, with everything completely on a vest and hidden in five years is pretty impressive.

Shaun:Yeah, I mean, we probably could have done it quicker, but I guess it’s the confidence and the proof of concept that I had to prove to myself and others, that it was a viable project, you you know?

Jacob: So the the characters that you’re projecting, I’d love to hear a little more about those, because I think you’re right, video in general tends to be constrained by the bounds of 16:9, or 4:3, or some kind of a rectangle. That’s just baked into the cake, but these days it’s getting off the leash a little bit, particularly with the the uptick in projection mapping we’ve seen all over the place, right?

Shaun:Yeah, absolutely.

Jacob: Projecting on architectural surfaces, which this seems to be almost a natural cousin of, I mean, it is essentially projection mapping, but just mobile, and you’re doing it on the fly. In terms of not being constrained by the bounds of essentially drawing boxes around things, I’d love for you to talk a little bit about the characters.

What are the characters? You were talking about flocks of birds and some other, I know you’ve got some other kind of little animated characters, what’s the process there? Is that something on a brand by brand bases? Or you said, I think, you have a collaborator you work with that designs some of these. I’m curious, what those are, how they come about, how they would, you know, how we would find them at our next event…

Shaun: We originally called the performance Harajuku Zoo. The whole ethos was these little Japanese anime cutesie characters escaped from a zoo…

(laughs)

And went wild, you know on the, on the streets. Piklipita already had some of these characters created. Once we started testing them, we started realizing that their simplistic form and their bright colors, work very well on lots of surfaces.

We did experiments in changing the colors. Giving them little white outlines to make them pop. So, it was just like a little trial and error basis, and we realized that the style actually does work and people do respond to it. We have commissioned more characters in a similar style for different clients. We are able to lend these characters to different events by changing the colors of them as well.

So it can fit into the color scheme, or make them more glowy like neon, or make them look cold and icy like they’re in the North Pole. They could potentially lend some of our characters, but then also maybe create one for themselves that, that is their avatar or their logo for their company or for the event. So that, you know, then they’re able to expand the whole family in a way.

Jacob:I like that you were able to figure something out that on it’s own makes sense not being constrained by the bounds of video. You know, something that can exist with rough organic edges and naturally move from surface to surface, you know? It makes sense in your mind’s, well not even in your mind’s eye, it makes sense to your actual eye when you see that in an environment, traveling around on the wall, and going from the wall to the soffit to the ceiling, or something. That’s something you guys have done really well. That’s really cool.

Shaun: I haven’t been a huge gamer, but what I have seen of gaming, it’s quite cool for a character to be able to walk, to move, and to turn, so you can go in any direction. So we’ve kind of just been trying to take the core elements of that and bring it to the real world. I’m a little bit gutted that we weren’t in a better position when the movie that came out with Pac Man and all the big gaming characters just recently?

Jacob: Oh, I know the one you mean. I don’t, I don’t remember the movie, but yes, I know what you’re talking about. Pixels is what we’re talking about?

Shaun: Pixels, absolutely, absolutely.

Shaun: I mean that would have been the dream project. We still can do it in a more retro style now. I guess it would feel retro because the movie’s already out.

Those kind of characters would work really well also, kind of the ghosts and all the old school snake and things like that, that would look amazing.

Jacob: Oh yeah. That would be great. I mean, even some of the the 2D scrolling games that like right before, say Wolfenstein 3D or Doom, some of the, some of the the 2D scrolling stuff that’s kind of my era of gaming would be pretty rad to see put into a real scenario.

I mean frankly, Mario Brothers right? Like that style of stuff. Those characters could be walking and running and jumping all over the walls at an event. And I mean I feel like that 8 bit stuff has made a real resurgence. There’s 8-bit music and that style is big I think.

Shaun: Even glitch. 8-bit and glitch go so well together and you could really play with that and kind of blow people’s minds. I mean we’ve been just talking with a client just recently about opening up a new hotel and and engaging pedestrians on the streets. And obviously for them to be able to notice that there’s a new hotel there.

In a kind of interesting way. I had an idea and was suggesting that we kind of inadvertently bring people into a game whether they realize they’re in it…

Or not. By having performers immersing them on the streets. If they chose to stop and take part, or be be aware of it that’s fine. But even if they’re just walking past you can make them part of it by by following them with characters or circling them with a character.

Projecting on peoples backs, or projecting on their front, or projecting on the ground around them is, is, with our little projectors is a very strong image, even with street lights, and even at sunset time. So yeah, rush hour traffic gives us and audience and gives us team players (laughs).

Jacob: Oh, I love the idea, that would be so cool. So I feel like we’ve covered a lot of what Walkabout and what WALK3D is. Without giving away any trade secrets, I’d love to ask you a little bit about the technology. Particularly I know you and I’ve talked before about how the hell you deal with focus? I mean you’re projecting something on a surface that is at a variable distance and I think you have kind of a cool way that you deal with keeping everything in focus. And then, to whatever extent you’re comfortable sharing about your technology, I’m sure people are really curious to see, what exactly are you using. You’ve been doing this for five, six years now and you’ve got a pretty fine tuned approach.

Shaun: Yeah, in it’s current form, and in it’s most kind of exciting and accelerating kind of quickly form, we were using an on-board computer. That that’s only happened in the last twelve months. So, that’s where it is all coming alive and we actually are developing a lot for it. I worked in conjunction with Piklipita with that. He was in a really good position where he’s a coder who writes operating systems, apps, and he’s also the 3D character modeler. So he’s been able to bring all these things together and work seamlessly. Otherwise we would have had to have teams of people.

But because there is a lot of trial and error. Even now when we’re creating a new character we have to test it to make sure it’s working, to make sure it’s responding. Even even with this latest one that we’ve created with the teenage girl, you know, it should have been fairly simple with all the ones we’ve created before, but she was a more realistic character, even though we kept her very simple. She actually required a lot more testing than what we were expecting. But that’s fine. Now we’ve kind of got our heads around that. As the trade secrets go, it’s a Unity 3D gamers engine. That’s the core of it, really.

It’s not groundbreaking, otherwise people would just continuously ask us how we’re using it. It’s pretty standard kit now. I’ve seen some really complex set ups in some performances using Unity 3D, so it’s doing super well and it’s a really good platform. For all intents and purposes, our characters are 3D games characters.They just aren’t confined to a game universe, you know. The the the real world is their universe; is their game location.

Jacob: I absolutely love what you’re doing with Walkabout. It’s so cool. I can’t wait to see it in the flesh myself. For now, I guess I’ll have to do with seeing the videos on your website.

Is there any other stuff that you’d want to add to the conversation that I haven’t asked about, or that we haven’t talked about, uh to, to get it in there?

Shaun: I’m very excited to be able to track skeletons and then be able to project generative animation on to the body, a naked body works well. Or even project clothing. We’ve done some projects for BBC like that.

We kind of did that through a studio process, asking the models to wear the clothing then we projected it back on, on to them, and when they were just wearing white outfits. It’d be really good to be able to take that proof of concept and make it a reality using depth sensors and body tracking them with fashion or with animation. That’s something that, with technology moving so quickly, that we are going to be able to offer this quite soon.

Jacob: That’s so interesting. So now you’re talking about going from a mentally undressing somebody to visually undressing somebody by projection mapping on their body potentially, or dressing them.

You and I even talked talked about doing some x-ray vision effects where you integrate a live wireless camera feed into the system and have it look into a wall and see what’s on the other side.

Shaun: Uh yeah, absolutely, absolutely. I mean yeah, the projector company, Optima, they are sending me tomorrow a new little projector, the short throw version of what we have, and it has like a little USB plug as well that that accepts HD wireless video. So that kind of project that we’re talking about, sending video from one side of the world to the other, is very doable and very workable. And and and because now it’s short throw we can create a really big image.

Jacob: I can’t wait to see it here in Seattle or make a trip over to London, or any of the other cities where you’re going to be bringing some of this stuff.

Shaun: That’s the beauty of it as well, because the kit is so small, we’re able to fit it in a small little bag, we’re able to put all the necessary equipment for one of the kits, so we’re able to throw four of these kits into onboard luggage and travel internationally and train up local people. If not bring our expert performers. That just opens the world up completely.