The Iron Man returns! This time accompanied by Playstation Move, Novint Falcon and Leonar3Do

As I went to see the second installment of Iron Man, I kept thinking whether they will still have the cool visualizations and interfaces present in the first one. Maybe the audience didn't really respond to them? Was it only a geeky segment that would be cut off for more space pursuits and lasers?

Turns out my fears were completely misplaced. Not only did they increase screen time, they went crazy with ideas! 3D scanners! Surrounding interface! Digital basketball!

So, keeping with that theme, if you thought part one had too many videos, prepare for another video-post exxtravaganza!

Let's run the analysis on this one. If in the first movie I referred to the Autocad-like interface as direct manipulation. In this video, the interface Tony Stark is using expands ﻿that concept, placing the operator in the middle of the interface, stopping a few steps short of Star Trek's Holodeck. Despite that, I'll try describe how we would start building such a thing.

Again, center stage is accurate response to direct manipulation. But this time, Tony is surrounded by the elements, by his engines, digital representation of his suits and is able to manipulate any of them as he pleases. Do note that he is moving through the room/garage when working. So if previously Tony used a stylus/table area on which he worked - focus is on working area, Tony is facing the "computer" - now the entire mode of operation is focused on him. Pardon the corniness, but I'd call this type of interface a Renaissance UI, as it redefines itself around its human operator. How would we go about building this? Skipping the topic of holograms for now, let's presume we have a way to display the 3D elements in a spatial manner. We would need to track the entire "work area", meaning the room, for accurate positioning within it. Each of the digital entities would need to have an XYZ coordinate. Once that is in place, the human operator's position and full body posture needs to be digitized and accounted for. There is a large amount of hand and head tracking in place.This needs to be very accurate and responsive. In this particular instance, head tracking needs to be especially sensitive. I think that Tony's work room would have to be absolutely packed with sensors, projectors and cameras, sitting above a cluster of servers that would process the entire thing. To top it off, the room is not a cube-like (even perhaps dome-like), completely empty box as the Star Trek holodeck, there are cars! desks! robotic equipment! Oh the headaches these Iron Man designers give me... As Robin Williams once said about golf, "They put shit in the way!".

I would say that a dome-like, empty room, inward projected, 3D imaged, with full body tracking would be an interesting middle-of-the road solution. Sure, you might not get to run around it, crumple engines into basketballs and dunk them into previously invisible baskets ( Note to Iron Man designers: yes, sometimes, ﻿programmers like to put in easter eggs, but come on, a virtual basketball forming out off nowhere?)

So what do we have here? Enough projectors to cover all viewing angles (depicted only two), a similar array of sensors for body tracking and our main character. Now, I didn't give him glasses to make him a geek, I gave him active shutter glasses to see the 3D images (the red cube).

Hypothetically, let's say we have gathered all the needed hardware. Several dozens projectors, a lot of tracking sensors and 3D glasses. From the software point of view, we would need a complicated process to render the 3D imaging. Active shutter glasses work best when the distance between the screen and glasses is known (more on that later). Then we would need an approximation of where the objects are when "seen" in 3D to give the XYZ for a certain element. We need to do this to realize the accurate direct manipulation.

Leona3Do makes 3D work

Earlier this year I found out about a 3D editing solution for *CAD software. Here is their video.

This is a very impressive demo and a really useful tool for modelers.It's available for purchase for 750 €. Leonar3Do is the first commercial offering I've seen that is truly a 3D sculpting tool. The setup is rather involved, with 3 hardware dongles, screen measuring, distance points mapping on the monitor and so on. However, once it's up and running, it's a very interesting solution in the field of visualization. It is not as accurate as the g-speak (doesn't detect hands), not as setup-free as the Kinect, but it comes closest to our Renaissance UI, because it projects 3D images in a ﻿real space where they are directly manipulated and seen as real objects .

This looks like our solution, right? Well, it does have several limitations, some of which can be eliminated, some that can't be.

﻿Accuracy on the Move

The Move works in combination with the PS Eye camera for the PS3 and it also capable of head-tracking and accurate 3D manipulation. So how does it work? In the demo, you see the Move controller's motion can be detected back and forth in 3D space. The colored round ball on top of it is responsible for that. The PS Eye camera picks up the ball, sees it as a circle and measures the diameter. Based on that diameter it can calculate the distance. The most impressive part of the demo is, however, the accuracy of the sensors within the controller. Even the slightest changes are picked up by the controller and rendered accordingly. Two accelerometers in each Move controller are feeding back data to the PS3. Together with the Eye, they provide a full picture on XYZ coordinates of the controller and its orientation. I have not seen any convincing 3D (active shutter glasses variety) demos with the Move yet, but I think it has the potential to surpass Leonar3Do in 3D sculpting. I've read that some internal Sony teams already started to use the Moves as 3D sculpting tools. For more *ahem* entertaining uses, see this.

So, 3D, accurate, direct manipulation? Sure, it would be nice if it had the firepower to work only with our hands, but Sony realized the hardware and software are not good enough at this moment for that. As it turns out, you need some buttons. Honestly, I think Microsoft made a big mistake when it embarked on the "You are the controller" slogan. The Kinect would have been perfect working with the Move controller. As it is, Sony is supplementing the lack of sensors in the Playstation Eye with software and the Cell processor's huge capabilities for number crunching. So, while Kinect sensitivity might increase in the future, I believe Sony delivered the better solution for this moment.

And there is the one thing extra that Move brings to the table. Feedback.

The case of the missing feedback

﻿A topic I have not covered yet is input feedback. It's missing in Iron Man, Kinect, g-speak or Leonar3Do and it's a huge part of the experience. It's also one of the problems that falls in the same category as accurate holograms: something for which, given the amount of freedom we would want in the Renaissance UI, we don't have a solution yet.

Why is feedback important? I've written about the benefits of having an interface that mimics physical properties. Kinetic scrolling, inertia simulation, acceleration and deceleration are all bits that enhance the user's experience and provide familiarity in a new setting. We all learn the physics of this world from the earliest age. A natural user interface would be usable by a 2-year old, as it would mimic the world he already learned. And if your interface is usable by a 2-year old, it means other users will thank you for it. But I digress.

The missing bit from all this is reactivity or feedback. In the game console vernacular, it's also referred to as force feedback. You grab a stone, it is heavy. You pull a string, it opposes resistance. You press a key, it has a travel distance and a stop point which gives you a confirmation that a key has been hit.

In all the Iron Man scenes, that feedback is missing. In playing with Kinect, that feedback is again missing. To accurately throw an object and hit a target, our body has learned to apply the required amount of force based on the object's weight (sensory input), distance(visual input) and previous experience (throwing other objects). Hitting a target through air can be hit or miss. Typing several paragraphs of this article on an iPad, I can tell first hand that missing feedback is the thing that kept my typing speed down the most.

The Playstation Move has a force-feedback engine inside it. So, while driving a car, hitting a wall would make the controller bounce in the opposite direction. This adds a lot to the realism of operating such an interface.

The Novint Falcon product is probably the best feedback device on the market right now. ArsTechnica had this to say about it:

When you fire the shotgun, the orb kicks in your hand; it's impossible to fire any fully-automatic weapon for more than short bursts while keeping your aim in one place. This is interesting, because the game suddenly becomes much more tactical—you have to think about what weapon you're using, and aiming is much more difficult. I mean that in a good way; this feels more real in many ways.[...]The Falcon makes Half-Life 2 much more engaging and immersive.

I really don't have much to add on this, so please check out Arstechnica's review.

In closing, there are many challenges ahead for building a real-life Renaissance UI, but with systems like Kinect, Move, g-speak, we are moving closer to that reality every day. Besides g-speak, you could purchase any of the other devices and have unique and novel interfaces for computers in your living room.

We are still at the very infancy of interfacing with such controls. Navigation, data visualization, gestures, workspace organization are all things which will be solved in software and that we will need to figure out in the coming years. I will address my ideas for most of these in a future series of articles.

﻿There is something to be said about sparking imagination. The role of filmmakers is inspiring real life - Star Trek, Minority Report, Iron Man all raised a bar that we can now only reach with our imagination. And it is that imagination that drives us forward to build a ladder to that bar.

And now, here's the Iron Man song, as performed by Black Sabbath. No, it has nothing to do with interfaces, design or innovative controllers. It's just 5 minutes of awesome rock.