Ubi turns any wall into a touchscreen using Microsoft’s Kinect

The company, called Ubi Interactive, won $20,000 from Microsoft for its system.

"We can turn any surface into a 3D touchscreen," explained Anup Chathoth, one third of Munich-based startup Ubi Interactive. Such claims typically conjure up images of floating Minority Report-style touchscreens made from curved glass, but that's exactly what this three-person team has developed.

Ubi's system uses a Microsoft Kinect sensor to turn a regular projector into a multi-touch PC projection system, where regular PowerPoints, web pages, even games no longer require clickers or wireless mice to be navigated. By using the motion-tracking and depth-perception cameras in the Kinect, Ubi is able to detect where a user is pointing, swiping and tapping on a surface and interpret these gestures as if they were being performed on a giant touchscreen or interactive whiteboard.

All well and good in principle, but does it actually work? Wired.co.uk visited Ubi Interactive at Microsoft's Westlake offices in Seattle this week for a hands-on demonstration. And the answer was a resounding yes.

A conventional boardroom projector lit up a pane of frosted glass that was suspended in the centre of a low-lit office. On the other side of the pane was a Kinect sensor, which was capturing the movements and gestures of our hands in front of the glass and sending the data to Ubi's software, running on the same Windows PC that was sending the live image to a projector.

Responsiveness was excellent, with only a split second delay between performing a gesture and action happening on-screen. We played with a 3D model of Earth (as used on Microsoft's Surface), using two hands to zoom in and out of the virtual planet, spin the globe around and locate ourselves in downtown Seattle. Naturally, this was followed by a successful test of Rovio's AngryBirds.

Universal compatibility

Being able to play a PC version of Angry Birds highlights an important aspect of Ubi's software-based system—it works with Windows' built-in touchscreen support and works with any PC application.

"It's all Windows touch-based gestures," Chathoth explained. "We wanted to start with an experience everyone knows, but we can open up our API for 3D gestures. It knows exactly how far your fingertip is from the surface—when you actually touch it, that's a click; when you're not touching, it becomes a hovering motion."

This depth-based action was also demonstrated. Hovering a finger over a strip of book covers projected onto a wall let you gesture left and right to browse titles from an ebook download store's app. Actually tapping one of the covers selected that book for download.

Ubi is one of 11 startups that won $20,000 (£12,300) of funding and support from Microsoft as part of its Kinect Accelerator programme, but it remains independent as a startup. Its business model has several aspects: selling the software to individuals who already have a Kinect and a projector; selling a unit that bundles together a projector with a Kinect, as well as the software; retrofitting office spaces with Kinect sensors and software; working with advertisers and outdoor agencies who want to add interactivity to public displays and surfaces.

"We are already deploying some advertising displays," Chathoth explained. "It's attractive to a customer because of the cost, the ease of deployment; it's portable—an 80-inch screen you can carry inside a laptop case—it's flexible and it's also an engaging experience."

The software itself, Chathoth noted, would cost an individual "close to $500" (£320), and the company is keeping its options open for working with companies who want to integrate richer 3D applications that require custom gestures and software APIs.

He also highlighted the potential uses within educational establishments. "Kids can go and scratch on a wall, but nothing happens," Chathoth said, highlighting that by using a regular wall instead of a costly smart whiteboard means there's nothing a particularly aggressive pair of young hands could break.

History

Ubi Interactive is the name under which its three employees—Anup Chathoth, Chao Zhang and David Hajizadeh—operate. Chathoth and Zhang were school friends who wanted to develop an alternative to smart pens such as LiveScribe; Hajizadeh was a friend of a friend who came on board to develop the business side of the startup.

The original system the trio built was not based on the Kinect, however. Instead it was a bespoke device with custom software and an in-house operating system.

"We built a prototype [and] then the questions came up like, 'Yeah, the content is not there. You have your own OS, but we don't like it because we want to run PowerPoint on it', and the projector lacked the intensity," explained Chathoth. "But we we're a small start-up; we can't build a huge hardware business. So we were working with David at that time as he was helping us with the business side of things, and that's when we started using Kinect.

"We thought it could probably do everything we wanted to do—we just have to make the software. So we started playing with Kinect for Xbox when the beta software development kit (SDK) came out and then a couple of months ago we came across the Windows version and we thought this was the right business model for us—what we want to sell today is software for people who can buy Kinect for Windows from some third-party, or from us."

The result is a low-cost technology that can be retrofitted into existing projection systems without any technical experience. Without an effective demonstration it might appear to be a bit too good to be true, but this young German startup has proved its system works. What it needs to prove next is that it can scale itself well enough to support the demands of major clients. It'll likely be an interesting business to watch over the next couple of years.

I just don't see the point. For education I haven't seen any uses of "smart board" like technology that brings to the table anything that a regular $100 whiteboard + simple projector and pulldown screen doesn't. In fact all of the techologies that I have seen that attempt to allow you to use the "smart board" as you would a regular whiteboard are terrible. Significant delays between "writing" and it becoming seen on the board. Contextual issues (where the board or it's software running on the computer) don't know what should be on the "top" or what you're trying to do. In all it's a kludgy system that doesn't work well. This company has a huge amount of work cut out for them to prove that their system is a better investment than the low-tech version.

For business: really? You're going to stand up in front of the room and wave your hand around to change the slide instead of clicking the button on your remote/laser pointer? You're going to wave your hands around in front of the room trying to select the icon for your presentation/slide/whatever instead of having it already up and ready to go? I don't see there being much use (or desire) for this there either.

For the home: Simply put I'm not going to be waving my hands around in front of the tv to take 50-100% longer to get something done than simply navigating to it with the remote/controller. What I do want is for someone to get their shit straight and get us a standardized system for controlling all of our devices and selecting/pushing content to those devices from the true successor to the remote: smartphones/tablets. Think a Logitech Harmony that supports the DLNA remote spec all running from your smartphone or tablet.

I just don't see the point. For education I haven't seen any uses of "smart board" like technology that brings to the table anything that a regular $100 whiteboard + simple projector and pulldown screen doesn't. In fact all of the techologies that I have seen that attempt to allow you to use the "smart board" as you would a regular whiteboard are terrible. Significant delays between "writing" and it becoming seen on the board. Contextual issues (where the board or it's software running on the computer) don't know what should be on the "top" or what you're trying to do. In all it's a kludgy system that doesn't work well. This company has a huge amount of work cut out for them to prove that their system is a better investment than the low-tech version.

We had smartboards in college, early ones, and they were brilliant for interactive presentations. We used them a lot in cartography classes to really nice effect.

Quote:

For business: really? You're going to stand up in front of the room and wave your hand around to change the slide instead of clicking the button on your remote/laser pointer? You're going to wave your hands around in front of the room trying to select the icon for your presentation/slide/whatever instead of having it already up and ready to go? I don't see there being much use (or desire) for this there either.

You're looking at this wrong; clients can interact with the display without getting their grimy mitts on it.

Quote:

For the home: Simply put I'm not going to be waving my hands around in front of the tv to take 50-100% longer to get something done than simply navigating to it with the remote/controller. What I do want is for someone to get their shit straight and get us a standardized system for controlling all of our devices and selecting/pushing content to those devices from the true successor to the remote: smartphones/tablets. Think a Logitech Harmony that supports the DLNA remote spec all running from your smartphone or tablet.

This is pretty impressive IMO. At any kind of meeting or presentation anyone in the room can walk up and interact with the display in an extremely hands-on way with an interface most people are very familiar with.

Its also durable, and looks to be very cheap compared to a large touch-screen display. It has some issues that are inherent with projectors...

But theres a question. Would this tech work well with a normal LCD display? Would this turn any TV into an interactive touch-screen(I havnt read all the links yet )?

Ubi? Any relation to Ubisoft? I see a name change (possibly forced) in their future if this takes off.

Also, forget Kinect. Even the Xbox 360 version costs twice what it should. Why not use the $70 Leap? Looks more accurate than Kinect. The price is less than half (and a little more than a quarter of the PC-intended version), too.

I don't get it either. Most business presentation occur from behind a laptop or the presenter just uses an inconspicuous wireless remote / mouse to control the presentation if standing in front of a group. I believe it is the same way for education.

I don't now, the whole market of interacting with devices in this way to me is still a gimmick. It isn't very fluid yet, even Kinect, as impressive/promising it has shown to be for certain applications, just doesn't quite cut it. Remotes/game-pads, etc are still the preferred choice for me.

Lastly, I do not like launch front-ends such as the one shown on their website demo. It either has to integrate with the OS in a transparent way or I am not adopting it. Kudos to the developers for doing this, I take no credit away, I just don't see it as something I would personally use or recommend for that matter.

But theres a question. Would this tech work well with a normal LCD display? Would this turn any TV into an interactive touch-screen(I havnt read all the links yet )?

It wouldn't be practical with a normal display as you would end up standing between the screen and the kinect device. It works with projectors because if you do end up standing between the projector and the screen you move because you're blocking what you are trying to see.

I just don't see the point. For education I haven't seen any uses of "smart board" like technology that brings to the table anything that a regular $100 whiteboard + simple projector and pulldown screen doesn't. [snip]

I'm a college teacher. With the SmartBoard (tm), I can start with something basic in PowerPoint, like the datapath of a computer. I draw in the control signals and show what the ALU does. Then I capture PowerPoint plus drawing and put it on the web for my students. It took me a while to learn to use the SmartBoard effectively, but it really is worth the cost.

I just don't see the point. For education I haven't seen any uses of "smart board" like technology that brings to the table anything that a regular $100 whiteboard + simple projector and pulldown screen doesn't. In fact all of the techologies that I have seen that attempt to allow you to use the "smart board" as you would a regular whiteboard are terrible. Significant delays between "writing" and it becoming seen on the board. Contextual issues (where the board or it's software running on the computer) don't know what should be on the "top" or what you're trying to do. In all it's a kludgy system that doesn't work well. This company has a huge amount of work cut out for them to prove that their system is a better investment than the low-tech version.

As a biology teacher, I use a Promethean smart board in my classroom everyday. There are no delays, the system is rather simple, and it enhances my lessons. My students love it as well. Maybe you just haven't used the right system? I would prefer this though as it is much cheaper and can be used anywhere. With our system now, the bulbs are probably as much as a Kinect.

I just don't see the point. For education I haven't seen any uses of "smart board" like technology that brings to the table anything that a regular $100 whiteboard + simple projector and pulldown screen doesn't. [snip]

I'm a college teacher. With the SmartBoard (tm), I can start with something basic in PowerPoint, like the datapath of a computer. I draw in the control signals and show what the ALU does. Then I capture PowerPoint plus drawing and put it on the web for my students. It took me a while to learn to use the SmartBoard effectively, but it really is worth the cost.

Yeah.

It is similar to the file:save:... process that people carried over from when they would write a report, 'file it' under whatever scheme their organisation did or did not use. The problem is that process is precisely what computers are good at. Mundane easily repetitive tasks. If you want something different to the standard save system, you should be able to do that too, of course. But making people responsible for the standard task of saving simply means data is lost.

Using a smart board/presentation slides/displays people likewise carry over their old 'show and tell' format from kindergarten, that while effective from the communicator's perspective, is dependent on the recipients' prior understanding. It gives no method for the recipient to assimilate the information in effective learning. Smart boards are capable of allowing the clients to manipulate the data to produce their own grasp of what concepts they need to understand and what is possible.

These technologies aren't for the communicator. They are for the audience, whether that is a group of clients or traffic. Imagine signs in your local bank being able to have customers plug in data relevant and understood by them to your latest inflationary figures. Skin cancer clinics being able to give interactive information based on the person in front of the sign. There are heaps of ways smart boards can be used for encouraging people to work with the data till they understand.

We have lots of data available for many things but people aren't assimilating it as fast as is possible. These technologies can help that,[::edit::] but they have to be used by the communicators in a more sophisticated manner than just putting up a sign.

[::edit::] added in the end of the thought. Helps in communication to actually say what you mean rather than leave it for others to work out, or so I've been told.

Also to plug the Promethean software. It is extensive and no delays. I wish people would do their slides in ActivInspire rather than PowerPoint whether they have a smart board or not. Much higher quality slides would be produced despite being education centric rather than business. (I'm sure they have catered to business somewhere in their line up.)

Could you please try that again in English? [edit] I recognize all the words, but I don't get the point. Are you in favor of the technology or opposed to it?

:blink: Really? Other than post ::edit:: there isn't much more to add without repeating; I can break it down further but it becomes, "Hulk likes smart-boards; smart boards good."

I'll contemplate on my writing to uncover how I may enable the recipients to extract my point with far less trouble. The first paragraph obscured the point a bit I guess. Maybe if it had started out that 'using the smart-boards just the same as a presentation screen made it just a presentation screen', it would not have been so obscured.

Wait. How does it work on any wall when the demo had the projector one side and the kinetic on the other side of the frosted glass?

Back projection eliminates the shadow that results when you stand in front of the projector. The Kinect is watching for things that get in front of the screen. Touch the surface and the distance is zero...hover an inch or two off and the Kinect detects position and distance for the 3D motions mentioned.

When the projector is used with a movie screen or wall, there will be a shadow where the operator is blocking the picture being projected. Back projection is preferred, front projection works.

What I do want is for someone to get their shit straight and get us a standardized system for controlling all of our devices and selecting/pushing content to those devices from the true successor to the remote: smartphones/tablets. Think a Logitech Harmony that supports the DLNA remote spec all running from your smartphone or tablet.

Yuck. Smartphones/tablets are terrible remote substitutes. I tried once and it was in no way a "true successor"

For one, it's slower and awkward to use:- Smartphone: "ok wait, I have to pick up my phone, take it off sleep, unlock it, then navigate to the remote app, then wait for it to find the device, then........ ok ready to use now, FINALLY"- Remote: Pick up remote, then just use it immediately

Secondly, lack of physical buttons means no tactile operation. I don't know about you, but I don't need to look at my remote controls when I operate them, I have memorized the button layout so I can find a button just by touching it. While you're watching TV, not having this is a big drawback.

Ubi? Any relation to Ubisoft? I see a name change (possibly forced) in their future if this takes off.

According to their website, they're actually called "Übi Interactive". They did, however, decide to go for the non-umlaut domain. Couple that with the fact that many people unfamiliar with umlauts just ignore the dots, and Ubisoft might get annoyed at some point.

To those teachers who gave examples of how they effectively use a smartboard: thanks. There are definitely some ideas there that I hadn't thought about. A couple that I still think probably would be easier on a proper laptop with projector, but at least different.

rodehouse wrote:

Maybe you just haven't used the right system?

Probably. It's been a few years so I wouldn't be surprised if the technology has matured a bit. Still, (besides some of the ideas presented in this thread) there isn't a lot that I can see a smartboard benefiting over doing it ye olde fashioned way. The ideas presented here have gotten me to think a bit more about the uses in the classroom though.

Boskone wrote:

This would be awesome for digital scrapbooking, games, and gaming.

Yeah, the Kinect has been great for gaming. In fact there has been one game that has both 1) been a game and 2) used the Kinect effectively. And that's a dancing game (i.e. not my cup 'o' tea). Everything else has been either shoddily developed, not a game or trying to use the Kinect to fit a round peg into a square hole (anything that tries to add a layer of voice control onto a traditional game).

Until a series of games come out that both cater to the traditional gaming crowd AND are worth playing AND use the Kinect-style motion control well my opinion isn't likely to change. At least the Move and Wiimote actually can be effectively used to control a specific subset of games (those you point something on the screen at something else; traditionally shooters, but anything akin to a shooter they could work well with).

Yeah, the Kinect has been great for gaming. In fact there has been one game that has both 1) been a game and 2) used the Kinect effectively. And that's a dancing game (i.e. not my cup 'o' tea). Everything else has been either shoddily developed, not a game or trying to use the Kinect to fit a round peg into a square hole (anything that tries to add a layer of voice control onto a traditional game).

Until a series of games come out that both cater to the traditional gaming crowd AND are worth playing AND use the Kinect-style motion control well my opinion isn't likely to change. At least the Move and Wiimote actually can be effectively used to control a specific subset of games (those you point something on the screen at something else; traditionally shooters, but anything akin to a shooter they could work well with).

I agree the kinetic games are a joke and I think that is why people doubt the kinetics actual power, also the kinetics for windows has faster reaction processing and overall ability, cause it knows it has a strong pc usually with a top end graphics processor that isn't taxed waiting in the wings for added processing power. But even with xbox kinetic it's not the hardware / device it's the coders. Game coders still working to master the technoloy themselves, and well Microsoft went kiddy games with it to suck up the Wii gamer market. But really Wii and Move are old technology a few accelerators, and IR blaster readers. I mean heck nintendo had the power glove, and a light gun that isn't too far behind what you can do with Wii remote. And that was almost 30 years ago we had that. Kinetic is a whole new beast to learn and tackle the data how its read how to make use of it, how to make your code read it accurately, etc takes a lot of work. But as more people get ahold of it, basic library APIs get written and people start having the hardware in their systems. Projects like these will accelerate, and key thing is cost of entry. I can make a $1000 advertisement display that is user interactive has no components to be broken that a coat of new paint can't fix. Unlike normal touch screens some kid at the mall punches it and your out $5000 bucks / use of that ad space till you spend $2000 to fix the screen. Instead some dry wall and for about 50bucks and any common repair person can fix it. Now think about being at a mall that has mapped their building with google like street view. Our low in crappy mall just did this, they put kiosk in with computers and a mouse so you can "navigate" the store and see where to go or where a shop is, they even working on mapping the individual stores. Anyways problem is the computers get abused by people, mouse is smashed half the time etc. Now put in one of these units, only thing to break is some boring dry wall, and poof I get a nice interactive mall map with little to nothing to be broken. I think that is the bigger piece people also miss out. preventive care

For a side story to prove point, we were in a meeting with a vendor who was touching the screen, showing various things in the software, he pulled the screen up and used the whiteboard behind it to start drawing on the "screen" to highlight things, which worked well for us visuallizing things / changes to the product we wanted. We left one of the aids pulled the screen down again, we come back and he black dry erased on the projector screen not knowing it was down. Granted bone head on his part, but still $900 bucks for us to get replaced.

In this case he wouldn't needed to use the whiteboard, 2nd even if he did write on the wall a paint brush would of fixed the problem.

But to circle around don't take xbox random game developers use of kinetic as a reference point for how powerful kinetic actually is. Game developers just don't think outside the box well.

The most interesting thing about this story is that Ubi has three employees. Ten years ago, we'd be waiting for Sony or Nintendo to spend $200 million developing functionality like this. Today innovation lives far outside big companies, and Microsoft is smart for realizing it needs help from independent developers and entrepreneurs.

I just don't see the point. For education I haven't seen any uses of "smart board" like technology that brings to the table anything that a regular $100 whiteboard + simple projector and pulldown screen doesn't. In fact all of the techologies that I have seen that attempt to allow you to use the "smart board" as you would a regular whiteboard are terrible. Significant delays between "writing" and it becoming seen on the board. Contextual issues (where the board or it's software running on the computer) don't know what should be on the "top" or what you're trying to do. In all it's a kludgy system that doesn't work well. This company has a huge amount of work cut out for them to prove that their system is a better investment than the low-tech version.

As a biology teacher, I use a Promethean smart board in my classroom everyday. There are no delays, the system is rather simple, and it enhances my lessons. My students love it as well. Maybe you just haven't used the right system? I would prefer this though as it is much cheaper and can be used anywhere. With our system now, the bulbs are probably as much as a Kinect.

With pretty much any projector that's going to be true, Smartboard (tm) work with Toshiba, Panasonic et al and ALL of them have pricey bulbs that don't last. Ones that don't use 'bulbs' mean dimming the room lights more than clients like.