Apple has shown interest in an entirely new way for users to interact with and control their Mac, with a new system allowing users to perform gestures with their hands in a three-dimensional space.

The concept was disclosed this week in a U.S. patent application discovered by AppleInsider entitled "Three-Dimensional Imaging and Display System." It describes a system akin to Microsoft's Xbox Kinect platform, which allows for controller-free interaction with devices.

Apple's solution would optically detect a user's hands and fingertips and measure their movements. In this manner, users could use their hands to control a Mac without the need for a mouse or keyboard.

In its application, Apple notes that current computer input devices like the mouse allow users to control a system in two dimensions, along the 'X' axis and 'Y' axis. But when manipulating objects in three dimensions, input methods like a joystick or mouse can be cumbersome.

"A need thus remains for uncomplicated, economical, yet highly effective 3D input devices for computers," the filing reads. "Such devices need to be able to detect, analyze and measure objects located in a 3D volume, and to observe and track any motions thereof."

Apple's filing notes that for any system to be successful, it needs to be "economical, small, portable and versatile." It must also provide a user with "integral, immediate feedback," and it needs to be adapted well for use with a number of devices, ranging from full-fledged Macs to portable electronics like the iPhone.

The hardware described by Apple in the application would be capable of tracking a user's head as well as their hands. Potential components include an infrared sensor or visible laser, high-speed photo detector, digital signal processor, dual-axis scanning device, and subsystems for analog and video.

Apple envisions a user interface that would allow a person to more naturally use their hands and fingertips to control a device. An accompanying display could show knobs, sliders and buttons that a user could virtually manipulate in a three-dimensional space.

In one image accompanying the application, a user is shown using both of their hands to select inputs on an iMac. With their left hand, the user is twisting a virtual knob, while on the right hand, their index finger is being used to press a button.

The system would aid the user with on-screen visual cues, allowing them to more easily manipulate objects on the screen. For example, Apple's system includes a virtual representation of the user's hand displayed on the screen of the iMac.

In the application, Apple provides a number of potential uses for its described system. For example, head tracking could be used to tilt an image or zoom in on the screen based on the location of a user's head. Accordingly, moving person could be followed and tracked with a motorized camera system.

With hand gesturing, the system could track individual elements including the fingers, thumb and palm. Using gestures, a user could accomplish complex tasks like 3D rotate, zoom, scroll and volume control.

Another feature of the system described by Apple is user presence detection. With this, the system could detect whether a user is sitting in front of a display, and could even identify unique users. The system could also automatically shut down and save power when the user walks away.

Other potential uses listed by Apple are surveillance, bar code reading, object measuring, image substitution or replacement, or a virtual keyboard. In the keyboard example, Apple notes that individual fingers could be tracked for typing, and the image of a keyboard could be projected onto a desktop to give users a visual aid while typing.

The proposed invention, made public this week by the U.S. Patent and Trademark Office, was first filed by Apple in August of this year. It is credited to Christoph H. Krah.

Nice and all, but how's this 'entirely new' if MS' Kinect already allows "users to perform gestures with their hands in a three-dimensional space" as a substitute for device input/controllers?

It just is!!! Anyone that know anything about Apple knows that they invented this in August and were the first to invent it.

All the similar things like:

- People making this sort of thing a few years ago using Nintend Wii controllers
- Microsofts well known plans for this going back a few years, releasing Kinect last year and the SDK for Windows at the start of this year
- My Sony Vaio that I bought earlier this year accepting hand gestures using the web cam

are all copy's of Apple's patent filed this year after all of them existing.

Too bad Apple didn't purchase the technology in Kinect since it was offered to them first.

Microsoft did not invent the motion controls in Kinect, they licensed the technology from Prime Sense. Prime Sense is not owned by or controlled by Microsoft and licenses its technology to any company willing to pay.

Nice and all, but how's this 'entirely new' if MS' Kinect already allows "users to perform gestures with their hands in a three-dimensional space" as a substitute for device input/controllers?

Because ideas like this are not supposed to be patentable, only the specific implementation of the idea. And while I will admit that the patent office violates that concept all the time (like permitting Amazon to patent "one-click", which Apple then had to license from Amazon), it's how patents are supposed to work. Otherwise, "information is presented on a screen" would be patentable.

This is exciting! While there are already numerous theories and products that already implemented in what I called 'air gestures'*, do be reminded that this is Apple we're talking about here - a corporation that innovates. Many of Apple products are already created physically and in theory, from the mouse to the mobile phone. However, none of these things are met with real-world success, most are just gimmicks or too bulky or without any design taste. Whenever Apple are interested or developing these things, it literally changed the world.

Well, at least when the late Steve Jobs was around.

Hopefully it won't need any extra cameras such as those employed by M$ and previously Sony with their PS2 EyeToy. I'd seen Kinect working but it isn't like this - this one will enable us to 'feel' the objects rather than just waving our hands to interact slides.

As for Project Sixth Sense, it uses a wearable finger band and a camera - the Apple one looks like it only requires a board. Project Sixth Sense would rather make more sense if Sony would invest in them to advance their Move gadget (really, it's design is embarrassing - it currently looks like a microphone or worse, a dildo).

Apple's track record in touches, both in performance and commercial results, is great. Surely, everyone remembers the multi-gesture trackpad on the Macbook Pro? I think that was before touch-screens was popular. In fact, touch screens became popular AFTER Apple innovate them - there were touch screens invented before but it never really took off! Apple had just put a gas propeller on that and then some!

Now imagine if Apple pulled this air gesture gadget off:

- 3D models can be sculpted, modelled and move around freely
- Drawing on canvas will be easier - no more worrying about smudges or stylus pen nibs exhausted
- Games will be more interactive; we saw these on the iPad and iPhone, now imagine doing so WITHOUT touching the screen or anything!
- Web browsing will also be enhanced
- HECK, a lot of possibilities I can't even think of!

And because this is Apple, quality is assured. It does not only works but it works smoothly. As always, every new piece of tech has it's flaws but, again, because this is Apple, people are going to be more confident in buying and then using them.

I know I am.

* based on that we are just waving our hands and moving our body without touching anything

Head tracking would be good for a computer. Not sure people would want to take their hands off the kb and mouse to do hand gestures. Maybe just left and right air-swipes to change app, similar to iPad. Or you could jerk your head left and right to switch app.

Where gestures would really come in to their own is controlling the TV. But MS is way ahead on this one.

Because ideas like this are not supposed to be patentable, only the specific implementation of the idea. And while I will admit that the patent office violates that concept all the time (like permitting Amazon to patent "one-click", which Apple then had to license from Amazon), it's how patents are supposed to work. Otherwise, "information is presented on a screen" would be patentable.

Patents, including the "one click purchase system" by Amazon which is actually called "METHOD AND SYSTEM FOR PLACING A PURCHASE ORDER VIA A COMMUNICATIONS NETWORK", require a very detailed explanation to accomplishing your task. Amazon had to explain a method and system to accomplish that task, so if another person comes out with a different process or system to accomplish the same task, Amazon can't sue. And there certainly are many other ways to accomplish a one click purchase, it's just that Apple decided to go with Amazon's solution.

Amazon had to submit a patent that begins with explaining that there are many clients connected to the internet which are browsing Amazon's site, etc.. . So you can imagine the details required to explain the exact process in accomplishing the one click purchase.

I know, strange this would just be filed now with the diagram showing an early G4 iMac (notice the screen size/ratio). They obviously were working on this back in 2001/2002 (I think this iMac was introduced at Macworld SF 2002). But why submit the patent in August of 2011, nearly 10 years later? strange indeed.

Quote:

Originally Posted by AHrubik

Is this the predecessor comment to saying Microsoft must have copied Apple again?

Well, no. MS has something on the market now, so Microsoft wins this round in the to production race. Having just gotten and Xbox 360/Kinect on Black Friday, I have to say, it is cool. BUT... It is slow. it isn't as simple as "touching" the air in the space you want it to be. You have to hover your hand in the area of the (rather large by necessity) button and hold it there for a few seconds while the sensor recognizes the hand, and to prevent any stray interaction, while a circle fills up around the "hand" to confirm. This works for gaming menus, but would be too slow to be useful on a computer. The dashboard update that came out Tuesday allows more voice interaction, which, in my limited use of it over the 40-48 hours it has been live, is pretty good, especially as a first "live" implementation. It isn't quite to the point of me being able to say "Play Fruit Ninja" from anywhere but more menu navigation in steps. "Xbox...Games...Play"

Quote:

Originally Posted by zoetmb

Because ideas like this are not supposed to be patentable, only the specific implementation of the idea. And while I will admit that the patent office violates that concept all the time (like permitting Amazon to patent "one-click", which Apple then had to license from Amazon), it's how patents are supposed to work. Otherwise, "information is presented on a screen" would be patentable.

and I realize we have a fair share of "trolls" who exist herein to declaim that "someone else has already done this", or "Apple copies", or "it's not innovative" or we had that back in [name your decade], etc.

The reality however is that Apple looks at what a sensible and easy to use interface would act like and builds around those targets. Which is why, for example, the touch UI is foremost in processing in the iOS system, unlike other touch interfaces. Likewise first to the market with a concept doesn't make the concept practical, workable or easy to use. It's sweating the details that does that - which is Apple's bread and butter approach to things.

Now, instead of focussing on JUST using touch, or air gestures, or voice, or classical user interfaces (keyboard/pointing devices), picture a simple, easy to use blend of these interfaces: intellient agent (IA) voice controls and dictation scaled to how you want to use them, combined with air gestures, combined with tradition inputs and touch. You recombine these tools to do whichever combination best suits what you are doing at the moment. If you are doing a 3D model, then using air gestures and IA/voice (think Ironman/Tony Stark's AI JARVIS, using keyboard and other inputs only as needed. You can manipulate the model in virtual 3D (and if the ability to do inexpensive holo projection - in real 3D). Likewise, if you want to initiate a digital connection with a friend you could have your IA make the connection, but then you would disconnect via tactile interface.

All of this assumes that the capacity to support true IA/AI moves from large hardware to small hardware somehow - via cloud support, or by vastly improving the logic that underpins IA/AI, or some other means. But it will be and should be a BLENDED interface, easily customized to the individual, again on target for Apple, which has driven simple user interface from almost the beginning.

If you are going to insist on being an ass, at least demonstrate the intelligence to be a smart one

...first to the market with a concept doesn't make the concept practical, workable or easy to use. It's sweating the details that does that - which is Apple's bread and butter approach to things.

bingo.

Quote:

Originally Posted by fecklesstechguy

But it will be and should be a BLENDED interface, easily customized to the individual, again on target for Apple, which has driven simple user interface from almost the beginning.

Apple builds incredibly simple interfaces, but customizable? Steve thought he knew a better way for us to do everything. personal preference hasn't been a factor in Apple's design. i'm amazed i don't have to use felt tip [anymore] to view notes on my iPhone.

the key that i assume Apple has figured out is how to use - and track - small movements to eliminate user fatigue. that's the biggest obstacle in gesture controls. i'm not going to be flapping my arms and neck around 10 hours a day to use Creative Suite.

Kinect's motion detection uses a brute-force method. It projects thousands of infrared laser dots throughout the room, then detects changes in the patterns of dots as you move. Complex hardware, simple software.

Apple's approach will most likely be the exact opposite: simple hardware, complex software. Just a small camera with advanced image-recognition software. Instead of reacting to changes in a dot pattern, Apple could analyze the camera image directly. They already have years of experience with face recognition in iPhoto, and that could be extrapolated into hand / arm / leg / finger recognition too.

I think Apple could actually one-up Microsoft, since the Kinect dots don't seem to be close enough together to detect individual finger positions. This will be interesting.

Microsoft did not invent the motion controls in Kinect, they licensed the technology from Prime Sense. Prime Sense is not owned by or controlled by Microsoft and licenses its technology to any company willing to pay.

Really?

: I use to think only someone stupid will sell that intellectual proprietary.: So they approached Apple, and Apple said NO, we are not interested?

Kinect's motion detection uses a brute-force method. It projects thousands of infrared laser dots throughout the room, then detects changes in the patterns of dots as you move. Complex hardware, simple software.

Apple's approach will most likely be the exact opposite: simple hardware, complex software. Just a small camera with advanced image-recognition software. Instead of reacting to changes in a dot pattern, Apple could analyze the camera image directly. They already have years of experience with face recognition in iPhoto, and that could be extrapolated into hand / arm / leg / finger recognition too.

I think Apple could actually one-up Microsoft, since the Kinect dots don't seem to be close enough together to detect individual finger positions. This will be interesting.

I think you'll find kinect uses both, the infared dots are also producing a 3d model, something that is not possible to such accuracy with just an image.

The kinect sensor is also sensative enough to detect fingures. There's some videos around of it recognising sign language and someone else built a visual studio add to to type code using it.

On the Xbox its biggest limiting factors are making sure it uses hardly any processing power as that's needed for the actual games, and delays at pushing buttons are all about making sure the user really wanted to press it.

I think you'll find kinect uses both, the infared dots are also producing a 3d model, something that is not possible to such accuracy with just an image.

The kinect sensor is also sensative enough to detect fingures. There's some videos around of it recognising sign language and someone else built a visual studio add to to type code using it.

On the Xbox its biggest limiting factors are making sure it uses hardly any processing power as that's needed for the actual games, and delays at pushing buttons are all about making sure the user really wanted to press it.

The other huge limitation of Kinect is USB 2.0. Kinect 2 (rumoured to be due late-2012 or early-2013) will not have this.

Kinect's motion detection uses a brute-force method. It projects thousands of infrared laser dots throughout the room, then detects changes in the patterns of dots as you move. Complex hardware, simple software..

You clearly do not understand how Kinect works. There is significant and powerful software powering the device. Many articles have been posted written about it. The raw data is receives from the IR sensor is meaningless without the software they have to process it.

Nice and all, but how's this 'entirely new' if MS' Kinect already allows "users to perform gestures with their hands in a three-dimensional space" as a substitute for device input/controllers?

Quote:

Originally Posted by timgriff84

It just is!!! Anyone that know anything about Apple knows that they invented this in August and were the first to invent it. __snip__ you can see how they copies Apple your just an idiot!!!

You don't get it... This happens all the time.

Patent Award #1 was based on ... usually prior art ////
Patent Award #2 was based on #1 however is an 'invention' due to it's uniqueness.
Patent Award #3,4,5,6,7 etc etc etc are all based on 1,2,3,4,5,6 etc etc etc

This is why you read about patent lawsuits and say.... "how could they have a patent when WE know _____ was done YEARS before in _____". These kind of patents are ways companies try and protect themselves from one another. Some make it thru some don't. None the less, It's always worth trying. All one needs to do is word the application in such a way that it compiles with the definition of 'invention' as defined by the patent office.

Kinect's motion detection uses a brute-force method. It projects thousands of infrared laser dots throughout the room, then detects changes in the patterns of dots as you move. Complex hardware, simple software.

Apple's approach will most likely be the exact opposite: simple hardware, complex software. Just a small camera with advanced image-recognition software.

What you're explaining is almost exactly the same thing. In both cases the software is monitoring images generated from a camera.

The Kinect's infrared dot array is just running at a relatively low resolution compared to a modern camera.

The API into the Kinect is actually pretty advanced. For example, a developer is able to access the raw Kinect data, but they can also receive "skeletal" updates (i.e. Kinect converts the raw data into virtual skeletons)

The rumour is that Kinect 2 (due next year for PC only) will add finger tracking, lip reading and "emotion detection" into the Kinect API.

I assume this will be made possible by increasing the number of dots in Kinect's infrared dot array.

I think the fanboy mindset tends to hamper a lot of the good discussions on this board.

Kinect has a head start on Apple on this technology, not only in the adoption of it, but also go to market. Keep in mind, the Kinect is THE fastest selling product - something like 8 million units in 2 months (per Guiness). So it's definitely got not only the groundbreaking factor, but cool as well.

I'm not sure if any of you guys actually own the XBOX with Kinect. I do.

And with the new XBOX Live (OS) upgrade this last week, it's pretty much you were thinking Siri would do for TV but also with Gesture AND Voice control.

I will concede it's not TOTALLY there yet, I'd give it another 3 to 6 months when the content finally is up and running, but with XBOX and Comcast partnered up, it's may be what Apple HDTV will be... except without the Apple.