Artificial Intelligence

06/27/2014

How a 19-year-old hacker behind Oculus Rift set out to invent a gaming headset but ended up reviving a dead technology and building a global communications platform, worth $2 billion to Facebook in a surprise deal announced this week

After WhatsApp and Oculus, Is There Anything Facebook Won’t Acquire?

To understand why Oculus Rift matters, it helps to know who John Carmack is. You already know his work, even if you don’t know his name: Carmack is the programmer who in the early 1990s cracked the problem of how to write a video game that takes place in three-dimensional space.He’s the reason that when you play a state-of-the-art game, you’re not leaping from platform to platform or wandering through a two-dimensional dungeon, you’re running and jumping around in proper space-time, all six axes in play, backward and forward, side to side, up and down. He’s responsible for Quake, the first true 3-D game, which begat Halo and Call of Duty and all the rest of it. Carmack did for computer games what Masaccio did for painting: he turned a plane into a space.

The legendary John Carmack from id Software is pushing hard to make VR headsets a part of every gamer's standard equipment. (Click Image To Enlarge)

As such, he’s the principal architect of a medium that has generated literally billions of person–hours of entertainment over the past 20 years, and like most people who’ve started a revolution, he keeps a weather eye out for the next one. That’s how he spotted Palmer Luckey and Oculus Rift two years before Mark Zuckerberg and most of the rest of the world.

On March 26, Facebook announced that it was purchasing Oculus VR, the company Luckey started in 2012, in a deal worth $2 billion. (See my blog post dated March 25, 2014) The social-networking giant is getting top-flight engineering expertise as well as the technology behind the company’s flagship and only product, a virtual-reality headset. Facebook CEO Mark Zuckerberg said.

“Mobile is the platform of today, and now we’re also getting ready for the platforms of tomorrow. Oculus has the chance to create the most social platform ever and change the way we work, play and communicate.”

Two billion dollars is a lot of money—a head-snapping amount—for a social network to pay for a two-year-old hardware company with an ultra-nerdy name that has yet to ship a consumer-ready product and whose founder is still only 21. But what’s really surprising is that Zuckerberg is putting down a massive bet on virtual reality, which until very recently was considered not just a failure but a punch line. The Oculus deal makes for a twist ending to one of the greatest and weirdest comeback stories in the history of technology.

Oculus VR CEO and co-founder Palmer Luckey (Click Image To Enlarge)

Palmer Luckey—the name suits him—grew up in Long Beach, Calif., the son of a housewife and a car salesman. He was a natural-born tinkerer. “Self-taught!” is how he describes himself.

“Explore the world around you, take things apart, put ’em back together. You can learn a lot if you do nothing but spend your entire life in your garage working on projects or in your room reading on the Internet.”

As a teenager one of Luckey’s hobbies was taking apart old video-game consoles and reassembling them inside portable cases. Another one was virtual reality.

It was an odd hobby for a person Luckey’s age because the received wisdom at the time was that VR was a failed technology. Everybody has an idea of what VR is, or what it’s supposed to be: a simulated, three-dimensional, interactive world that surrounds you completely. It’s been a staple of science-fiction classics—-Neuromancer, Snow Crash, Tron, Star Trek, The Matrix—and a core component of our collective pop-cultural vision of the future for decades.

But apart from niche applications like designing cars and surveying oil fields, VR never made it to market. As Luckey puts it,

“The idea existed, the will existed, the people existed, the demand existed—and the technology did not.”

It baffled engineers, frustrated consumers and ate up billions of dollars of R&D money. Like flying cars and robot butlers, VR is one of those revolutions that went from wow to lame without ever actually materializing in between. Nintendo tried its hand at it in 1995 with the Virtual Boy game console and lost millions. The list of virtual-reality products that launched and then died of neglect is long.

Luckey owns most of them. He probably has the world’s most complete collection of VR headsets anywhere, more than 40 of them at last count. He bought them because he was among the very few people anywhere who still thought virtual reality was cool. Unfortunately, none of the headsets worked very well. He says.

“I didn’t start out trying to build something. I started out trying to buy something that would do what I wanted. And it became apparent that there wasn’t anything like it.”

So he started building it himself.

Luckey wasn’t the only person who still cared about virtual reality, but he almost was. There was a small community of true believers, less than a hundred, who hung out on a web forum called MTBS3D to talk about it. (MTBS stands for “meant to be seen.”) Luckey was one of them. John Carmack was another.

Carmack thought VR had potential too, in spite of all the failures, and every few years he would check in on the state of the art to see if it was usable yet. In April 2012, Carmack was tinkering with a VR headset made by Sony, and he posted about it on MTBS3D. Luckey responded. He told Carmack about his own prototype, and Carmack said he’d like to buy one. Luckey was in awe. He says.

“You cannot take money from Carmack. It would be like if Jesus said, Give me your clothes.”

He sent Carmack the prototype, his only working model, for free, via regular mail.

In 2012, what interest there was in VR was mostly in creating a kind of virtual cinema: you’d look in the headset and see a simulated version of a giant screen hanging in the air in front of you, and you’d watch a movie on it. Not many people did. Not only was it Skymall stuff, it was pricey—Sony’s head-mounted set costs $1,000.

Luckey’s device wasn’t like other headsets. Luckey’s device was different. It was designed to run games and to immerse you in them. It ran fast, and its field of view was very wide: the display wrapped around to eat up your peripheral vision, putting you well and truly in another world. Luckey says.

“That’s the only way to get any kind of immersion. I didn’t want to just have a TV you could wear.”

Carmack agreed. He adapted his latest game engine for Luckey’s headset. Two months later he took it to E3, the biggest video-gaming trade show in North America, where he announced to a startled press corps that virtual reality had finally arrived. A lot of people started asking Luckey for demos. Among them were Brendan Iribe and Nate Mitchell, both alumni of a gaming-software company called Scaleform.

Mitchell says.

“The first time I saw the Rift, it was in a hotel in Long Beach. Basically Palmer had a bunch of circuit boards, and a bucket of cables and wires, all this stuff tangled up. He set it up, plugged it in—it took him a little while, and I was sitting there being like, Is this really going to happen? Is this going to work?”

At that point Luckey’s prototype was just a box that you held up to your face, running a simulation of a bare room. But when Mitchell looked inside it, something new happened. He says.

“There was no interactivity, nothing moving. But it gave you the sensation that, wow—there’s a world inside this little box.”

Two years later, the Oculus Rift—the dorky name is a point of nerd pride—still doesn’t look particularly futuristic. It looks like a pair of chunky ski goggles with opaque black plastic where the lenses should be. Time will tell whether it’s a gateway to a new virtual frontier, but one thing is clear already: you look weird wearing it.

Oculus Rift headset (Click Image To Enlarge)

But put it on anyway—it embraces your head slightly more forcefully than would be ideally comfortable—because you’ll get the rare sensation of experiencing a technology that is genuinely new. Google Glass feels like what it looks like: you put it on and think, Yup, it’s a pair of glasses with a tiny screen in one lens. Oculus Rift is different. It’s not what you expect.

The first time I tried the Rift (which seems to be winning out over Oculus as the shorthand of choice) it showed a simulation of a craggy, rocky mountainside. I turned my head experimentally, and the view changed, with no discernible lag, just as it would have in reality. Instinctively my brain started looking for the edge of the image—but it didn’t come. I kept turning until I was looking all the way behind me. There was nothing but mountain back there.

Then I looked up and watched snowflakes sift down out of a gray sky straight into my face. That’s when my brain admitted defeat. It surrendered to the illusion that it was in another world. It wasn’t going to find an edge. There were no edges. The Oculus Rift is the first visual medium that doesn’t have a frame around it.

Another demo put me in the driver’s seat of an old-fashioned race car. Just sitting there, without even starting the engine, was a revelation. I leaned over and stuck my head out the window and admired the car’s exposed left front wheel assembly. If I leaned in to the dashboard I could read the fine print on the gauges.

When you’re in the Rift you become the camera. You control the point of view with your body, the way you would in reality.The Oculus Rift has limitations. The resolution isn’t high enough yet, so you have a slight sense that you’re viewing the world from inside a screened-in porch. Look down and you’ll notice that something’s missing: your entire body. Oculus can bring your eyes and, with headphones, your ears into the virtual world, but nothing else. You haunt the virtual world as a floating, disembodied spirit.

And yet it’s convincing. It’s visceral. VR offers a new kind of illusion. There’s a name for it in the industry, this deep and abiding conviction that you’re somewhere else: presence. I tried a simulation of a dogfight in outer space, and when my one-man fighter was shot out of the mother ship into the cold black void, my stomach dropped through the floor. After South by Southwestthis year, a viral video circulated of the actress Maisie Williams trying an Oculus Rift simulation of the 700-ft. wall of ice in Game of Thrones. She’s standing on solid ground, but she has a full-blown panic attack—she’s afraid of heights. The illusion of being on a cliff edge is tenacious. Mitchell says.

“You can’t do that on a TV monitor. You can’t do that on a phone. You’ve never been able to do that before in the history of humankind. You know you’re not going to fall, but your brain’s saying, Don’t take that step.”

Two years after he mailed his prototype headset to John Carmack, Palmer Luckey is somewhere else: a black glass office tower in Irvine, Calif., the headquarters of Oculus VR, where he now has the title of founder. Carmack, 43, is his CTO. After 22 years he quit his job at Id Software, the company he co-founded, to work at Oculus. Brendan Iribe is now Oculus’ ebullient, hyperverbal CEO. He left behind unvested options from his last company to come to Oculus. It has that effect on people.

Why could Oculus solve VR when nobody else could? The answer takes some explaining. VR presents an intractable mass of interconnected engineering challenges, most of which start with your brain.The problem with your brain is that it’s smart, and it’s difficult to fool. The human brain is constantly taking in data about the world. Some of it comes in through your eyes; some of it comes from your vestibular system, your inner ear, which provides your sense of balance and orientation. Your brain’s constantly cross-checking those data sources to make sure they match up. If they don’t, bad things happen.

Say, for example, you’re wearing a virtual–reality headset that is telling your eyes that you’re on Mars. If you move your head, the view of Mars has to change too—instantly, with no latency, the way it would in reality. If it doesn’t, your eyes get out of sync with your inner ear. Even a delay of 50 milliseconds between head-turn and view-change is too much. Your brain will spot it.

In fact, it’ll get really upset about it. So much so that it makes people feel nauseated—it’s one cause of a phenomenon known as simulator sickness, which is similar to motion sickness. Individuals’ tolerance for latency varies, but at Oculus they peg the maximum allowable lag at 20 milliseconds. On a technical level, that’s a challenging specification to hit. By comparison, an eyeblink takes about 300 milliseconds.

A headset also has to deliver new frames to the eye absurdly fast in order to keep the image from smearing or freezing when you move.It has to have two tiny high-definition monitors in it, one for each eye, and they have to cover a field of view wide enough that it blankets your peripheral vision. It has to be simple enough to mass-produce and cheap enough that people can afford it. It has to be light enough that it doesn’t hurt your nose.

Getting this kind of precision requires tight integration of hardware and software—it’s one of the mantras you hear around the Oculus offices. And beyond that, it takes a solid grasp of the fundamentals of gaming technology. That’s where a guy like Carmack, who invented some of the technology in question, comes in handy. Irebe says.

“The science around this is so close to the metal. It’s so close to what bits are happening when. Carmack knows he can go in and get that fully optimized.”

Oculus began on Aug. 2, 2012, with a campaign on Kickstarter. The goal was to raise $250,000; the project passed that figure in two hours. By the time the campaign closed 30 days later, backers had pledged $2,437,429. Since then Oculus has taken 75,000 orders for its development kit, which is a nonfinal, prerelease version of the headset intended primarily as a tool for people who want to write software and develop content for it. In December it closed a $75 million round of financing from venture-capital firm Andreessen Horowitz. Then, of course, came the Facebook purchase.

Not even the founders saw it coming, or not at first. Zuckerberg first met Iribe last November. Irebe recalls.

“He came down and we showed him some of the internal prototypes, and he got so excited about the vision of what we were doing and about the potential that this is truly the next computing platform. He actually said that to us. And it’s like, ‘Wow! We are looking at this whole thing being just that gaming platform. But tell us more, Mark.’ And he started to describe it, and we started to believe it too. And we started to relate it to a lot of the experiences we were having.”

It had been dawning on Luckey and Iribe and their colleagues for some time that they might not be as clear as they thought they were on what virtual reality is actually for. It began as a gaming technology, but it turned out first-person shooters weren’t the killer app they expected. Irebe says.

“Pretty quickly we realized, ‘O.K., maybe running down hallways at 40 m.p.h. isn’t exactly the most comfortable thing to do in VR when you’re sitting in a chair.’ As we started to build these made-for-VR experiences, we started to realize that intense gaming, where there are bullets flying at your head, can be actually a little too intense.”

So they started thinking more broadly about what exactly it was they were building. Iribe mentions virtual vacations and a 3-D VR encyclopedia as future possibilities.Mitchell describes a “magic school bus” that could take a bunch of kids on an instant field trip to Florence to look at Michelangelo’s David. But the really big opportunity, the mainstream, billion-user opportunity, was in virtual reality as a next-next-generation communications medium. Irebe says.

“When you add other people to it, and you can actually see somebody in that place and you can make eye contact, and you can look at them and they can look around, you can now have this shared sense of presence in this new gaming experience, entertainment experience or just social experience that really starts to define what virtual reality is all about.”

The news that Facebook was acquiring Oculus was not received with universal happiness in the gaming community that had backed the company in the first place. The announcement on Oculus’ blog quickly grew a comment trail 900-plus posts long essentially arguing, in various ways, that Oculus had abandoned its hardcore hacker roots to become a bland, corporate, three-dimensional ad-serving platform. Markus Persson, the creator of Minecraft, was an early backer, and he visited the Oculus offices earlier this year. He summed up the attitude when he tweeted to his 1.54 million followers,

“We were in talks about maybe bringing a version of Minecraft to Oculus. I just canceled that deal. Facebook creeps me out.”

Luckey is quick, very quick, to assert that this isn’t a pivot away from gaming and toward something else. He says.

“Nope. No pivot. We’re doing what we’ve always done. We’re continuing to operate independently, and if anything, we’re putting more resources into games, not less. This lets us invest in content, make better tools for content, better developer relations, and build a much better platform for games.”

Iribe is right behind him:

“People have not even seen our final form. There are so many cool things that happened directly because of this deal. It’s one thing to have an initial first impression of a deal that might not make sense on the outside. It’s another to see the proof of it once the big picture becomes clear.”

Iribe points out one concrete benefit for users: cheaper headsets. Now Oculus can afford to sell them at cost.He says.

“It changes our priorities from making money to making virtual reality happen."

Iribe rejects the idea that he and his colleagues sold out. He says.

“If anything, I think Facebook got an incredibly good deal. If we stayed independent, we could probably have made a lot more.”

“They want to seed the market. They want to get it in front of more developers and more early adopters. And that’s the way to do it, to give it away as cheaply as they can.”

Zuckerberg clearly has a lot of faith in the Oculus team, because as far as they’ve come, there are a lot of technical challenges left to solve before virtual reality can become a social medium at all.It will have to track more than your head: it’ll have to track your hands, your mouth, your facial expressions, your gaze. That’s not part of the existing technology. At the moment virtual reality is still a pretty lonely place.

It will also have to morph into a form factor that nontechnophiles will be willing to put on their faces. And it will face competition. Earlier this month Sony unveiled a new VR headset of its own, with the working name Project Morpheus.It will presumably connect to its popular Playstation 4 console, which already has millions of users.

For the next few years at least, Oculus VR is going to be what it started out as: a high-end gaming peripheral, supplemented with content from adventurous creatives in the broader entertainment world. Irebe says.

“We’re working a lot with people who want to do things like immersive movies or music videos or meditation or relaxation applications. It’s kind of like the beginning of film. It’s going to take this whole new set of mechanics and engineering to master it. We have no idea what really works in VR. People ask us, What’s the holy-grail app going to be? I have no idea! Don’t know.”

The uncertainty doesn’t bother him.

For now, Luckey and Carmack and the rest of them are still poised at the crest of the wave. Their money worries are over. Now they just have to safeguard what made Oculus so exciting in the first place, back when it was just a box with a room inside it. Luckey says.

“I think people have always wanted to experience the impossible. That’s one of the reasons games have caught on. They want to actually do things themselves, have a say in how that world works, instead of just watching someone else do it.”

COMMENTARY: I can certainly see a bright future for Oculus and its Rift virtual reality headset. The development of applications running on Oculus Rift will be key, as it was when Apple introduced the iPhone to the world, and again when it introduced the the iPad. The fact that many Kickstarter contributors were techies and 3D virtuality applications developers, and they will be receiving a copy of the Oculus Rift developer's tookit is a good start.

Oculus has the ability to do what Zynga did for Facebook with its social games. This means potentially lucrative 3D virtual reality games or even 3d virtual reality social games could become a big part of Facebook's future income streams. Facebook is still too dependent on advertising revenues, and could feel the pinch, if the economy goes into a tailspin. They must diversify. The departure of Zynga, left a huge, gaping hole to fill, and Oculus could just be what Facebook needs, but only time will tell.

That $75 million in new venture capital will certainly come in handy, but if Zynga is any indication, a whole lot more, perhaps as much as $500 million in additional VC rounds, will be needed to take Oculus into a big league 3D virtuality gamemaker.

I am still not sure how Facebook's 2D social platform will work in a 3D virtual reality world. Is there a future for 3D virtual reality social networks? Is this the vision of Mark Zuckerberg? Is this how Facebook users will connect and engage in the future? I can assure you that a 3D virtual reality Facebook will not be for everybody but the most overzealous and affluent Facebook users, who just have to have the next best thing to enrich their social media experience.

Having said this, I don't believe that Facebook will be running in a 3D virtuality reality format for quite some time, perhaps five years on a limited basis is my guess. Or, possibly never. Of the forty or so startups that have given 3D virtual reality a run for its money, not one has managed to generate substantial revenues, or any revenues, for that matter. Oculus could become the next big loser in that effort. It is going to take lots of capital, bright and creative 3D virtual reality software developers, and the technology to support this will have be developed. It's going to be a bandwidth eater for sure. I don't see 3D virtual reality apps working very well with good ole standard American WIFI. You will need really fast DSL line to do it reasonably well, and this raises the bar, and the cost to the consumer. So only time will tell whether Oculus will become Zuck's next big bad investment, or a huge success, taking Facebook to the next level in social networking.

10/25/2013

Artificial intelligence software and a technique called deep learning could help Facebook understand its users and their data better.

Facebook is set to get an even better understanding of the 700 million people who use the social network to share details of their personal lives each day.

A new research group within the company is working on an emerging and powerful approach to artificial intelligence known as deep learning, which uses simulated networks of brain cells to process data. Applying this method to data shared on Facebook could allow for novel features and perhaps boost the company’s ad targeting.

Deep learning has shown potential as the basis for software that could work out the emotions or events described in text even if they aren’t explicitly referenced, recognize objects in photos, and make sophisticated predictions about people’s likely future behavior.

The eight-person group, known internally as the AI team, only recently started work, and details of its experiments are still secret. But Facebook’s chief technology officer, Mike Schroepfer, will say that one obvious way to use deep learning is to improve the news feed, the personalized list of recent updates he calls Facebook’s “killer app.”

Facebook already uses conventional machine learning techniques to prune the 1,500 updates that average Facebook users could possibly see down to 30 to 60 that are judged most likely to be important to them. Schroepfer says Facebook needs to get better at picking the best updates because its users are generating more data and using the social network in different ways.

Schroepfer told MIT Technology Review.

“The data set is increasing in size, people are getting more friends, and with the advent of mobile, people are online more frequently. It’s not that I look at my news feed once at the end of the day; I constantly pull out my phone while I’m waiting for my friend or I’m at the coffee shop. We have five minutes to really delight you.”

Shroepfer says deep learning could also be used to help people organize their photos or choose which is the best one to share on Facebook.

In looking into deep learning, Facebook follows its competitors Google and Microsoft, which have used the approach to impressive effect in the past year. Google has hired and acquired leading talent in the field (see “10 Breakthrough Technologies 2013: Deep Learning”), and last year it created software that taught itself to recognize cats and other objects by reviewing stills from YouTube videos. The underlying technology was later used to slash the error rate of Google’s voice recognition services (see “Google’s Virtual Brain Goes to Work”).

Meanwhile, researchers at Microsoft have used deep learning to build a system that translates speech from English to Mandarin Chinese in real time (see “Microsoft Brings Star Trek’s Voice Translator to Life”). Chinese Web giant Baidu also recently established a Silicon Valley research lab to work on deep learning.

Less complex forms of machine learning have underpinned some of the most useful features developed by major technology companies in recent years, such as spam detection systems and facial recognition in images. The largest companies have now begun investing heavily in deep learning because it can deliver significant gains over those more established techniques, says Elliot Turner, founder and CEO of AlchemyAPI, which rents access to its own deep learning software for text and images.

He says.

“Research into understanding images, text, and language has been going on for decades, but the typical improvement a new technique might offer was a fraction of a percent. In tasks like vision or speech, we’re seeing 30 percent-plus improvements with deep learning.”

The newer technique also allows much faster progress in training a new piece of software, says Turner.

Conventional forms of machine learning are slower because before data can be fed into learning software, experts must manually choose which features of it the software should pay attention to, and they must label the data to signify, for example, that certain images contain cars.

Deep learning systems can learn with much less human intervention because they can figure out for themselves which features of the raw data are most significant. They can even work on data that hasn’t been labeled, as Google’s cat-recognizing software did. Systems able to do that typically use software that simulates networks of brain cells, known as neural nets, to process data. They require more powerful collections of computers to run.

Facebook’s AI group will work on applications that can help the company’s products as well as on more general research that will be made public, says Srinivas Narayanan, an engineering manager at Facebook who’s helping to assemble the new group. He says one way Facebook can help advance deep learning is by drawing on its recent work creating new types of hardware and software to handle large data sets (see “Inside Facebook’s Not-So-Secret New Data Center”). He says.

“It’s both a software and a hardware problem together; the way you scale these networks requires very deep integration of the two.”

COMMENTARY: For several years, Google has been heavily involved in a branch of artificial intelligence or AI called deep learning. Deep-learning software attempts to mimic the activity in layers of neurons in the neocortex, the wrinkly 80 percent of the brain where thinking occurs. The software learns, in a very real sense, to recognize patterns in digital representations of sounds, images, and other data.

The basic idea—that software can simulate the neocortex’s large array of neurons in an artificial “neural network”—is decades old, and it has led to as many disappointments as breakthroughs. But because of improvements in mathematical formulas and increasingly powerful computers, computer scientists can now model many more layers of virtual neurons than ever before.

Jeff Dean, hired by Google CEO Larry Page to headup the development of large scale computers capable of handling the computing required for modern day artificial "neural networks" and deep learning, explains Google's research into deep learning and how this technology is appled across Google's product offerings:

With this greater depth, they are producing remarkable advances in speech and image recognition. Last June, a Google deep-learning system that had been shown 10 million images from YouTube videos proved almost twice as good as any previous image recognition effort at identifying objects such as cats. Google also used the technology to cut the error rate on speech recognition in its latest Android mobile software.

In October 2012, Microsoft chief research officer Rick Rashid (YouTube video below at 33:00) wowed attendees at a lecture in China with a demonstration of speech software that transcribed his spoken words into English text with an error rate of 7 percent, translated them into Chinese-language text, and then simulated his own voice uttering them in Mandarin. That same month, a team of three graduate students and two professors won a contest held by Merck to identify molecules that could lead to new drugs. The group used deep learning to zero in on the molecules most likely to bind to their targets.

Google in particular has become a magnet for deep learning and related AI talent. In March the company bought a startup cofounded by Geoffrey Hinton, a University of Toronto computer science professor who was part of the team that won the Merck contest. Hinton, who will split his time between the university and Google, says he plans to “take ideas out of this field and apply them to real problems” such as image recognition, search, and natural-language understanding, he says.

Extending deep learning into applications beyond speech and image recognition will require more conceptual and software breakthroughs, not to mention many more advances in processing power. And we probably won’t see machines we all agree can think for themselves for years, perhaps decades—if ever. But for now, says Peter Lee, head of Microsoft Research USA, “deep learning has reignited some of the grand challenges in artificial intelligence.”

Click Image To Enlarge

In June 2012, Google demonstrated one of the largest neural networks yet, with more than a billion connections. A team led by Stanford computer science professor Andrew Ng and Google Fellow Jeff Dean showed the system images from 10 million randomly selected YouTube videos. One simulated neuron in the software model fixated on images of cats. Others focused on human faces, yellow flowers, and other objects. And thanks to the power of deep learning, the system identified these discrete objects even though no humans had ever defined or labeled them.

What stunned some AI experts, though, was the magnitude of improvement in image recognition. The system correctly categorized objects and themes in the ­YouTube images 16 percent of the time. That might not sound impressive, but it was 70 percent better than previous methods. And, Dean notes, there were 22,000 categories to choose from; correctly slotting objects into some of them required, for example, distinguishing between two similar varieties of skate fish. That would have been challenging even for most humans. When the system was asked to sort the images into 1,000 more general categories, the accuracy rate jumped above 50 percent.

Big Data

Training the many layers of virtual neurons in the experiment took 16,000 computer processors—the kind of computing infrastructure that Google has developed for its search engine and other services. At least 80 percent of the recent advances in AI can be attributed to the availability of more computer power, reckons Dileep George, cofounder of the machine-learning startup Vicarious.

There’s more to it than the sheer size of Google’s data centers, though. Deep learning has also benefited from the company’s method of splitting computing tasks among many machines so they can be done much more quickly. That’s a technology Dean helped develop earlier in his 14-year career at Google. It vastly speeds up the training of deep-learning neural networks as well, enabling Google to run larger networks and feed a lot more data to them.

Already, deep learning has improved voice search on smartphones. Until last year, Google’s Android software used a method that misunderstood many words. But in preparation for a new release of Android last July, Dean and his team helped replace part of the speech system with one based on deep learning. Because the multiple layers of neurons allow for more precise training on the many variants of a sound, the system can recognize scraps of sound more reliably, especially in noisy environments such as subway platforms. Since it’s likelier to understand what was actually uttered, the result it returns is likelier to be accurate as well. Almost overnight, the number of errors fell by up to 25 percent—results so good that many reviewers now deem Android’s voice search smarter than Apple’s more famous Siri voice assistant.

Limitations of Deep Learning

For all the advances, not everyone thinks deep learning can move artificial intelligence toward something rivaling human intelligence. Some critics say deep learning and AI in general ignore too much of the brain’s biology in favor of brute-force computing.

One such critic is Jeff Hawkins, founder of Palm Computing, whose latest venture, Numenta, is developing a machine-learning system that is biologically inspired but does not use deep learning. Numenta’s system can help predict energy consumption patterns and the likelihood that a machine such as a windmill is about to fail. Hawkins, author of On Intelligence, a 2004 book on how the brain works and how it might provide a guide to building intelligent machines, says deep learning fails to account for the concept of time. Brains process streams of sensory data, he says, and human learning depends on our ability to recall sequences of patterns: when you watch a video of a cat doing something funny, it’s the motion that matters, not a series of still images like those Google used in its experiment. Hawkins says.

“Google’s attitude is: lots of data makes up for everything."

But if it doesn’t make up for everything, the computing resources a company like Google throws at these problems can’t be dismissed. They’re crucial, say deep-learning advocates, because the brain itself is still so much more complex than any of today’s neural networks. Hinton says.

“You need lots of computational resources to make the ideas work at all.”

What’s Next

Although Google is less than forthcoming about future applications, the prospects are intriguing. Clearly, better image search would help YouTube, for instance. And Dean says deep-learning models can use phoneme data from English to more quickly train systems to recognize the spoken sounds in other languages. It’s also likely that more sophisticated image recognition could make Google’s self-driving cars much better. Then there’s search and the ads that underwrite it. Both could see vast improvements from any technology that’s better and faster at recognizing what people are really looking for—maybe even before they realize it.

This is what intrigues Kurzweil, 65, who has long had a vision of intelligent machines. In high school, he wrote software that enabled a computer to create original music in various classical styles, which he demonstrated in a 1965 appearance on the TV show I’ve Got a Secret. Since then, his inventions have included several firsts—a print-to-speech reading machine, software that could scan and digitize printed text in any font, music synthesizers that could re-create the sound of orchestral instruments, and a speech recognition system with a large vocabulary.

Today, he envisions a “cybernetic friend” that listens in on your phone conversations, reads your e-mail, and tracks your every move—if you let it, of course—so it can tell you things you want to know even before you ask. This isn’t his immediate goal at Google, but it matches that of Google cofounder Sergey Brin, who said in the company’s early days that he wanted to build the equivalent of the sentient computer HAL in 2001: A Space Odyssey—except one that wouldn’t kill people.

For now, Kurzweil aims to help computers understand and even speak in natural language. “My mandate is to give computers enough understanding of natural language to do useful things—do a better job of search, do a better job of answering questions,” he says. Essentially, he hopes to create a more flexible version of IBM’s Watson, which he admires for its ability to understand Jeopardy!queries as quirky as “a long, tiresome speech delivered by a frothy pie topping.” (Watson’s correct answer: “What is a meringue harangue?”)

Kurzweil isn’t focused solely on deep learning, though he says his approach to speech recognition is based on similar theories about how the brain works. He wants to model the actual meaning of words, phrases, and sentences, including ambiguities that usually trip up computers. “I have an idea in mind of a graphical way to represent the semantic meaning of language,” he says.

That in turn will require a more comprehensive way to graph the syntax of sentences. Google is already using this kind of analysis to improve grammar in translations. Natural-language understanding will also require computers to grasp what we humans think of as common-sense meaning. For that, Kurzweil will tap into the Knowledge Graph, Google’s catalogue of some 700 million topics, locations, people, and more, plus billions of relationships among them. It was introduced last year as a way to provide searchers with answers to their queries, not just links.

Finally, Kurzweil plans to apply deep-learning algorithms to help computers deal with the “soft boundaries and ambiguities in language.” If all that sounds daunting, it is. “Natural-language understanding is not a goal that is finished at some point, any more than search,” he says. “That’s not a project I think I’ll ever finish.”

Though Kurzweil’s vision is still years from reality, deep learning is likely to spur other applications beyond speech and image recognition in the nearer term. For one, there’s drug discovery. The surprise victory by Hinton’s group in the Merck contest clearly showed the utility of deep learning in a field where few had expected it to make an impact.

That’s not all. Microsoft’s Peter Lee says there’s promising early research on potential uses of deep learning in machine vision—technologies that use imaging for applications such as industrial inspection and robot guidance. He also envisions personal sensors that deep neural networks could use to predict medical problems. And sensors throughout a city might feed deep-learning systems that could, for instance, predict where traffic jams might occur.

In a field that attempts something as profound as modeling the human brain, it’s inevitable that one technique won’t solve all the challenges. But for now, this one is leading the way in artificial intelligence.

Dean says.

“Deep learning is a really powerful metaphor for learning about the world.”

Conclusions

Although artificial intelligence has made some significant contributions in the science of deep learning, the technology is not even close to emulating the thinking processes that go in inside the human brain. Although IBM's Watson computer has demonstrated that it can compete and win against humans in a television game show like Jeopardy, we are a long way from developing systems that can match or outperform the human brain. Apple's personal assistant app SIRI uses voice commands to locate restaurants, movie theaters, Starbucks, and so forth, but results are often incorrect or way off course. In short, artificial intelligence is not really very intelligent. It is still computer programming that does a certain thing fairly well. When artificial intelligence has risen to the level of the HAL 9000 computer, and can think logically, intelligently, autonomosly, and learns and refines, then maybe we will take this science beyond the realm of science fiction to reality.