Death to CAPTCHA! Google wants to make them invisible using AI

In-depth: The future of search

This article was taken from the January 2013 issue of Wired magazine. Be the first to read Wired's articles in print before they're posted online, and get your hands on loads of additional content by <span class="s1">subscribing online.

Senior vice president and fellow at Google

Google/Connie Zhou

ADVERTISEMENT

Meeting Amit Singhal, a senior vice president and fellow at Google, and the man who orchestrates the battery of algorithms behind your queries, it's hard to resist complaining about search results. Imagine you were driving in northern Massachusetts looking for a restaurant that you vaguely remember had "Sugar Shack" in the title. Typing "sugar shack," however, yielded, as the number one entry, a business billed as "Wisconsin's Premier Adult Entertainment Club". Given that you were a thousand miles away, and had your wife and children in the car, let's just assume a lap dance in the US's dairyland was not exactly top of mind.

And yet it clearly was for Google. Granted, the search was abstract: another word or two -- what engineers call "refinements" -- and it would have found the right place. Given that many of us can remember a time when finding such information would have required knowing an address and wrestling with a billowing paper map, it seems almost rude to ask Singhal, who is sitting in a conference room in Mountain View: "Why didn't Google understand me?"

Read more

The WIRED 100

ByWIRED

He clucks a bit and replies, in a patient tone: "Search is by no means a solved problem." Singhal quickly gets diagnostic. "You were in Massachusetts. Near Hadley. Sugar Shack should have meant something else. I don't know what phone you were using but looking at that one," he glances at the iPhone on the table, "sometimes we don't get the location." Without context, we are stuck in what futurist Paul Saffo calls the "Boolean prison of search". (George Boole, a Victorian mathematician who pioneered the binary approach, is regarded as one of the fathers of computer science.) Encased by a particular word combination and what it statistically seems to represent -- for instance, sex and baked-goods emporia -- we are dragged down the paths of everyone else's preferences.

What is remarkable here is not that a search like this didn't quite work the first time, but the expectation that it should have.

In just a few years we have gone from search engines -- the name now sounds as archaic as the Victorian "difference engines" -- with their roots in the staid academic discipline of information retrieval, to, simply, "search", which is much more than an apparatus and something closer to a digital prosthesis. As John Battelle, author of The Search, says, "Search is now more than a web destination and a few words plugged into a box. Search is a mode, a method of interaction with the physical and virtual worlds. What is Siri but search? What are apps such as Yelp or Foursquare but structured search machines? Search has become embedded into everything, and has reached well beyond its web-based roots."

ADVERTISEMENT

Read more

15 innovations pushing human performance to the limit

ByJamie Millar

Search has become strangely intimate, a trusted friend pointing you in the right direction, or occasionally giving you an unsettling glimpse into the world. Search, Battelle suggests, went from looking for what we knew on the web to looking for what we don't know. Now, he says, even when we don't know what we don't know -- that Rumsfeldian state of "unknown unknowns" -- we head to Google, tentatively entering a few letters, waiting for the instant feedback of autocomplete (we no longer have to remember, as one Google engineer put it, we simply have to recognise), wondering what precise phrasing will yield the right data. We then stumble upon the footprints -- the digital "slime trail," as the investor and entrepreneur Esther Dyson describes it -- of where people have gone before. You hear a song with the lyrics "How do I know if he really loves me?" and you begin to type in order to discover the artist. But before you finish typing you're being pointed, via autocomplete, down darker corridors, simultaneously personal and aggregate: how do I know if I have bed bugs? How do I know if I have a yeast infection? We once used search engines to look for information, now we use search to find us -- what once seemed transactional now seems an extension of ourselves. Consider the exercise of finding the current time in Australia. This would once have entailed the following steps: 1. Knowing the current time where you are; 2. Searching for a reputable application that translated time zones; 3. Entering your location and choosing Australia from a separate menu. Now, on Google, you simply type -- or ask, via voice -- "What's the current time in Australia?" Google does the work and understands what you want. Understands. "As a scientist I can say 'understand' is a poorly understood concept," says Singhal. "Even how you and I understand something is not well understood."

The 'knowledge graph' contains more than 500 million entities

No one knows better than Singhal how much Google does not understand you. "'How big?' is a very ambiguous question," he says. "Are you looking for length or width? If you say, 'show me the money', you are talking about something else than if you say, 'show me a picture of a dahlia'." Or take a simple word such as 'kings'. "In yesterday's world, you typed this five-letter sequence, we find the best pages," he says. "We may find the Sacramento Kings

ADVERTISEMENT

Read more

Justin Lin on his high-octane, diversity-fuelled Star Trek reboot

[basketball team], we find the TV series Kings -- we don't understand any of this." Google is like the Thai champion

Scrabble-players who memorise the entire list of acceptable words -- without actually knowing what they mean.

However, in the search of the future that Singhal and his masters of disambiguation are constructing in Mountain View, Google will understand that these things are not simply matching sequences but that they are "things" with an internet life and place and history of their own. Based on who you are it will know which one of these, or any other "kings", that you are seeking. And it will do so via increasingly sophisticated methods: "be it understanding your speech, your gestures, or what you are looking at," says Singhal.

Singhal, a former student of Gerard Salton, the Harvard and Cornell University computer scientist who pioneered digital search, is old enough to remember when a cutting-edge, hypertext information retrieval system, such as InDecks or McBee, involved edge-notched cards and sorting rods. But his dream goes back further, to being a boy in India watching a black-and-white television. "We didn't produce enough content in India," he says, "so all I watched were Star Trek reruns." That's where his dream was born. "You walk up to the computer and say, 'What's the atmosphere on that planet down there?' That's what I want to build."

Read more

Inside Red Bull's extreme bootcamp where athletes become winners

ByJoão Medeiros

People have been trying to organise the world's information for a long time," says John Giannandrea, a bearded and garrulous Scot, over lunch at the Google campus (he picks the seat at a table in the midday sun: "I'm from Scotland, I never get tired of California"). "I'm fond of the story that Alexander the Great had the best teacher possible, Aristotle. And Aristotle knew roughly everything there was to know."

Google/Connie Zhou

Today, the quest for knowledge is less important than simply managing it. "We're in a world where almost everything is at your fingertips," says Giannandrea. "But how do you go through it all?"

Enter the Knowledge Graph. Envisioned as a database of all the world's useful information upon its creation in 2005 -- under the startup Metaweb, in which Giannandrea was joined by programmers Danny Hillis and Robert Cook -- in 2010 it was acquired by Google and received, as he puts it, "a massive turbo boost". "One of the things we're trying to do is first to catalogue everything in the world you might want to know about," he says. "We're also trying to marry that with the knowledge that the search engine already has about what people are actually looking for."

Read more

Behind the scenes with Chris Froome and Team Sky ahead of Tour De France 2016

ByJoão Medeiros

Google/Connie Zhou

Take, for example, New York City Mayor Michael Bloomberg. He exists as an "entity", one of the more than 500 million things in the Knowledge Graph. (Wikipedia, Giannandrea notes, has "about four million things it knows about in English.") Bloomberg's daughters, Georgina and Emma are entities, as is his university, Harvard Business School. In the vast semantic graph that the Knowledge Graph represents, the connections, or "edges", between Bloomberg and his daughters, and where he was educated, are also "things".

[Quote##Google launched its 'baby' into a core sample of

10 million still images####KeepInline]

Read more

Entire buildings are being deleted from China's 'Street View' and no-one knows why

ByRowland Manthorpe

And so when the user searches for "Michael Bloomberg", Google is not looking for the web pages that contain that string of letters, but for the entity known as Michael Bloomberg. "With the Knowledge Graph," says Singhal, "Google has become smarter. Search now understands that the Taj Mahal is a building, but also a music band, a casino and a bunch of restaurants." Things, not strings, as Google likes to say. For Michael Bloomberg, the way the Knowledge Graph "surfaces" this information is, to the right of its traditional blue links, a panel of curated data, including biographical details and, thanks to past queries about him, his net worth. Type in "Tom Cruise", and you'll see, prominently displayed, his height. Type in "Amit Singhal", and you'll see that he was born in Jhansi, as well as a link to his mentor, Salton.

The larger goal of the Knowledge Graph is to enable computers to understand the world the way humans do. "Our computers don't have any notion of these things that we take for granted," says Giannandrea. "We know there's a book called Infinite Jest, written by an author, David Foster Wallace. When I say Infinite Jest you say, 'Oh, he means that book'. Our computers, until now, didn't have anything other than data and text. They didn't put any meaning on text so they couldn't understand what they had."

Infinite Jest could have been anything; now Google understands Infinite Jest as a thing, and all its forms: hardcover, paperback, Kindle.

But what's an entity and what's an edge? If David Foster Wallace, a thing, went to Estonia, a thing, is there a new thing based on the metadata between Wallace and Estonia? "It's slippery,"

Read more

The robot uprising has begun and it's about to change your life

ByTom Vanderbilt

Giannandrea says. "What's the definition of an entity? I'll know it when I see it. It doesn't have to have a name. It could be an event -- an artist playing at a venue at a particular date. Is a specific plane trip an event? Yeah, probably, so every day there's another 30,000 events. Are all the unnamed stars in the universe entities?

Probably not." The Metaweb and now the Knowledge Graph, has been absorbing the world's structured databases. "St Andrews

[University] has detailed information about the careers of mathematicians; the same for philosophers at Stanford," says Giannandrea. "At Berkeley there's an expert in bees. He's got this database of 40,000 species. There are websites that catalogue roller-coaster rides, with specs about G-force and how many curves and when it was built. There are these incredible pockets of information about almost any vertical you can imagine." The work of the semantic graph is to make the connections that ­traditional search might overlook. "You'd be surprised at how many times there's a serendipitous link between two different things," he says. "It can be hard to describe for computers to understand that: what's the relationship between Einstein and Gandhi? They were both pacifists in later life." This might be a common search inquiry, he suggests, but the computer cannot work out why.

As much as trying to know everything, the Knowledge Graph is about trying to work out what you want to know, parsing the disambiguation ("did you mean?") and filtering noise. Search is bedevilled by things like hypernyms: words that mean the same thing as a more specific usage. Take the word, "jaguar", for instance. "It has, like, 26 different meanings," says Giannandrea. "The animal; the Mac operating system; there's a popular artist in South America." By recognising them as humans do and not just as groups of letters, the Knowledge Graph, he says, can help "change our understanding of user intent."

Read more

Warby Parker is bringing affordable eyewear to the developing world

ByRichard Benson

Google doesn't just let algorithms do this work. The Knowledge Graph was beta tested by any number of people in its User Experience Lab. "We ran 12 tests on Knowledge Graph," says John Boyd, the manager of the lab, which is equipped with two-way mirrors and eye-tracking devices. Early studies looked at whether users, habituated to Google's layout, even saw the Knowledge Graph.

Often they didn't, just as they often didn't see "Google Instant" results ("I'd ­characterise queries as a sort of quantum phenomena," Boyd says. "Often they're going to type it out, no

matter what they do.")

With the Knowledge Graph, Google has taken a different step towards the future of search: providing answers, not links. This raises the question of authority, long on the mind of Google engineers. A few years ago, Google faced controversy when it was revealed a search for the word "Jew" returned several anti-Semitic websites. Through brute algorithmic logic, it made sense: the sort of people who use the word "Jew" tend to have those sorts of proclivities. Now a search for that word leads in short order to an explanatory page from Google (which states, in part: "Someone searching for information on Jewish people would be more likely to enter terms like 'Judaism', 'Jewish people' or 'Jews' than the single word 'Jew'. In fact, prior to this incident, the word 'Jew' only appeared about once in every ten million search queries").

Read more

Perry Chen on how Kickstarter balances profit with purpose

ByPerry Chen

While Singhal says that "time and again we decided that Google shouldn't intervene in the [search] process," it is constantly shaping the world -- for example, it recently struck the peer-sharing site The Pirate Bay from autocomplete -- and the fact that "Holocaust denial" yields very different results than "Holocaust lie" is as much a social as a search issue.

The Knowledge Graph also challenges the organisational hegemony of ­Google's dozen blue links. "The web is very top down, in terms of links and anchors," says ­Giannandrea. "What you can't really do in a web browser is look at a page about a particular play and think, 'What other plays should I consider?' We need to be able to go sideways through human knowledge." This is figurative, but also literal: entering ­"London bridges" on Google, one now sees an image carousel of London's most significant bridges, arrayed horizontally. This is possible because those bridges have been encoded as entities in the Knowledge Graph. But what happens when that knowledge is not encapsulated in structured databases, when it's not a piece of text, or even when the subject of one's search is something the user is looking at in the moment?

Finding cat videos on YouTube is easy: go to YouTube and type in " cats". The reason you find them is because they have been tagged with the word "cats." What if you wanted to find every appearance of a cat, however fleeting, in all of YouTube's videos?

This was not what Jeff Dean, a fellow in Google's Systems Infrastructure Group, and his colleagues had in mind when they set out to create a neural network for "unsupervised learning", meaning recognising images, such as faces, when they weren't tagged in some recognition software trials. The algorithm ­consists of a vast number of "neurons" -- a billion ­"trainable parameters", which dwarfs other systems but is a still a fraction of the human visual cortex -- across an army of computers. "Each neuron looks at a fairly small patch [of a sample image]," says Dean. "It's taken inputs from the raw pixels and computed some function, and this neuron over here is doing something similar. You can have different neurons with different weights that look for different features."

Read more

How Change.org motivates millions to change the world

ByKathryn Nave

Dean, who speaks in an intense staccato, pauses. "It's sort of like what happens with a newborn baby." They get a lot of optical stimuli; they start to look at patterns. "One of the most common things you see as a newborn is faces. Pretty quickly they start to form associations that this is an important thing that I'm seeing a lot." Google launched its "baby" into a sample of 10,000,000 still images from random YouTube videos. It turned out that one of the neurons was highly selective to whether there was a face present, "even though we never told it any of the training data had a face in it." He shows some images on his laptop -- spectral faces with hollowed out eyes. "These images caused it to fire the most. It's picking up on the eyes, mouth and nose, and the circular nature of the face." With each pass the network gets better at recognising what it sees.

The other things the ­neurons were good at recognising were, it turns out, cats; because there are lots of images of cats on YouTube. The neural logic is: cats occur in a lot of images, so the network wants to optimise itself to recognise this thing that seems important. This is, in essence, search. What Dean calls "unsupervised learning" could be termed unsupervised search -- machines that not only find, but interpret what they find, a search engine that generates its own algorithms. And Dean envisions the networks will be useful for words as well. Words will be represented by high-dimensional ­vectors: a word such as "dolphin" will be put in a 100-dimensional space ("I'm only drawing a 3D space because I'm not good at drawing 100 dimensions," he jokes). "Over time, you're going to push words that you find to be closely related, closer together. And you push other words farther apart."

You need that many dimensions, he explains, "Where you can push some of the words in some of the dimensions without destroying their association, their proximity to other words in other dimensions." How closely these words lie near others will help determine context and relevance.

This is what the human brain would like to have by its side, when you're seeking information, or sometimes information comes to you without your seeking itAmit Singhal, Google

This sounds like the semantic web described by Tim Berners-Lee as a "web of data that can be processed directly and indirectly by machines." As Greg Linden, who invented Amazon's recommendation engines and founded Findory notes, says, "I don't think we'll ever get to the semantic web as it was envisioned -- detailed labelling and descriptions of web pages by humans -- but we are getting closer to its goal: deep descriptions and understanding of the web, through artificial intelligence and natural language understanding." Google, he suggests, has decided that labelling web pages is beyond humans, and is turning to machines. These are the pillars of Google's future of search: the vast knowledge of user behaviour and intent it already has and is compiling every second; the Knowledge Graph, in which strings become things; and Google's advances in artificial intelligence.

But the promise of that future should not disguise how hard search still is. When Google acquired Metaweb in 2010, a company statement noted that the deal would help it carry out more complex searches. The example it gave was "colleges on the West Coast with tuition under $30,000." Today, that search coughs up articles about the Metaweb acquisition. "Problems still plague search," says Linden. "The typical query is short and ambiguous -- such as [looking for] pizza. Dealing with that requires understanding of the underlying need." "A link is not at all an answer," says Oren Etzioni, professor of computer science at the University of Washington and founder of Decide, a search engine that analyses optimal purchase options and timing. "We've been conditioned by years of using Google to think it is." A pizza inquiry, he suggests, "is treated by a search engine as information retrieval. You wanted something that performed a deeper analysis, computed your location. You don't just want the nearest ones -- what are the nearest high-quality, well-reviewed places? That's actually a huge problem, they don't analyse the reviews." Etzioni, with some students, is working on revminer.com, a program that extracts data from Yelp reviews.

Singhal believes that search is best done on a mobile device. "We are building our technology where it's needed most," he says: Android 4.1.

With mobile search set to exceed desktop in 2015, according to research firm IDC, we will need what Singhal or demarcated by boundaries, as describes as, "a process running by [our] side. The perfect assistant."

The success of Apple's Siri -- analysts had to revise their sales estimates for the iPhone 4S upwards after its launch, such was consumer enthusiasm -- suggests, despite all its technological flaws, says Etzioni, "that people are very eager for that style of interaction, a conversation -- just give me that information without ten blue links." Increasingly, it's more than information that we're after. Dyson notes that Bill Gates told her: "the future of search is verbs." People, the argument goes, want search to do things, not just suggest things. With the Knowledge Graph, Google is building a world-historical collection of nouns. But will it help book a restaurant table? Or the cheapest flight? As synonymous as search is with Google, much of our search activity now occurs on apps. As Battelle notes, "the largest issue with search is that we learned about it when the web was young. When the universe was complete, the entire web was searchable," he says. "Now our digital lives are utterly fractured -- in apps, in walled gardens such as Facebook, across clunky interfaces. Reuniting our digital lives into one platform that is searchable is, to me, the largest problem we face today."

When it's suggested to Singhal that the future of search may not really be "search" at all, but some as yet undefined process, his answer is quick: "I won't get hung up on words," he says. "You can call it whatever you want. This is what the human brain would like to have by its side, when you're seeking information, or sometimes information comes to you without your seeking it."

***

How Google search works

1. Spider Dispatch

Like many search engines, Google uses "spiders", robot programs that scan new and updated pages and index every word (except "a", "an" and "the"). Called Googlebots, they follow links from page to page, making its index more comprehensive.

2. Indexing

The index doesn't contain just keywords, but also metadata: information on whether the keywords were capitalised, their font size, and where on the page they were found (in the title, subtitle or lower down), in order to help rank the importance of the page.

3. Ranking

The unique feature of Google's search algorithm is PageRank. It rates a page's importance based on the number and reputation of links that pointed to them. It also considers things like how often keywords appear, the freshness of a page and which sites link to it.

4. Definition

The algorithm uses over 200 signals to refine a search query.

These include a website's PageRank, a searcher's geographic location, which links they usually click, how they modify their search queries when they are unsatisfied and their search history.

Three further searchesBingStrengths: Related searches appear in the right-hand column, and it has a very refined image search. With the integration of Farecast, it can give the best fares.

Weaknesses: It can't fill gaps if you get the search term partly wrong. You cannot search for specific dates.

Yahoo!Strengths: Integrating results with original content - audio, video and images. You can personalise your page, to make the most relevant result to you appear first.