Smartest Machine on Earth

Jeopardy! challenges even the best human minds. Can a computer win the game?
Airing May 2, 2012 at 9 pm on PBS
Aired May 2, 2012 on PBS

Originally aired 09.14.11

Program Description

Jeopardy! challenges even the best human minds. Can a computer win the game?

"Watson," an IBM computing system, is gearing up for a first-of-its-kind challenge—taking on human contestants on the game show Jeopardy! With a brain the size of 2,400 home computers and a database of about 10 million documents, will Watson be able to compute its way to victory? Win or lose, the difficulty of mimicking the human thought process with software is showing artificial-intelligence researchers that there's more than one way to be "intelligent."

Transcript

Smartest Machine on Earth

PBS Airdate: February 9, 2011

NARRATOR: The stage is set for a showdown the world will be
watching. All eyes will be on one competitor, but he's not breaking a sweat. We're
not talking nerves of steel, more like a rock-solid memory of silicon.

WATSON: Hello, my name is WATSON.

NARRATOR: For the first time in history, a computer will
compete in the ultimate brain game.

ANNOUNCER
(Jeopardy!): This is Jeopardy!

NARRATOR: For the scientists who created WATSON, the
excitement and tension are palpable.

DAVID
FERRUCCI (IBM Research): It's going to be edge-of-your-seat. It's going to be
nerve-wracking.

NARRATOR: A four-year odyssey comes down to this.

DAVE
FERRUCCI: What really is going to happen? And you just don't
know. You don't know.

NARRATOR: There are more than reputations at stake. Some
say a WATSON victory could signal a new age, a revolution in artificial
intelligence.

LUIS
VON AHN (Carnegie Mellon University):
It used to be the case that intelligence
was chess, right? If you can play chess, that's intelligence.

NARRATOR: But taking on spoken language and ferocious word-play
means engaging by a whole new set of rules. It requires more than just memory
to match wits with humans.

TODD CRAIN(Training Game Host): WATSON?

LUIS
VON AHN: Just understanding the question is a big deal.

NARRATOR: How will WATSON do it?

NOVA
gets unique access to the making of WATSON, shadowing the IBM team working
around the clock, training and test-driving WATSON,...

TODD
CRAIN: Comedies directed by Blake Edwards.
WATSON?

WATSON: What is The Pink Panther?

DAVE
FERRUCCI: We couldn't write rules for every combination of
word or phrases or context.

NARRATOR: ...refining powerful new tools of computing,...

DAVE
FERRUCCI: We took a huge jump with machine learning. We were
like, "Woooh! Yes, he got it!"

NARRATOR: ...and creating a machine that can learn from its
mistakes.

WATSON: What is mosquito?

TODD
CRAIN (Training Game Host): No!

WATSON: Holiday?

TODD
CRAIN: No.

WATSON: What is artificial sweetener?

TODD
CRAIN: Come on, now. No.

DAVE
FERRUCCI: We don't understand the question. We don't
understand the category.

DAVE
GONDEK (IBM Research): Yeah, that was one of the tensest days I've ever
had.

DAVE
FERRUCCI: You fall flat on your face. Did I expect to get
fired? No, but...maybe.

We came back and the game was neck and neck.

TODD
CRAIN: Wordsworth said they "soar, but never roam." WATSON?

WATSON: What is skylark?

TODD
CRAIN: That is correct.

NARRATOR: Is WATSON ready to face the game's all time
champions?

KEN
JENNINGS (Jeopardy! Champion): This is too daunting a task for a computer.

NARRATOR: NOVA takes you inside the world of artificial
intelligence and the quest to build a computer like none before it.

ALEX
TREBEK (Jeopardy!): Good luck, WATSON.

WATSON: Smartest Machine on Earth, right now,
on NOVA!

NARRATOR: Dave Ferrucci is a nervous parent.

He's
spent the last four years building a revolutionary new computer, and it's about
to face its biggest test, in front of an audience of millions.

WATSON: Hello, my name is WATSON. I hope we have a good game
today, but first I have to test my voice.

NARRATOR: When the cameras roll, the computer, called
WATSON, will make history, as it competes on the popular quiz show Jeopardy!

ANNOUNCER
(Jeopardy!): This is Jeopardy!

This
is Jeopardy!

DAVE
FERRUCCI: It's frightening, right? It's a different
experience. It's a very different experience for a scientist to sit here and
have this happen live.

NARRATOR: Ferrucci has reason to fear. WATSON is playing
for a million-dollar jackpot against the game's toughest competitors: Brad
Rutter, Jeopardy's biggest money winner, and Ken Jennings, famous
for winning 74 consecutive games.

KEN
JENNINGS: There's some contestant's pride—I want to beat
my human competition—but you know, as a species, I would like mankind to
beat the big bad computer.

WATSON: Let's finish "Leaders of World War II."

TODD
CRAIN: All
right!

NARRATOR: This big bad computer is the culmination of four
years' intensive work. IBM has put WATSON through hundreds of practice games,
with a stand-in host and real contestants.

TODD
CRAIN: After Germany invaded the Netherlands, this queen,
her family and cabinet fled to London. Maria?

MARIA
(IBM Jeopardy! Practice
Contestant) Who is Beatrix?

TODD
CRAIN: No. WATSON?

WATSON: Who is Wilhelmina?

TODD
CRAIN: That is correct.

ERIC
BROWN (IBM Research): It's a human standing there, with their carbon and
water, versus the computer with all of its silicon and its main memory and its
disc.

NARRATOR: It seems like it should be easy for the computer
to win, with its enormous memory and processing power, but the human brain
makes an intimidating opponent, especially on Jeopardy.

DAVE
GONDEK: Jeopardy questions are tricky. They
have puns in them. They have little jokes in them.

LUIS
VON AHN: Just understanding the question is a pretty big
deal.

TODD
CRAIN: This trusted friend was the first non-dairy powdered
creamer. WATSON?

NARRATOR: If WATSON wins on Jeopardy, it will
be a major breakthrough in a quest that's gone on for decades: the audacious
dream to build a machine as smart as a person, the quest for artificial
intelligence, A.I.

MARVIN
MINSKY:
The first programs we wrote at M.I.T.
solved problems that only very educated people could solve, like problems in
calculus and then algebra.

NARRATOR: The computer pioneers thought they were on a fast
track to building human-like intelligence.

CLAUDE
SHANNON (Massachusetts Institute of
Technology/Film Clip): I
confidently expect that within 10 or 15 years, we will find, emerging from the
laboratories, something not too far from the robot of science fiction fame.

PAT
WINSTON (Massachusetts Institute of
Technology): In the beginning,
we thought, "Well, maybe 10 or 15 years and we'll have something that's really
smart."

TERRY
WINOGRAD: In the beginning, people really
were amazed at how much computers could do. When you see something that's
improving very fast, you simply assume it will continue improving that fast,
indefinitely.

NARRATOR: In the '60s, confidence was so high, it inspired
one of the most iconic film characters of all time.

KEIR DULLEA as DAVE (2001: A Space Odyssey/Film Clip): Hello, Hal, do you read me? Do
you read me, Hal?

RODNEY
BROOKS: But, it was intelligent, could
talk to people, could see people, could lip-read, could do all this stuff, a
machine that could do that, and I'd never even seen a real computer, at that
time.

NARRATOR: More than 40 years after the creation of HAL, no
real computer or robot has been able to interact with humans as seamlessly as
Hollywood imagines.

WILLIAM
HURT(Lost in Space/Film Clip): Tell me, what is love?

NARRATOR: The problem is our own human computer, the brain,
a complex entity that's defied any attempts at replication.

LUIS
VON AHN: We just had no idea how sophisticated the brain was.

NARRATOR: The computer has always been...is king when it
comes to calculation and processing huge amounts of data.

TEACHER:
Pen, as in the pen. What's the
middle letter?

SCHOOL
CHILDREN: E!

TEACHER: Excellent.

NARRATOR: But simple skills that humans master early in
life, like understanding language or recognizing objects, continue to baffle
researchers.

TOM
MITCHELL: You know, people vastly misjudged how subtle we are
when we're intelligent. People just hugely underestimated that.

NARRATOR: But the dream of building a computer that could
talk and match wits with humans never really died, and a few years ago a new
plan was hatched, sparked by an unlikely event.

ANNOUNCER
(Jeopardy!): This is Jeopardy!

ALEX
TREBEK: In Wagner's operas, this eldest Valkyrie is
stereotypically dressed in a horned helmet and breastplate.

KEN
JENNINGS: Who
is Brí¼nnhilde?

NARRATOR: In 2004, Ken Jennings' 74-game winning streak on Jeopardy
set the country abuzz, and caught the eye of an IBM executive while out to dinner.

CHARLES
LICKEL (IBM Research): All of a sudden, the entire restaurant cleared out
to the bar that I am sitting at, to go see Ken Jennings play Jeopardy.

ALEX
TREBEK: This Grand Ole Opry comedy star used to wear a straw
hat with the $1.98 price tag still attached.

KEN
JENNINGS: Who
is Minnie Pearl?

NARRATOR: Charles Lickel wondered if a computer could ever
play as well as Ken Jennings. So he pitched the idea to some of IBM's top
scientists.

CHARLES
LICKEL: For the ones that knew Jeopardy,
they said, "Charles, that's just too hard." Um, I think the, the prevailing
view was, the, these questions were difficult to understand, difficult to even
comprehend what was being asked.

DAVE
GONDEK: Yeah, I was like, "No way!" I was like, "No way!"

NARRATOR: But one researcher, Dave Ferrucci, was intrigued.

DAVE
FERRUCCI: My view was, maybe this isn't as completely
impossible as we think it is.

NARRATOR: For over 40 years, Jeopardy has
been pop culture's I.Q. test. Clues are given as answers, and contestants have
to respond in the form of a question.

KEN
JENNINGS: The show's central conceit is a little syntactic
reversal, whereby they give you an answer and you supply a question, you don't
say "George Washington," you say "Who is George Washington?"

NARRATOR: To win, contestants need to be human
encyclopedias.

KEN
JENNINGS: It's essentially everything under the Sun. You know
the categories at the same second Alex tells the folks at home the categories.

ALEX
TREBEK: One, you have to have a broad knowledge, because we
have 13 categories on each show.

Kate,
start.

KATE
( Jeopardy Contestant): "P.h."
for 400.

ALEX
TREBEK: For
the record, Thomas Edison invented the first practical one of these 1877.

NARRATOR: Contestants also have to be fast.

ARIEL( Jeopardy Contestant): What is the
phonograph?

ALEX
TREBEK: Good!

NARRATOR: They typically have three seconds or less to come
up with an answer.

ALEX
TREBEK: The
mortar and pestle is a symbol of this profession. Ariel?

ARIEL
:
What is a pharmacist?

ALEX
TREBEK: Pharmacist
is right.

NARRATOR: To compete on Jeopardy, IBM's computer must have
an enormous knowledge base, because it will not be connected to the internet.
But the far bigger challenge for the machine will be understanding clues which
can be extremely convoluted or obscure.

ALEX
TREBEK: You'll find this flower before "Pickle Bottom" in a
line of handbags and bedding.

That
would be petunia. Back to you, Ariel.

KEN
JENNINGS: There will be a lot of puns. There'll be double
meanings. And these are things that computers historically are terrible at.

NARRATOR: Human language is a minefield for computers.
Consider this sentence.

DAVE
FERRUCCI: How's it go? "I shot an elephant wearing my pajamas."
Was I wearing the pajamas? Was the elephant wearing the pajamas? So there are
different interpretations, different ways to parse the sentence. The word shot.
What's really going on there? There's already ambiguity in there. Could
actually be shooting, sort of a hunting shooting; if I'm a photographer and I'm
immersed in that context, I may interpret that as shooting with a camera. Which
one did I mean? You have to look at the context.

NARRATOR: But a computer has no context. It's just an
electronic brain in a box.

In
2006, Ferrucci tackles this challenge, along with the best and brightest
programmers from IBM and the country's top A.I. labs.

To
start, they run a test.

DAVE
GONDEK: We had a, um existing state of the art system that
people had worked on for a number of years, and we tried applying that to the Jeopardy
challenge.

NARRATOR: They feed one of IBM's most sophisticated
computer programs hundreds of Jeopardy questions, like this one: "In
1698, this comet discoverer took a ship called the Paramour Pink on the first
purely scientific sea voyage." The correct answer is "Who is Edmond Halley?" The computer says, "Who
is Peter Sellers?"

The
computer ran a search through a million documents, looking for key words from
the clue. It homed in on a description of one of The Pink Panther films, in which one character was a paramour,
or mistress. The star of movie? Peter Sellers.

It's
probably the last answer a human would come up with, but it's typical for
computers. The team has a long way to go.

Just
how far becomes clear when they compare the computer to the best human players.
They create a graph called the "Cloud." Each dot represents a Jeopardy champion's performance. Jennings
is at the top.

DAVE
GONDEK: What you see is a cloud, around...they answer around
50 percent, the winners do. And they get around 90 percent correct.

NARRATOR: And where is the computer in this cloud?

DAVE
GONDEK: If you ask it to answer all the questions, it would
be giving you 10 percent of the questions right. You can't go on Jeopardy
like that. I mean, the best humans are 90 percent, 92 percent. We weren't even
close.

NARRATOR: To win at Jeopardy, the team will
need a whole new way to tackle human language, one that takes advantage of the
computer's basic strengths.

At
its electronic core, a computer speaks a very simple language: binary code, on
or off. But with that simple code, it can follow instructions and solve complex
problems once reserved for intellectual giants.

LUIS
VON AHN: It used to be the case that intelligence was chess, right?
If you can play chess, that's intelligence.

NARRATOR: Computers have mastered the game.

LUIS
VON AHN: Chess is easy for computers, because the rules are
very well defined and very clear.

NARRATOR: The rules of chess are relatively simple: a board
of 64 squares; each piece—pawn, knight, queen—can move a certain
way; and there's a single goal: take out your opponent's king.

For
humans it is the ultimate game of strategy.

RODNEY
BROOKS: The way computers play chess is
not at all the way people play chess. We humans look at the board and have
conceptual ideas like, "Control the center," "Attack on the right." Very
different from the way computers play chess.

NARRATOR: A chess-playing computer looks at virtually every
possible move it could make and every response, every way the game could play
out.

RODNEY
BROOKS: Computers play chess through
searching a tree of moves down to a very deep level, looking ahead on every
possible path. But they do it by brute force, by going 20, 30, 40 moves ahead
and seeing all the bad things that can happen. A person can't look that many
moves ahead, broadly.

NARRATOR: This is the power behind the most famous chess
game in the history of A.I., when, in 1997, another IBM computer, named Deep
Blue, beat the reigning world champion, Gary Kasparov.

NATURAL
SOUND CORRESPONDENT (1997): The chess world
champion walked away from the match, never looking back at the computer that
had just beaten him.

NARRATOR: The victory makes Deep Blue look pretty smart,
but is it?

PATRICK
WINSTON: Deep Blue, it's only acting as if
it's intelligent, and it's not really intelligent in the way that we humans
are.

RODNEY
BROOKS: It's good at one thing. It's
playing chess. It can't do anything else. It has no other understanding of the
world. It's just about chess moves.

NARRATOR: This lack of understanding has hampered every
computer program that's tackled human language.

A
perfect example is a program from the 1960s, called Eliza.

TERRY
WINOGRAD (Stanford University): Eliza was one of the first programs that had
anything resembling human conversation. It was a dialogue. You typed things in,
it typed things back.

ELIZA
VOICE: How
do you do? Please tell me your problem.

TERRY
WINOGRAD: "I am feeling sad."

Then
it types back, "Did you come to me because you are feeling sad?"

NARRATOR: Eliza was programmed to respond like a
psychiatrist, but it had no real insight. Instead, it followed simple rules and
rearranged key phrases.

TERRY
WINOGRAD: So if I say, "I'm dead," it
responds, "Do you enjoy being dead?" It doesn't have any understanding that
dead is a different kind of condition. It really is just doing this sort of
fill-in-the-blanks kind of pattern matching.

NARRATOR: Anyone who tried to solve the language problem
hit the same brick wall: the computer's profound ignorance of what we take for
granted every day.

LUIS
VON AHN: There is just so much more that we know that we don't
know we know. I mean, just, we know all kinds of stuff, like you press the up button in the elevator, that
means it's going to go up. Or, milk is white, or water is wet. I mean, there's
just stuff that we know that we don't even realize we know. That's one of the
things that makes it hard.

NARRATOR: All the common sense knowledge a human brain
collects naturally seems much too complex to program into a computer. But that
hasn't stopped one scientist from trying.

DOUG
LENAT (Cycorp, Inc.): So we have actually manually entered about 6,000,000
rules. That's about three percent of what it's going to need to know in terms
of actually spanning what you and I would call human common sense.

NARRATOR: For the last 25 years, Doug Lenat has been
leading a team trying to create human-like intelligence by teaching a computer
common sense, rule by rule. The program is called Cyc, and at headquarters, the
walls are covered with logic diagrams.

DOUG
LENAT: In a way, the, the magic of this, the power of this
is if you just tell it each rule, one by one, by one, and you give it general
logical reasoning capabilities, that's all you need to do.

NARRATOR: So far, Cyc has 6,000,000 rules and can answer a
lot of common sense questions. Like this one:

NARRATOR: The entire World Book Encyclopedia,
Wikipedia, the Internet Movie Database, much of the New York Times archive, and the Bible are just some of WATSON's
resources.

And
to build on WATSON's foundation of data and rules, the team turns to a powerful
tool in the computing world. It's called machine learning.

SEBASTIAN
THRUN (Stanford University): Machine learning is just like human learning from
examples.

LUIS
VON AHN: Before, people would just write rules, write rules
by hand. Nowadays, it is all based on examples.

NARRATOR: To understand how machine learning works,
consider for a moment the letter A. What if you had to describe it to a
computer? It's a real problem faced by the U.S. Postal Service, whose computers
must decipher all kinds of addresses, printed and handwritten.

LUIS
VON AHN: We all know what an A looks like, I know when I see
it, but there is just way too many different types of As. There are fonts where
the A is just a triangle pointing up, that is an A.

Pretty
quickly, you realize there is no simple set of rules that you can write down,
currently, for a program to determine whether a letter is an A or not.

NARRATOR: Humans might not be able to come up with the
rules that reliably identify all kinds of As, but, it turns out, a computer can
do it for itself, if you give it enough examples.

LUIS
VON AHN: The way you do it, you just get an A, send it to the
program and say that is an A. Here is another A, different one. That is an A. Here
is another A. This is a different one. That is an A. Then you would give
another example, and you would give another example, and you would do that a
million times.

NARRATOR: The computer hunts for patterns among all those
examples, and it finds them. So, the next time it meets a letter A, even one it
hasn't seen before, it will recognize it. This is machine learning, and it's a
crucial element of WATSON's programming.

The
team trains WATSON, but here, instead of letters, the examples are tens of
thousands of old Jeopardy
questions, along with a cheat sheet of all the correct answers. Using machine
learning, WATSON will hunt for patterns between the type of question, the
correct answer, and the kinds of evidence that support that answer.

DAVE
GONDEK: Now, we do this over thousands of questions, so we
come up with some way to weigh the evidence on average, so that we come up with
the right answer.

NARRATOR: Now when he's faced with a brand new question,
WATSON uses what he learned from these patterns, and declares his confidence in
each possible answer.

DAVE
FERRUCCI: And in the end, we get a list that says, "Here's the
top answer, and we're 75 percent sure it's right."

NARRATOR: WATSON has now become a complex architecture of
rules, raw data and machine learning that enables him to use statistics to
choose the right answer.

To
test out this system, the team scours the halls for IBM employees who can play Jeopardy, and everyone squeezes into a
conference room.

The
Fifth Amendment says that private property shall not be taken for public use
without this.

WATSON: What is "just compensation?"

CHRIS
WELTY: Yes!

DAVE
FERRUCCI: We took that huge jump with machine learning.

CHRIS
WELTY: WATSON,
with a commanding lead of 24,863.

DAVE
FERRUCCI: We saw a huge jump in performance, and we were, like,
"Whooo!

NARRATOR: Up to now, appearing on the TV show has only been
a dream, but WATSON is performing so well, Dave Ferrucci decides it's time to
call Jeopardy.

In
December, 2009, Jeopardy
producers arrive at IBM to size up Dave Ferrucci's new creation. Like any human
contestant, WATSON must audition to earn his spot on the show.

DAVE
FERRUCCI: We spent all this time, you know, developing this
system, and pushing its capabilities. Then here you are, sitting here, all the
executives are there,...

HARRY
FRIEDMAN: You hear "computer" you think, "Uh,
well, of course a computer should have all the answers." You hear about Q&A
technology, well isn't this just a big search engine?

DAVE
FERRUCCI: ...and they're waiting to see, you know, what really
is going to happen. And you just don't know, you don't know.

NARRATOR: To impress the executives, IBM builds a makeshift
studio, hires comedian Todd Crain to act as game show host, and brings in
former TV contestants.

DAVE
GONDEK: Yeah, that was one of the tensest days I've ever
had. Because we'd never seen it play against Jeopardy players.

And,
I remember, like, the day before, we're tuning everything, and I was putting in
the best strategy that we had, I was putting the best stuff that we had, and I
thought, "Well, this is just going to kill them!"

MIRANDA
(Former Jeopardy!
Contestant): What is "The Cat's in the
Cradle?"

TODD
CRAIN: That is correct.

DAVID
(Former Jeopardy!
Contestant): What is "I Am the Walrus?"

TODD
CRAIN: Yes!

MIRANDA
:
What is "Crocodile Rock?"

TODD
CRAIN: Yes!

DAVE
GONDEK: You know, they were just like professional athletes,
you know, and it was a really tough few games for us.

NARRATOR: In the first round, it seems that WATSON is
auditioning not for a game show but a sit-com.

TODD
CRAIN: In 1682, he came to the throne at the age of 10,
along with his weak-minded half-brother, Ivan V. WATSON?

WATSON: What is Peter?

TODD
CRAIN: More specific?

WATSON: What is Peter 'I'?

TODD
CRAIN: No! Carrie or David? Carrie!

CARRIE
:
Who is Peter the Great?

NARRATOR: In "Final Jeopardy," where contestants must place
bets and write down the answer, things only get worse. Under the category "Flags,"
the clue is, "In a policy begun in 2002 as a symbol of the war on terrorism,
U.S. Navy ships fly the 18th century flag with this four-word motto."

TODD
CRAIN: You need to know a little bit something about18th
century flags. David, let's see if you did. What is the four-word motto we're
looking for, David?

DAVID
:
What is, "Don't tread on me?"

TODD
CRAIN: That is correct! Let's see if WATSON got it right. "What
is September 11, 2001 attacks?"

CARRIE
:
Whoa!

NARRATOR: WATSON did not recognize the word "motto," and
after scanning through millions of documents, he found the word terrorism
associated with September 11th so frequently that seemed like the best answer.

By
the time they break for lunch, it's humans two, WATSON zero, and it's not clear
if WATSON will ever be ready for primetime.

DAVE
FERRUCCI: This was taking a risk, for me, in the sense that we're
sitting here and saying, "You know what? I think this is possible." And then
you fall flat on your face, so... "We're never going to believe Ferrucci again."

Did
I expect to get fired? No, but...maybe.

NARRATOR: But after lunch, the producers are treated to a
different side of WATSON.

DAVE
FERRUCCI: We came back, and the third game was neck and neck,
incredibly competitive.

TODD
CRAIN: In Act Three of an 1846 Verdi opera, this Scourge of
God is stabbed to death by his lover, Odabella. WATSON?

WATSON: What is Attila?

TODD
CRAIN: Be more specific?

WATSON: What is Attila the Hun?

TODD
CRAIN: Thank you, very much; Attila the Hun. I'll take
that.

NARRATOR: That afternoon, WATSON climbs back in the game.

TODD
CRAIN: Wordsworth said these "soar, but never roam."

WATSON: What is skylark?

TODD
CRAIN: It's a device clamped to the wheel of a parked car.
WATSON?

NARRATOR: It may appear that WATSON has redeemed himself,
but the producers are troubled by his erratic performance. Their verdict:
WATSON isn't strong enough for Jeopardy, at least, not yet.

Why
is WATSON so erratic?

To
understand his weaknesses, you have to appreciate the complexity of the task.
Consider this clue: "Keanu Reeves had a Nokia phone, but it took a landline to
slip in and out of this, the title of a 1999 sci-fi flick." The correct
response is, "What is The Matrix?"
But how can WATSON figure that out?

First,
he breaks down the clue into grammatical parts, identifying key words and
phrases. Then, WATSON's powerful search engines churn through millions of
documents, including the Internet Movie Database.

DAVID
GONDEK: What we do next is we take these documents and we pull out candidate
answers. And we'll pull out "Keanu Reeves." That could be a candidate. We'll
pull out "Nokia;" we'll pull out "The Matrix."

NARRATOR: Other movies starring Keanu Reeves also become
possible answers.

DAVID
GONDEK: We'll pull out "The Matrix 2;" we'll pull out "Speed," "Bill & Ted's Excellent Adventure," all this stuff.

KEANU REEVES (Bill & Ted's Excellent
Adventure/Film Clip): Whoa!

NARRATOR: And WATSON pulls out other famous sci-fi flicks,
like "Blade Runner."

DAVE
FERRUCCI: And it generates hundreds of possible answers.

NARRATOR: With hundreds of choices, how can WATSON pick the
one answer that's correct?

DAVE
FERRUCCI: The next thing that WATSON is going
to do, it's going to take those answers and say, "Well, let's assume all of
them might be right." So, these are its competing hypotheses.

NARRATOR: WATSON starts considering evidence for and
against each candidate, using rules like "A movie is sometimes called a flick."

DAVE
GONDEK: And we'll look at things like, well it's looking for
a flick. Is this candidate answer a flick? Is The Matrix a flick?
Yes. Is Speed a Flick? Yes. Is Keanu
Reeves a flick? No.

Okay,
so we're starting to learn something.

NARRATOR: Within a matter of milliseconds, WATSON analyzes
every possible answer in hundreds of different ways and scores each piece of
evidence behind every answer in the list.

That's
a lot of scores.

DAVE
GONDEK: The problem is you have all these different scorers
and they don't agree. Some of the scorers are going to say "The Matrix"
is the right answer. Some of the scorers are going to say "Keanu Reeves" is the
right answer. Some are going to think "Matrix 2" is the right answer.

NARRATOR: And a lot of scorers think "Blade Runner"
is the right answer, because it shows up so often as a sci-fi flick.

DAVE
GONDEK: So you need someone at the end who is going to
listen to all these votes and decide what's going to be the best answer.

NARRATOR: This is where WATSON's machine learning kicks in.
Having studied thousands of other Jeopardy questions and their
correct answers, WATSON has learned what evidence is important and what's not.

DAVE
FERRUCCI: What machine learning will start
to do is learn how to weigh them differently. Say, "Wait, questions like this,
calling on a phone, not calling on a phone, not so important. This other stuff,
"Do I have a sci-fi movie? Is the person named a character in that movie?" Very,
very important for questions like this.

NARRATOR: In this case, he successfully weighs the evidence
and identifies sci–fi flicks from 1999, starring Keanu Reeves. So, he
picks the one answer matching all those elements, "The Matrix."

WATSON's elaborate system doesn't always work,
but without machine learning, he wouldn't stand a chance.

Machine
learning isn't just important for WATSON. It's driving a revolution in
computing. It plays a major role in computer models that predict the weather,
days in advance. And all those recommendations you get from Amazon or Netflix?
No human is writing up rules about your likes and dislikes. Instead, computers
are comparing your preferences to millions of other customers and finding
patterns and learning about you.

Today
machine learning is conquering many problems once thought too complex for
computers. Like speech recognition.

TOM
MITCHELL (Carnegie Mellon University): In the earlier days, people decided that they would
try to program computers to recognize speech.

Which
word am I saying now?

RESEARCHER
(Video/Clip): Pick up the big block at the
right side.

NARRATOR: In the '60s, this voice-directed block world was
the height of technology. Computers could be programmed to recognize the audio
signals of specific words and phrases.

RESEARCHER
(Video/Clip): Pick up the big block at the
right side.

Pick
up every small block.

NARRATOR: But they had to be re-programmed for every new
speaker, because everyone's speech is slightly different.

TOM
MITCHELL: Even though it is very easy for you and I to
recognize the word "ice cube..."

BOY's
VOICE: Ice cube.

TOM
MITCHELL: ...it is very difficult for us to write down the rules
that would allow a computer to look at the microphone signal and see that is "ice
cube."

MANY
VOICES: Ice
cube.

NARRATOR: But now, computers are trained with millions of
examples of human speech.

TOM
MITCHELL: Here is the microphone signal, and this is the word "ice
cube."

WOMAN'S
VOICE: Ice cube.

TOM
MITCHELL: Here is another one.

BOY'S
VOICE: Ice cube.

TOM
MITCHELL: You end up with much more successful speech
recognition systems.

NARRATOR: Today, speech recognition software, though not
perfect, is remarkably accurate and getting better all the time.

TOM
MITCHELL: All the ones we have today are based on machine
learning, simply because it works the best.

NARRATOR: And some programs are taking it a step further,...

ALEX
WAIBEL (Carnegie Mellon University): Where do you come from?

NARRATOR: ...not only transcribing your speech,...

CUSTOMER
: Shanghai.

NARRATOR: ...but translating it into a foreign language as
well.

COMPUTER: Shanghai.

ALEX
WAIBEL: It's
very nice to meet you.

COMPUTER: (Translation
of "It's very nice to meet you into Chinese)

NARRATOR: Every language has so much ambiguity, double
meanings and metaphors,...

ALEX
WAIBEL: All
of these interactions are so complex that you couldn't, in your lifetime, write
all these rules It's just too enormous, too daunting an effort.

My
wife asked me to buy some crackers.

STORE
EMPLOYEE: (Asks question in Chinese)

COMPUTER
TRANSLATOR: What kind of crackers?

ALEX
WAIBEL: Rice
crackers.

NARRATOR: Alex Waibel fed a computer millions of examples
of English texts, together with their translations into about a dozen different
languages. Now he's got a program that can run on your phone or iPad...

ALEX
WAIBEL: Do
you often shop here?

COMPUTER
TRANSLATOR: Do you often shop here?

SHOPPER: (Replies
in Chinese) Every day .

NARRATOR: ...and translates on the go.

Machine
learning has been so successful, mastering more and more tasks once only done
well by humans.

ALEX
WAIBEL: The
computer program translates between languages.

NARRATOR: Some researchers believe it may be a crucial
building block for making real artificial intelligence.

TOM
MITCHELL: There are two ways of building intelligence: you
either know how to write down the recipe or you let it grow itself. And it is
pretty clear that we don't know how to write down the recipe. Machine learning
is all about giving it the capability to grow itself.

NARRATOR: Some people find the idea of a machine that can
learn threatening, but when WATSON's having a bad day, it's hard to imagine him
threatening anyone.

DAVE
GONDEK: Did
you see the "Daily Double?"

DAVE
FERRUCCI: No.

DAVE
GONDEK: So,
we were way ahead. We were almost locking out.

TODD
CRAIN: The other "Daily Double!"

DAVE
GONDEK: We
get the "Daily Double," there's like two or three clues left. And so what do we
do? We bet big so we can lock them out.

WATSON: I'll wager $5,200.

TODD
CRAIN: We-e-el!

NARRATOR: The team is nervous. They know WATSON will never
make the cut on Jeopardy, unless he can stop making dumb
mistakes.

TODD
CRAIN: Here's your clue.

DAVE
GONDEK: It was about letters, and it was, "A woman wrote to
this '40s artist...

TODD
CRAIN: In the late '40s, a mother wrote to this artist that
his picture "Number 9" looked like some of her son's finger-painting.

DAVE
GONDEK: We answered with "Rembrandt."

WATSON: Who is Rembrandt?

TODD
CRAIN: Really?

NARRATOR: Although WATSON recognizes most dates, he doesn't
know that the '40s refer to the nineteen forties.

DAVE
GONDEK: It's a '40s artist. We answered with "Rembrandt," so
there's a time, a time problem.

TODD
CRAIN: Very bad. We all know that it was Jackson Pollock.

NARRATOR: WATSON loses that game.

DAVE
GONDEK: We double wrong and "Final Jeopardy!"

NARRATOR: And his opponents show him no mercy.

MEHRUN
(Former Jeopardy!
Contestant?): We both beat him!

MARIA:
Good for you!

MEHRUN:
Humans! Whooo!

DAVE
FERRUCCI: Sometimes the reason something is the right answer
is very obvious to a human, like, for example, it may be asking for a she or a
he.

TODD
CRAIN: This
first lady was born Thelma Catherine Ryan, on March 16, 1912, in Nevada. WATSON?

WATSON: Who is Richard Nixon?

DAVE
FERRUCCI: Oh, here you go! Right?

TODD
CRAIN: Patricia?

PATRICIA
(Former Jeopardy!
Contestant?): Who is Pat Nixon?

TODD
CRAIN: That is correct. Richard Nixon was never a First
Lady.

DAVE
FERRUCCI: I want to understand what went on there.

DAVE
GONDEK: Our
new gender stuff is not in the system.

CHRIS
WELTY: People take offense at being called the wrong
gender. WATSON doesn't care about stuff like that. It's making statistical
judgments based on how pieces of evidence have gone together in questions and
answers that we've given it.

TODD
CRAIN: The two famous comedians' noses that make
impressions.

WATSON: What is Jimmy Durante?

TODD
CRAIN: More specific?

WATSON: Sorry, all I know is "What is Jimmy Dur-an-tay?"

TODD
CRAIN: Saying it slower doesn't make it right.

DAVE
FERRUCCI: Now, you hear how Todd Crain
makes fun of the computer? Now, I had my kids, I had them sign the
confidentiality agreement and come in and see a couple of these games. And
their comment was, "Why does the host make fun of WATSON, Daddy?"

TODD
CRAIN: This what-are-you-doing? Web site's
name also refers to a type of nervous laugh.

WATSON: What is "evil laugh?"

TODD
CRAIN: Ho ho! No!

In
terms of comedy duos, he is the best straight man in the business,...

You're
going to kick yourself. Twitter!

...because
he doesn't, he doesn't get it. He doesn't get why his inappropriate answer is
funny, and you can't ask for better writing than that.

DAVE
FERRUCCI: Once in a while, okay, we can all take a joke, but
over and over again? And WATSON's defenseless, right? So he's making fun of and
criticizing a defenseless computer that represents people with real feelings
and real families. All right, maybe if I don't have any feelings, my kids have
feelings.

NARRATOR: And feelings are intensified by the fact that
there's just one more month before the Jeopardy producers will
return to make a decision.

TODD
CRAIN: WATSON, you have control.

NARRATOR: The pressure is on to boost WATSON's strengths
and eliminate his weaknesses. His strengths are obvious. He dominates when it
comes to purely factual questions.

ERIC
BROWN (IBM Research): WATSON usually does very well at these factoid
questions, looking up facts, you know? History, geography, entertainment.

TODD
CRAIN: Clark Gable was happy to see him come in and finish
directing Gone with the Wind. WATSON?

TODD
CRAIN: Good for 12. Last clue on the board for $800, here
it is; Revenge of..., in 1978 was the fifth in this series of
comedies directed by Blake Edwards.

WATSON: What is The Pink Panther?

TODD
CRAIN: I just want to check, your buzzers are working? Just
wanted to, wanted to check in and wake you from your nap.

NARRATOR: WATSON has made it into the cloud, but nowhere
near championship level. Ferrucci must come up with a plan to somehow step up
his performance, before the Jeopardy producers return.

The
team has spent almost four years, enormous intellectual and emotional energy,
and tens of millions of dollars on developing WATSON. But all this effort isn't
just for a machine that can play Jeopardy.
IBM has much bigger goals.

DAVE
FERRUCCI: ...and dialoguing with you to make sure that you get
what you want.

COMPUTER
(Star Trek/Film Clip): Affirmative.

DAVE
FERRUCCI: Do you have one of those? I, I, I don't have one of
those, right?

NARRATOR: You don't need to be a starship commander to find
this helpful.

In
a world overflowing with data, intelligent expert systems that can answer vital
questions have been the Holy Grail. Now, we could be close to a WATSON, M.D.

DAVE
FERRUCCI: Well, think of it this way, there's a bunch of
information: new diagnoses, new treatment options, new discoveries. Can anybody
keep that all in their head all at once?

NARRATOR: A machine that could access and organize all that
information could help doctors analyze symptoms and wade through piles of
medical journals.

DAVE
FERRUCCI: It changes the paradigm in which we work with
computers. That's the vision behind this.

WATSON: How
am I doing? I hope we will have a good game today, but first I have to test my
voice.

NARRATOR: Before that vision can be fulfilled, before
WATSON can even compete on Jeopardy, the team needs to get more
bugs out of his system.

One
of WATSON's most embarrassing weaknesses is he cannot hear. Instead, WATSON
receives each Jeopardy clue as an
electronic text message, at the same moment his competitors see it on the
board. As a result, he doesn't know the other contestants' answers.

TODD
CRAIN: Only the female of this "equine" pest, of the family
Tabanidae, feeds on blood; the male feeds on nectar.

BILL
(Former
Jeopardy! Contestant?) : What's
a mosquito?

TODD
CRAIN: No. WATSON?

WATSON: What is mosquito?

TODD
CRAIN: No!

HARVEY
(Former Jeopardy!
Contestant?): What's a horsefly?

TODD
CRAIN: Yes! Thank you for not saying mosquito! Good job. Good
for $2,000, Harvey.

NARRATOR: Ferrucci and the team have been working furiously
to boost WATSON's performance. As part of their plan, WATSON will now receive
correct answers electronically, after they're revealed. If the fix works, it
will be in the nick of time.

DAVE
FERRUCCI: Hey, how are you?

NARRATOR: The Jeopardy producers are back and
they're about to determine WATSON's fate.

DAVE
GONDEK: This was a measure of our progress, and we wanted to
hear, "Yeah, you're there. You've made it."

NARRATOR: The new and improved WATSON gets his first big
test with a category called "Celebrations of the Month."

TODD
CRAIN: Administrative Professionals Day and National C.P.A.s
Goof-Off Day.

WATSON: What
is "holiday?"

TODD
CRAIN: No! That's not even close, really.

NARRATOR: WATSON fails, because he doesn't understand that
in this category, the answer must be the name of a month, something his human
competition quickly figures out.

TODD
CRAIN: Arthur?

ARTHUR
(Former Jeopardy!
Contestant?): What is April?

TODD
CRAIN: What is April? April 18th is...

DAVE
FERRUCCI: I don't understand. I don't understand why we don't,
we don't understand the question. We don't understand the category, basically.

TODD
CRAIN: D-Day Anniversary and Magna Carta Day.

NARRATOR: But now he electronically receives these correct
responses.

TODD
CRAIN: Arthur!

ARTHUR
:
What is June?

NARRATOR: Can he learn from the answers?

TODD
CRAIN: National Philanthropy Day and All Souls' Day.

DAVE
GONDEK: What WATSON does here is, it sees that, "Well, all
the answers I've seen have been the month in which the thing in the clue
occurs."

TODD
CRAIN: Matt?

MATT (Former Jeopardy! Contestant): What is November?

TODD
CRAIN: Good for four.

DAVE
GONDEK: So then it knows, in the next clue, to look for, um,
"What month does this thing occur in?"

MATT: "Celebrations" for six.

TODD
CRAIN: "Celebrations" for six. National Teacher Day and
Kentucky Derby Day. WATSON?

WATSON: What is May?

DAVE
GONDEK: Yes!
He got it! He figured it out!

It
took us four!

NARRATOR: Thanks to the team's efforts, WATSON is soaring
higher in the cloud and now approaching the level of champions like Ken
Jennings.

WATSON
is on a roll.

TODD
CRAIN: Choose again.

WATSON: "You're Tripping," for 1,600.

TODD
CRAIN: Jacques Cartier found this largest island of the
Hochelaga Archipelago while searching for gold. WATSON!

WATSON: What is Montreal?

TODD
CRAIN: Yes.

WATSON: Who is Zebulon Pike?

TODD
CRAIN: Good.

WATSON: What is Providence?

Aquarius?

Texas?

Beau
Brummell?

TODD
CRAIN: WATSON?

WATSON: What is "ciao?"

TODD
CRAIN: That was right and cute all at the same time!

NARRATOR: The Jeopardy executives have
finally seen enough.

HARRY
FRIEDMAN: I think we've gone from
impressed to blown away.

TODD
CRAIN: Very nicely done, WATSON!

VOICE
OFFSTAGE: Quiet!

NARRATOR: Finally, WATSON will get his chance on Jeopardy:
a computer, playing against human champions,...

ALEX
TREBEK: They
say it can think...

NARRATOR: ...in a game that is a very symbol of intelligence.

ALEX
TREBEK: ...but
can it think like a Jeopardy champion?

NARRATOR: It will be a contest the world has never seen.

ALEX
TREBEK: Good
luck, WATSON.

NARRATOR: But does this mean that the dream of artificial
intelligence is coming true?

RODNEY
BROOKS: So artificial intelligence, to me,
is trying to get computers to do stuff that, if people did them, you'd say, "Oh,
they're demonstrating their "peopleness." That's what makes humans humans, that
stuff they're doing."

NARRATOR: But without experience or emotion, can a computer
like WATSON ever learn and understand the world the way we humans do, from
early childhood on?

PAT
WINSTON: Right now, no machine can understand the meaning of
a play, what it means to be King Lear or Macbeth or Hamlet... "to be or not to be:
that is the question."

No
machine can understand the parables of the Judeo-Christian Bible. All they can
do is shovel through data and find regularities.

NARRATOR: But for Dave Ferrucci, that kind of understanding
was never the goal.

DAVE
FERRUCCI: It's not going to emerge as a human, because it
doesn't connect the information to human experience, to human cognition. When
you think about a great symphony, and when a human sits down with that, that
music is affecting that human on an emotional level. The computer doesn't have
that human experience, doesn't have that human emotion. It's not human, it's a
computer.

NARRATOR: WATSON may never experience the world the way we
do, but with his enormous knowledge base, his skill at interpreting language,
and his ability to learn...

TODD
CRAIN: WATSON?

WATSON: What is May?

DAVE
GONDEK: He
figured it out!

NARRATOR: ...could he actually be considered intelligent?

SEBASTIAN
THRUN: Oh, my god. It is more intelligent
than the average Jeopardy player in answering Jeopardy questions. That's impressively intelligent.

WATSON: Good afternoon, Mr. Trebek. I've been waiting for
this moment for a very, very, very long time.

NARRATOR: Finally, it's show time. WATSON is now taking the
stage, where his intelligence will be put to the ultimate test in front of
millions of Jeopardy viewers.

DAVE
GONDEK: I think I dream about Jeopardy
questions now. I have nightmares about Jeopardy questions.

NARRATOR: Has the team done enough?

They're
about to find out, as WATSON meets the world's two best Jeopardy players: Brad Rutter and Ken
Jennings.

DAVE
GONDEK: We've never had this caliber of a player. And there's
a reason why Ken won 74 games in a row; there's a reason why Brad has never
been beat by a human.

NARRATOR: The whole team has been waiting four years for
this moment.

ALEX
TREBEK: Let's play Jeopardy! Here we go!

Four-letter word for the iron fitting on the hoof of
a horse or a card-dealing box in a casino.

WATSON: What is shoe?

ALEX
TREBEK: You are right. You get to pick.

NARRATOR: At first, man and machine look evenly matched.

ALEX
TREBEK: WATSON!

WATSON: Who is Jude?

ALEX
TREBEK: Yes!

WATSON: What is Last Judgment?

ALEX
TREBEK: Right.

Brad.

BRAD
RUTTER: What is the nineteen-aughts or the 1900s?

ALEX
TREBEK: Yes! Ken.

KEN
JENNINGS: What is Greece?

NARRATOR: At the end of the first round, it's anyone's
game.

ALEX
TREBEK: Sauron is right. That puts you into a tie for the
lead.

DAVID
GONDEK: The first round, five thousand-five thousand. They could beat us. We
have seen it happen in sparring matches.

DAVE
FERRUCCI: Certainly, if you wanted a cliffhanger after the
first game, it is going well. Me? Not so much.

NARRATOR: But as the match continues, WATSON dominates,
consistently beating the humans to the buzz.

ALEX
TREBEK: WATSON.

WATSON: Who is Isaac Newton?

ALEX
TREBEK: You are right!

WATSON: Who is C.S. Lewis?

ALEX
TREBEK: Yes!

WATSON: What is guitar?

ALEX
TREBEK: Right.

NARRATOR: It starts to look like a blowout...

ALEX
TREBEK: And with that you move to 36,681.

NARRATOR: ...when, in the last round, Ken Jennings stages a
comeback.

ALEX
TREBEK: This national newspaper raised its newsstand price
by 25 cents, to a dollar. Ken?

KEN
JENNINGS: What is USA Today?

ALEX
TREBEK: Right.

Ken?

KEN
JENNINGS: What are national borders?

ALEX
TREBEK: Good.

KEN
JENNINGS: Martin Luther King?

ALEX
TREBEK: Yes, you're right!

NARRATOR: Going into "Final Jeopardy," anything can happen.

DAVE
FERRUCCI: We just want to avoid a stupid answer at this point.

ALEX
TREBEK: Here is the clue: William Wilkinson's An Account
of the Principalities of Wallachia and Moldavia inspired this author's
most famous novel.

"Who is Bram Stoker?" You are correct; the author of Dracula.

Over to Ken Jennings now, and we find, "Who is
Stoker?"

I,
for one, welcome our new computer overlords.

NARRATOR: If WATSON gets this one, he will win.

DAVE
FERRUCCI: If we get "Final Jeopardy" right, I'm going to kiss
Eric.

ALEX
TREBEK: Now we come to WATSON. Looking for Bram Stoker, and
we find, "Who is Bram Stoker?"

DAVE
GONDEK: It's kind of overwhelming. We won. And I think
people will say, "Yup, WATSON was the best player there."

NARRATOR: No question, WATSON's victory is a major
milestone for A.I., but perhaps it's an even greater achievement for the human
intelligence that created him.

ETS
IBM Corporate Archives
MIT Museum and Office of Naval Research
iStockphoto
The Conus Archive
Dept. of Special Collections and University Archives, Stanford University Libraries

The producers gratefully acknowledge the cooperation of IBM Research and the producers of Jeopardy!

Special Thanks

Andrzej Rucinski, University of New Hampshire, Durham
Adam Perkins, University of New Hampshire, Durham
The Duquesne University Chess Club
Moharimet Elementary School
Raj Reddy, Carnegie Mellon University
Cat Anderson

National corporate funding for NOVA is provided by Draper and 23andMe.
Major funding for NOVA is provided by the David H. Koch Fund for Science, the Corporation for Public Broadcasting, and PBS viewers.