Creators: Watson has no speed advantage as it crushes humans in Jeopardy

Watson crushed competitors Ken Jennings and Brad Rutter at their first …

Watson answers a question in the topic "Étude, Brute?", showing its top three answers and the corresponding confidence.

Watson, the computer built by IBM to play Jeopardy, outdid itself in the second half of its first official game against champions Ken Jennings and Brad Rutter. Watson finished with $35,734 to Rutter's $10,400 and Jenning's $4,800, despite coming up with a very wrong answer to what appeared to be a fairly straightforward final Jeopardy question on the topic of "US Cities."

The answer was, "Its largest airport was named for a World War II hero; its second largest, for a World War II battle." Both Jennings and Rutter got the correct question— "What is Chicago?"— while Watson put down "What is Toronto???" Dr. Chris Welty, who worked on the algorithms team during Watson's development, said that the phrasing of the question demonstrated again Watson's difficulty with implicit meanings and how quickly it can become tough for the computer to sort out what type of question the answer is looking for.

"If you change the question to 'This US City's largest airport…', Watson gets the right answer," Welty said during a panel at Rensselaer Polytechnic Institute's Experimental Media and Performing Arts Center. Welty pointed out that though categories in Jeopardy seem like they will have a set type of answers, they almost never do, and Watson was taught not to assume they would.

For instance, in the category of state capitals, it might seem like the answer will always be of the form "What is [state capital]?" But an answer could be "The capital of New York is bordered by this river," making the question "What is the Hudson?" Because the phrasing of the answer was not precise enough for Watson to pinpoint the type of the question it was looking for, it struggled to get the correct question. "We do look at the category, but we don't necessarily trust that information as a 'type,'" Welty said. Toronto was also associated more with World War II battlefields than Chicago was in Watson's information store.

Welty said Watson was also confused by the second clause of the answer, "its second largest, for a World War II battle," because it can't understand context. It's obvious to humans based on phrasing that the second part of the answer is also referring to an airport, but Watson is unable to parse implicit information.

Fortunately, Watson only bet $947 on the final question, so it held on to most of what it had earned through the rest of the game and the two Daily Doubles it nailed earlier, questions on which it also bet very specific dollar amounts. While most humans bet in round numbers, the game theorist that worked on Watson's gambling abilities chose not to make the computer so reserved. On its first Daily Double, Watson told Trebek, "I'll wager 6,345 dollars." Trebek shook his head and said, "I won't even ask."

Buzzer beaters

Though Watson seemed to be running the round and beating Jennings and Rutter to the punch with its answers many times, Welty insisted that Watson had no particular advantage in terms of buzzer speed. Players can't buzz in to give their questions until a light turns on after the answer is read, but Welty says that humans have the advantage of timing and rhythm.

"They're not waiting for the light to come on," Welty said; rather, the human players try to time their buzzer presses so that they're coming in as close as possible to the light. Though Watson's reaction times are faster than a human, Welty noted that Watson has to wait for the light. Dr. Adam Lally, another member of Watson's team, noted that "Ken and Brad are really fast. They have to be."

Welty also commented on Watson's seemingly scattershot process of selecting squares on the board, saying it is similar to Jennings' approach of Daily Double-hunting. "Jennings opened up starting to look for the Daily Double right away—that's the style he plays—but Watson was doing the same thing. That's why they were always selecting clues in the bottom three," Welty said.

Though Watson enjoyed a landslide victory in the first game, Welty and Lally said they weren't at all confident Watson would go on to win the second game and therefore the match. When the first game was over, "our anxiety had not lessened by even a slight bit," Welty said. "The next game could very easily go the other way." Watson, Jennings, and Rutter will play a second game of Jeopardy tonight, and the player with the highest winnings over both games will be declared the winner of the Jeopardy IBM Challenge.