Slashdot videos: Now with more Slashdot!

View

Discuss

Share

We've improved Slashdot's video section; now you can view our video interviews, product close-ups and site visits with all the usual Slashdot options to comment, share, etc. No more walled garden! It's a work in progress -- we hope you'll check it out (Learn more about the recent updates).

wjousts writes "Technology Review is reporting on IBM's plans to take on Trebek at his own game. The 'Watson' computer system uses natural-language processing techniques to break down questions into their structural components and then search its database for relevant answers. A televised matchup with Trebek is planned for next year. 'David Ferrucci, the IBM computer scientist leading the effort, explains that the system breaks a question into pieces, searches its own databases for "related knowledge," and then finally makes connections to assemble a result. Watson is not designed to search the Web, and IBM's end goal is a system that it can sell to its corporate customers who need to make large quantities of information more accessible.'"

I wonder how they plan to do with categories that have implications for all the answers. I've seen categories where words must be so many letters in length or perhaps start with certain things and Alex will interject while reading the category such as "'Cats'--and that means all the words in this category start with 'Cat'." Now, with that in mind, a clue could come in as "They are the popular makers of earth moving equipment." Might prompt Watson to find the most popular makers of earth moving equipment--Who is John Deere? The category of 'Cats' would do nothing for Watson without the aid of Alex's interjection... thus failing at finding "Who is Caterpillar?" (bonus points if you also thought of "Who is Bobcat?" but that answer doesn't start with Cat).

As a fairly avid though novice crossword puzzler, my mind explodes with questions. Could Watson discern a four letter word for "Pleasant French city" (Nice)? Or what about a four letter word for "Beefy Laker" (Kobe)?

Lastly, will Watson have something inane and boring to talk about during the break?

Alex Trebek: Now, Watson, it says here that you are named after Thomas J. Watson who forbade his employees to drink and even frowned upon it while off the job? Watson: That is correct. It is against IBM regulation 4-245 Section 8 to consume alcohol on the premises of any facility. Alex Trebek: Fascinating, I'm sure you've never broken that strict regulation, ha ha. Watson: Good sir, I am a computer, drinking is not within my capacity. Alex Trebek: Um, right. So could you tell us something interesting about yourself? Watson: *pauses to search records* During the fabrication of my circuitry, several engineers went months without sleep. Leading one to go insane and killed his wife and kid before taking his own life in a double homicide/suicide case. Alex Trebek: How unfortunate. Well, I wish you the best of luck today in Jeopardy. Watson: Thank you, my snide game show master.

How tastelessly incorrect. Extra-terrestrials don't come back to life. Watson would cross reference The Bible with many recent movies and come up with the correct question we were looking for: "What was a zombie?"

I know you were going for funny, but in reality Jeopardy never uses one(or two)-word answers. There are too many possible questions for a one-word answer.

Example:A: Caesar

Q: Who crossed the Rubicon and said 'the die is cast'?Q: Who said 'Et tu Brute?' when he was assassinated?Q: Which Roman emperor adopted his great nephew Octavian who later became Augustus Caesar?Q: What salad is made with Romaine lettuce, anchovies, garlic and lemon?etc. etc.

I know you were going for funny, but in reality Jeopardy never uses one(or two)-word answers.

They often do, but they're typically framed within a narrow, specific category. For example, the category might me "National Drinks" or some such thing. Typical Q/As might be Japan (Sake), Russia (Vodka), and so forth. Jeapoardy! also has a few other categories (anagrams, for example) that frequently use one or two words.

Ah, you're right. I forgot about those cases. But they are exceptions, not the norm. I haven't watched Jeopardy! in a long time.
I always thought I would have done well on Jeopardy! but not as well as that guy from Utah.

Presumably they will either have to take into account the clues that come from the category itself (as in your example) or rig the system by avoiding "trick" categories. It's not an easy problem and it'll be very interesting to see what IBM come up with.

An example from last night, they had a category "Knockouts" in both the first and second round. In the first round, all the answers were hot women (i.e. knockouts!), in the second round all the answers were about boxing. How will Watson deal with this? I don

Presumably they will either have to take into account the clues that come from the category itself (as in your example) or rig the system by avoiding "trick" categories. It's not an easy problem and it'll be very interesting to see what IBM come up with.

An example from last night, they had a category "Knockouts" in both the first and second round. In the first round, all the answers were hot women (i.e. knockouts!), in the second round all the answers were about boxing. How will Watson deal with this? I don't know.

Yes, there are categories which require the contestant to have an active imagination and it's these categories I wish the article had addressed instead of a vanilla one. And I believe it's these categories that makes Jeopardy fresh and new after decades.

In retrospect, I should have broke out the conversation into a different post so that this wasn't modded +5 Funny. I'm seriously interested in how IBM plans to address things that require the natural speech recognition of Alex Trebek. Does it take into account other answers in the same category to "catch on" like some contestants obviously do?

Then there's the folks running Jeopardy who could pick some categories that would wreck Watson and give the humans the creative advantage. I hope they exploit this creative ability humans have and write an entire category in... oh, say Pig Latin!

In reality, they stand to have much more to gain if the machine comes close to winning... as they could make this into an annual competition drawing fans and viewers much like the quest to beat the world chess grand masters.

It's possible that the questions for that particular show will be specifically chosen to be more explicit and less ambiguous (avoiding the show's characteristic punny wordplay) to put the machine on a more level playing field, keeping its score closer to those of the contestants', which will make the episode more exciting to watch.

Without speculating on the specifics of tweaking the AI, my guess is that IBM has tried to think through these things. Having put together a few AI bots myself (purely recreationally - you know, just for kicks), I know that I let them play in the real world for quite a while to work out the kinks before unveiling them to nerdy friends and family to show them off and demonstrate just how much time and sleep I'd wasted. My poker-bot played thousands of games in free online rooms before I told anyone that I was even working on him.

IBM has probably been feeding Watson DVR'd episodes for a while now so that they could identify (if not fix) the kind of gotchas that you're thinking about.

I'm on a mailing list with some participants in the Loebner Prize Contest, a version of the Turing Test. ("Robitron," if you want to see the list.) We were talking about this Jeopardy AI project recently.

The most-discussed type of AI on that list is a "chatterbot" like ALICE or the classic ELIZA -- one that basically looks for key words and finds pre-written responses to them. That approach could probably tackle a variety of Jeopardy-type questions if someone took the trouble to feed the AI a bunch of sui

So you can always figure out the last digit of the cube root of a number VERY easily (no, you don't need to memorize that list).

Then you use the size of the number to get a range, and then estimate. If you're feeling ballsy, you can go for it. Spend the first few seconds (before people buzz in) and get your range down. Then buzz in and spend a couple seconds estimating, then answer (just say "what is..." right when you buzz in). If someone else buzzes in first, more time for you to think.

It may not be... would it hit the buzzer and hope it had time to compute the answer like many people on those shows, or would it wait until it had time to compute, and then ring in only if it has the answer?

Seeing as it can read the question instantly, instead of having to wait for Trebeck to read it out (or read it as text at human speed), I think it's got a serious advantage.

I don't think the computational I/O speed is really going to be a limiting factor in this. The crux will be the dataset it has and whatever kind of strategy (being more or less gutsy with the buzzer, bidding more or less on the daily doubles / final round, etc. based on confidence of being right) it uses.

Well shit, I thought the 1st round was going to have knockouts (boxing) that occurred during the first round, and in the second round, the category would be about knockouts occurring in the second round.

Alex will interject while reading the category such as "'Cats'--and that means all the words in this category start with 'Cat'."

Then the bot would read the closed caption that the category is "CATS MEANING ALL RESPONSES HAVE A WORD THAT STARTS WITH CAT" and include that in its reasoning. Then the clue "They are the popular makers of earth moving equipment" becomes something like "They are the popular makers of earth moving equipment, starting with 'CAT'".

Its an automated crossworld puzzle solver. How it works (and my advisor led the project, though I don't work on anything remotely similar) is that it has a large number of solver modules that are each good at a certain kind of clue. One might be really good at looking up famous people based on keywords. Another might be good at... I dunno some other type of crossword clue.

A lot of Jeopardy questions are wordplay-dependent, something AI doesn't have the hang of yet (unless IBM has been toiling in secret on something truly amazing). Categories like "Rhyme Time" and questions like "Qhat does a Pharoah need when he has a cold?" (Answer: an Egyptian Prescription) are beyond the ken of a data search.

Many Jeopardy "answers" have the key to the answer within the question, though in some cases it may be enough to throw the program off. IE in a category like "Musicals" an answer like "Unlike his other hits, this musical wasn't 'the cat's meow' on Broadway." Raw data crunching will pair musicals, Broadway and "cats" but won't know where to go with "unlike." Only an aficionado will know that Andrew Lloyd Weber's "Starlight Express" tanked on Broadway.

So the writers, given any knowledge of the limitations of AI, can set a challenge which will be nearly impossible for current AI to meet. John Henry will live another day.

Right. That's why it's interesting. It needs to interpret the questions, construct a query, process and rank the results, as well as store and index all the information it needs for the game (no live connection to the internet).

If the questions aren't from the same group as usual, it won't be worth much. I would hope they wouldn't be specifically designed to either help or stump the computer.

It says its not designed to search the web. This doesn't imply that its database must be stored locally. It can have a live internet connection, but only be talking to the database. I'd imagine the database is quite large.

IBM is laying off American citizens, but hiring in Asia, and yet are spending all this money on gimmicks. This is the kind of thing that gives big companies bad names. Hopefully, as a consolation prize, the laid-off Americans can watch their former company go down in smoke on the game show, hoping it starts smoking and sparking like a cheesy Trek android meltdown.

It's not a gimmick. It is very important research on AI and natural-language processing. Jeopardy! just happens to give them a very difficult problem to tackle. If they can develop a system that can handle Jeopardy!, it'll be a huge break through for other fields.

Well, since a big portion of IBM is consulting (50%?), it probably makes sense for them to hire consultants who are citizens of the countries they will be working in... How many Americans are fluent in a second language? How many are willing to relocate for 5+ years to Asia? I'd guess that hiring local gives IBM a GOOD name in those markets.

Plus, the people they are hiring are human beings too. Why is hiring an American a more noble thing than hiring an Indian or Chinese person? They need to feed their

Because Thomas J. Watson was the man who turned IBM into a global empire, and Thomas J. Watson Jr. brought it into computers. They successively held the top position at IBM for 57 years. So it's a very important name at IBM, and the connection with Sherlock Holmes is serendipitous.

Third, no, phrasing answers in the form of a question is easy, have you evern watched Jeopardy before? That's the easiest part of the whole thing, "What is/are X" and "Who is/are X" are all you have to do the answers to phrase them as a question. No, the interesting thing will be if IBM's machine can parse the question in the correct context and come up with the correct answer fa

The system is not designed to access the web?Horse shit.That huge fucking pile of data is getting in there from the web. It won't be accessing the web during the game, but it's still a fucking cache of random shit (mostly geography and world history) from the internet.

So? Your point? It's not accessing the web live, is it? No? So it's not getting it from the web.

The web isn't some magical information generating device. That information did not originate on the web, somebody put it there either from their own mind or from an offline location. By your own reasoning accessing the web isn't even "getting it from the web", as the web is just a huge cache of information from people's homes, schools, and even just plain their own minds.

Ok, first, you totally missed my point. My point was that by extrapolating your objections to having a database of information obtained via the web, you actually get the data indirectly from the source, in some form. I don't see your objections to the human players who have been crawling for data for their own databases for the last 40+ years. Some of them have been crawling the web for data for the last 20 years!

It's the same damn thing, data is data, half of Jeopardy is knowing shit. That's an area co

All they need to do is use their super computers to generate some digital footage of Alex Trebek engaged in beastiality (tappin' Rosie O'Donnell) and then tell him that if they win, the footage disappears forever.

It'd be far cheaper than what they are planning to do...and they can always leak the footage to youtube after they walk off with the winnings...

It's a shame that 'Watson' works by breaking down questions into their structural components and then searching its database for relevant answers. After all, on Jeopardy all the answers are given freely.

So the new profit model for IBM is to make computers that can win game shows? Maybe someone needs to tell IBM that more money can be won in poker tournaments than Jeopardy. I can hear the board meeting conference, profit margins across our divisions were flat year over year, however our new game show competition division were good enough to bank roll our expensive luxury stay here on the island.

A good poker AI might actually be more interesting than a Jeopardy AI, even if the game it played was online so as to eliminate the factor of reading body language. For Jeopardy questions that boil down to "What is the capital of X" or "In what year did X happen?", a winning AI could basically just be Google running on an internal database. In contrast, winning at poker would involve social reasoning about questions like "When this guy suddenly raises his bet, is he often bluffing, and how likely is it that

In explaining the chain of reasoning, they weirdly left out that the name 'Pagliacci' is explicit in the lyrics, and proposed that Watson would deduce 'tears' as a form of feelings!? (Maybe they don't want to include a database of song lyrics?)

They claim they won't use Web data, but there's no way they can compile enough databases on their own to handle Jeopardy's general knowledge. Awards, lyrics, plots, characters... the list goes on and on and on.

Here's a way to build a simple Jeopardy player that would kick a human's ass and doesn't require 4 years of programming:

- Type entire "answer" as given on the board directly into google without quotes.
- Search the returned page for the most common word (ignoring 2 letter ones) in the titles of the pages.
- If the most common word appears more than 3 times, print "What is X?" where X is the common word.
- If no one term appears that often, don't ring in.

Google results are far from useful without context information, especially with Jeopardy where the context is usually hidden by a pun or a play on words.

You just don't notice it because most people are able to process this contextual information on the fly, but it is a huge challenge for a machine to do it. There are litterally thousands of little bits of information that we collect as the answer is being given, including the context of the category, i.e. whether

IBM has a history of inadvertently making terrible PR for themselves with these man-vs-machine stunts. Everyone here should remember Kasparov vs. Deep Blue. Expect IBM to win Jeopardy, and expect there to be a hailstorm of "IBM cheats" controversy after the game.

The machine will probably be able to come up with an answer (maybe not the right one) much faster than all of the human opponents. But, what confidence will it have in that answer, and will it realize that a wrong answer will cost it?

Obviously if the machine just answers immediately (and no 'confidence' factor is involved) then it could provide wrong answers very quickly, and thus just lose money on every question as it "presses the button" to answer the question before the oppo