Watson brought down our best humans in Jeopardy; now the real work begins.

IBM’s Watson, the supercomputer that gave our best two Jeopardy-playing humans what-for in three nights of play two years ago, is now showing mortals how to do better at another classic human struggle: curing cancer. Watson has spent the last year parsing data on cancer treatments from the Sloan-Kettering Memorial Center and is now being offered as a cloud-based application for determining the best course of action for cancer patients.

While Watson’s turn at Jeopardy was entertaining and a true battle of man versus machine, the computer’s higher purpose was always in medicine. During a panel discussion of Watson held as the computer did battle with Jeopardy champions Ken Jennings and Brad Rutter, Dr. Chris Welty, a member of Watson’s algorithms team, noted that the computer had a future in helping diagnose medical conditions (as well as in tech support).

According to the Associated Press, Watson has improved its performance by 240 percent since its Jeopardy stint. In March 2012, scientists at Sloan-Kettering set Watson about the task of internalizing 600,000 pieces of medical evidence, 1.5 million patient records, 2 million pages of texts from medical journals, and 1,500 lung-cancer cases.

Watson can see connections and trends in the data that humans, or even humans and databases, may not be able to. Now that the data has been compiled, Watson is being offered as an app that can be accessed through a tablet or computer that will compare a cancer patient’s medical records with Watson’s index. The results Watson returns are recommendations on treatment, in descending order of confidence on effectiveness.

The algorithm can actually be run in two ways. Operating strictly on cancer, Watson stacks treatments in the order of likelihood that they’ll succeed. But the results can also be augmented by insurance coverage, in which case Watson considers which treatments will be authorized for payment.

The application is owned by WellPoint and is set to be adopted by medical groups at the Maine center for Cancer Medicine and WestMed in Westchester County, New York, according to the AP. WellPoint will sell the applications to other institutions at a price yet to be determined, and IBM will receive a cut.

Ars Science Video >

Apollo: The Greatest Leap

In honor of the 50th anniversary of the beginning of the Apollo Program, Ars Technica brings you an in depth look at the Apollo missions through the eyes of the participants.

Apollo: The Greatest Leap

Apollo: The Greatest Leap

In honor of the 50th anniversary of the beginning of the Apollo Program, Ars Technica brings you an in depth look at the Apollo missions through the eyes of the participants.

Casey Johnston
Casey Johnston is the former Culture Editor at Ars Technica, and now does the occasional freelance story. She graduated from Columbia University with a degree in Applied Physics. Twitter@caseyjohnston

Medical data is a great fit for this sort of system. Watson is basically a self teaching expert system anyways and medicine has been one of the favorite whipping boys for the expert system crowd since the 80s.

in which case Watson considers which treatments will be authorized for payment.

Somehow, a little bit of my wonder at the miracle of this invention just got diminished.

This.

(During my treatments, I had the opportunity—through sources I shall not here divulge—to view actual costs versus charges to insurance. Phenomenal. Not to mention unnecessary add-on materials, procedures, and on, and on. Yes, I was cured. I'm not sure Watson wouldn't have a conflict of interest.)

in which case Watson considers which treatments will be authorized for payment.

Somehow, a little bit of my wonder at the miracle of this invention just got diminished.

Even within a system of socialised medicine, any available treatments will still be analysed for cost as well as medical effectiveness to determine whether they should be made available.

Still, I'd prefer not to be dealing with insurance companies if my health was on the line.

Who would you prefer pay for your life saving treatments? You know that Medicare and Medicaid don't cover everything a Dr. would possibly want to prescribe right? They've got lists of approved procedures and meds and they tend to be more restrictive than insurance companies.

Unless you're a special person you're just a statistical blip to the government that has no obligation to treat you at all. England's system is pretty brutal if the articles are fairly accurate. I can at least buy "Cadillac" insurance and the insurance company is obligated to pay up under the terms and they can't unilaterally renegotiate after the fact.

Honestly my biggest complaint with the affordable care act is that it does away with high deductible plans. I want an insurance plan that pays unlimited but has a deductible over 2 or 3K. I'm willing to risk that much on my annual health, and the higher deductible keeps the plan affordable so I could buy the upgrades to get the most cutting edge treatments if I had one of the truly ruinous diseases strike me. I'm very OK spending 5K or 10K to not die. It drives me nuts hearing people complain about the cost of healthcare but demanding treatments that were the stuff of Star Trek a decade ago.

And on topic, I think Watson is a pretty cool application. Doing this will save lives and by taking into account reimbursements it'll probably save even more lives. Dr's that spent time going down a list of treatments to find one that is approved will now have the options presented to them. That saves them time, lets them spend more time on more patients.. or filling out EMR charts but that's another issue.

I'm very OK spending 5K or 10K to not die. It drives me nuts hearing people complain about the cost of healthcare but demanding treatments that were the stuff of Star Trek a decade ago.

When you look at these 'wonder' treatments, they're usually pretty far from being miracle cures. You're not spending 5K or 10K not to die, you're spending 100K to live an extra 3 months, often in significant pain, discomfort, and reduced mobility.

The low hanging fruit of healthcare were picked years ago and what we're left with are very expensive (and getting more so) treatments that offer minimal improvements in patient outcomes.

Excuse me, but how does the Jeopardy outcome surprise anyone? Either the computer knows the answers, or it does not know them. If it knows the answers, is it checking itself in order to not kick our asses that badly and seem smart vs. all knowing (pre-programmed).

I mean, once it has been given the answer, it isn't forgetting it is it? That would just be a testament to the guys who preload the system with knowledge right?

Winning at chess is more impressive, but then again these "super-computers" are just calculators to the very core, correct? If this than that, etc. but not an actual intelligence of any kind.

What makes Watson impressive is not that it knows the answers - but that by itself IS a pretty amazing accomplishment considering how it did not know any of the questions in advance (and yes I know that in actuality the questions/answers are actually answers/questions) - but that it could locate the correct information and phrase the information in the form of a question just as fast as a person. It isn't just a big database, but a big database than can perform lookups much, much faster than previously devised and then spit out what it found in a human understandable language.

When you look at these 'wonder' treatments, they're usually pretty far from being miracle cures. You're not spending 5K or 10K not to die, you're spending 100K to live an extra 3 months, often in significant pain, discomfort, and reduced mobility.

[/quote]

I wish! A bed in the ICU alone runs something like $1k a day. Add monitoring, pumps, drips, labs, lines, x-rays and and it's easily over $5k per day. We haven't even touched actual treatment yet!

Also, call me a dickhead, but if this isn't going to save me money on my health insurance, like Geico can with my car insurance, what is the point?

Don't doctors specializing in cancer already know the best route to take?

I also thought it was funny that the article basically said "Watson armed with a database is more effective than people without a database." Duh? Then it goes on to say it "may" be more effective armed with a database than people armed a database, which tells me not a whole lot.

Either way, if this lowers costs then it is great, but I imagine insurance companies will only be tacking on a fee for a "Watson diagnosis" vs. lowering costs. Such is the world we made for ourselves.

Excuse me, but how does the Jeopardy outcome surprise anyone? Either the computer knows the answers, or it does not know them. If it knows the answers, is it checking itself in order to not kick our asses that badly and seem smart vs. all knowing (pre-programmed).

I mean, once it has been given the answer, it isn't forgetting it is it? That would just be a testament to the guys who preload the system with knowledge right?

Winning at chess is more impressive, but then again these "super-computers" are just calculators to the very core, correct? If this than that, etc. but not an actual intelligence of any kind.

What makes Watson impressive is not that it knows the answers - but that by itself IS a pretty amazing accomplishment considering how it did not know any of the questions in advance (and yes I know that in actuality the questions/answers are actually answers/questions) - but that it could locate the correct information and phrase the information in the form of a question just as fast as a person. It isn't just a big database, but a big database than can perform lookups much, much faster than previously devised and then spit out what it found in a human understandable language.

That it can single out the correct answer from all the noise, unlike something like google, and that it can see relationships that even most people can't see, is amazing. A lot of the stuff on jeopardy is pretty damned hard for a computer to understand with all the puns, plays on words, and double-meanings in the questions. Answering jeopardy questions didn't just showcase its ability to regurgitate a list of facts from an encyclopedia, but to understand all that other stuff too.

Excuse me, but how does the Jeopardy outcome surprise anyone? Either the computer knows the answers, or it does not know them. If it knows the answers, is it checking itself in order to not kick our asses that badly and seem smart vs. all knowing (pre-programmed).

I mean, once it has been given the answer, it isn't forgetting it is it? That would just be a testament to the guys who preload the system with knowledge right?

Winning at chess is more impressive, but then again these "super-computers" are just calculators to the very core, correct? If this than that, etc. but not an actual intelligence of any kind.

What makes Watson impressive is not that it knows the answers - but that by itself IS a pretty amazing accomplishment considering how it did not know any of the questions in advance (and yes I know that in actuality the questions/answers are actually answers/questions) - but that it could locate the correct information and phrase the information in the form of a question just as fast as a person. It isn't just a big database, but a big database than can perform lookups much, much faster than previously devised and then spit out what it found in a human understandable language.

I would be more impressed if I couldn't press a button on my phone, speak to it, and have my question answered in similar fashion. Google voice and Siri have already been doing this at a level a human cannot match, for a good amount of time now. It is, after all, all about the algorithms. They are complex to be sure, but nothing new.

I am 100% sure if I played Jeopardy vs. my android phone armed with Google voice search I would be handily beaten every single time. I'm confident my phone could hit the buzzer the second the question is asked and it could just wait for the answer to arrive, and win every damn question.

You don't know how Jeopardy works. For one, you're not asked a question. You're given an answer. You have to figure out what the question was. That means you need to take the answer, parse it, figure out what's important, understand any play on words/linguistic gymnastics that Jeopardy loves doing, and then you can start trying to figure out what the question was by analyzing absurdly large data sets.

Seeing as they aren't questions Google and Siri will never get one.

And Watson doesn't come up and figure 'this is the right solution' (the right question), it generates a massive list of possible solutions (questions), and assigns a probability to each. It then decides if it's confident enough to actually make guess. And mind you, it's having to process a huge amount of data... and it's playing against two of the best humans to ever play Jeopardy. Watch the NOVA special on Watson, they explain quite well how this isn't an easy task. You also get to see various tests they ran, and how horribly Watson was doing. IBM certainly wasn't gimping Watson to keep from showing up the humans. Not at all.

Quote:

I think you would be rather impressed with what your basic voice search is capable of if you try it out. Keep in mind the search should include the category of the answer in addition to the answer for optimal results. Of course then again, what is Google if not itself a supercomputer at heart?

I use Siri all the time. I also, just now, Googled for a list of Jeopardy questions and asked her a few... she just asked if I wanted to search the web for each. Also, Watson played the first game (or two) completely ignoring the category of the question... which was part of the cause of Watson famously blundering this.

What you're really saying is if you have a human pre-process the answer (remove all those challenging parts) and turn it into a simple, straightforward question, Siri/Google would be fairly decent. And they would be. But you just removed a huge chunk of the challenge. And even then, Google just points you at something, and Siri is very restricted in the possible questions it can answer (just like Google Voice).

Doesn't Watson suffer from selection bias, though? Say a new therapy comes out that hasn't seen widespread use -- isn't their a chance that Watson would rank it low in the results from lack of popularity? Conversely, a treatment that it puts at the top of the results is more likely to be chosen by doctors, making it more likely to appear at the top next time.

People should read up on the Watson project before commenting. It isn't just a dumb computer that regurgitates facts. It's not google search. It's not Siri. It is not the equivalent of a search engineer running sql queries. It is something completely different.

In regards to health care I was at first taken aback by the comment that it weighs health insurance coverage and whether it would be covered, but on second take I think it is a VERY good idea.

One of the main problems with healthcare in the US is that neither doctors NOR patients care about the cost. Just 'let insurance handle' it. By combining effectiveness with costs with treatments and presenting it to the people that need to care, doctors and patients, maybe they will start using it to make 'purchasing' decisions. Treatment B costs half as much as treatment A and has the same chance of effectiveness. Win-win.

One thing that I think that gets left out when discussing healthcare is that insurance companies aren't fighting tooth and nail to reduce medical costs. In fact, the more people spend on healthcare the more of that pie they can slice out. We need a system where someone, anyone, cares about the price. And Watson could contribute to that. Bravo.

But where you are wrong, is that I just did a search like I suggested was possible, with the words "my life as a dog in the 1890's francis barraud painted his dog nipper while listening to one of these" and lo and behold google presented the correct "question" to the answer, "what is the cylinder phonograph"

So if Google search can find the correct answer, albeit without correct Jeopardy presentation, how far is it from there to get Google to spit the answer I wanted out, in the format I wanted it in? My guess is the hardest part of the equation is already done, since the information was present in the top hit.

Because Google simply checks against a database of Jeopardy questions (Google has no problem doing a bit of munging), and sees it in there, and returns the connected data. Google is not figuring out the solution. What Google is doing is still impressive, but it's completely different than actually playing Jeopardy. What Watson is doing is far, far more difficult.

Try watching it live and asking then... Unless Google gets very lucky, it won't be getting it.

But where you are wrong, is that I just did a search like I suggested was possible, with the words "my life as a dog in the 1890's francis barraud painted his dog nipper while listening to one of these" and lo and behold google presented the correct "question" to the answer, "what is the cylinder phonograph"

So if Google search can find the correct answer, albeit without correct Jeopardy presentation, how far is it from there to get Google to spit the answer I wanted out, in the format I wanted it in? My guess is the hardest part of the equation is already done, since the information was present in the top hit.

Because Google simply checks against a database of Jeopardy questions (Google has no problem doing a bit of munging), and sees it in there, and returns the connected data. Google is not figuring out the solution. What Google is doing is still impressive, but it's completely different than actually playing Jeopardy. What Watson is doing is far, far more difficult.

Try watching it live and asking then... Unless Google gets very lucky, it won't be getting it.

Uh actually, no. Why don't you do the search yourself and come back and fix what you typed. Google did not do what you said, I am pretty sure of that. Google linked me to a wikipedia article, and in the little bit of text beneath the link for the page was the answer. I did not have to even click the link and search. The answer was right there for my eyes to see.

And? Why do you think that has anything to do with how Google came to the conclusion to show that (Google has never operated that way!)? How about doing what I suggested. Or even coming up with a similar riddle that Google can't already know just by checking pre-existing databases/doing its using Page Rank of links that contain significant amounts of the quoted text pointing to Wikipedia.

If this was as easy as you suggested, Watson would have been a breeze for IBM to build, and would have been done many years ago.

Is it already time to start establishing Asimov's Laws of Robotics? The first law is: A robot may not injure a human being or, through inaction, allow a human being to come to harm.

By not suggesting the best course of action to medically treat a patient, Watson is actually violating the first law by allowing a human being to come to harm.

Not sure where you're going with this... seems like if you accept that you don't know all the facts then you have to accept wrong decisions ("Hindsight is 20:20") so long as justifiable given the facts at the time. Waiting for the umpteenth lab result has a cost all its own. Or, are you suggesting that the Robotic Laws are due to be implemented as goal states for these robots?

But where you are wrong, is that I just did a search like I suggested was possible, with the words "my life as a dog in the 1890's francis barraud painted his dog nipper while listening to one of these" and lo and behold google presented the correct "question" to the answer, "what is the cylinder phonograph"

So if Google search can find the correct answer, albeit without correct Jeopardy presentation, how far is it from there to get Google to spit the answer I wanted out, in the format I wanted it in? My guess is the hardest part of the equation is already done, since the information was present in the top hit.

Because Google simply checks against a database of Jeopardy questions (Google has no problem doing a bit of munging), and sees it in there, and returns the connected data. Google is not figuring out the solution. What Google is doing is still impressive, but it's completely different than actually playing Jeopardy. What Watson is doing is far, far more difficult.

Try watching it live and asking then... Unless Google gets very lucky, it won't be getting it.

Uh actually, no. Why don't you do the search yourself and come back and fix what you typed. Google did not do what you said, I am pretty sure of that. Google linked me to a wikipedia article, and in the little bit of text beneath the link for the page was the answer. I did not have to even click the link and search. The answer was right there for my eyes to see.

And? Why do you think that has anything to do with how Google came to the conclusion to show that (Google has never operated that way!)? How about doing what I suggested. Or even coming up with a similar riddle that Google can't already know just by checking pre-existing databases/doing its using Page Rank of links that contain significant amounts of the quoted text pointing to Wikipedia.

If this was as easy as you suggested, Watson would have been a breeze for IBM to build, and would have been done many years ago.

Yeah man, THAT IS MY POINT. Google ALREADY DID THIS, YEARS AGO. It answered a Jeopardy "answer" by me typing in the category and question. It did not give me ANYTHING JEOPARDY RELATED. It linked to a wiki article and had the answer underneath the link that took me to the full article.

What I am saying is Google took me 99% of the way to the answer. I had about 20 words in front of me. I was looking for a fucking noun, and I found it with ease. Watson does the same fucking thing, it just goes 1% further. Hence unimpressed.

And you know why it did that? Because people linked to that article with that text! You know why people linked to that article with that text? Because it was a question on Jeopardy! Google's search algorithm is based massively on people linking to sources (and Wikipedia, being massively linked to, tends to be very high). Google's search algorithm is very impressive, but it's very dependent on humans making connections for it. It's very, very, very impressive (and massive) data mining, but it's a different problem. Also, Watson wasn't connected to the Internet, and only had a relatively limited set of data to pull from, so it had to be more creative than brute forcing.

And also watch PBS's NOVA special (just click the "WATCH THE PROGRAM" button). They do cover why this was such a challenge to do, and how bad prior attempts were. You even see them doing trial runs at IBM against random team members, and how Watson was hilariously bad. You really should watch it to get a better understanding of Watson.