Though the field of oncology has some AI-driven tools, overall, physicians report the reality isn’t living up to the hype

Artificial intelligence (AI) has been heavily touted as the next big thing in healthcare for nearly a decade. Much ink has been devoted to the belief that AI would revolutionize how doctors treat patients. That it would bring about a new age of point-of-care clinical decision support tools and clinical laboratory diagnostic tests. And it would enable remote telemedicine to render distance between provider and patient inconsequential.

But nearly 10 years after IBM’s Watson defeated two human contestants on the game show Jeopardy, some experts believe AI has under-delivered on the promise of a brave new world in medicine, noted IEEE Spectrum, a website and magazine dedicated to applied sciences and engineering.

In the years since Watson’s victory on Jeopardy, IBM (NYSE:IBM) has announced
almost 50 partnerships, collaborations, and projects intended to develop
AI-enabled tools for medical purposes. Most of these projects did not bear
fruit.

However, IBM’s most publicized medical partnerships revolved
around the field of oncology and the expectation that Watson could analyze data
and patients’ records and help oncologists devise personalized and effective
cancer treatment plans. Success in helping physicians more accurately diagnosis
different types of cancer would require anatomic pathologists to understand
this new role for Watson and how the pathology profession should respond to it,
strategically and tactically.

But Watson and other AI systems often struggled to
understand the finer points of medical text. “The information that physicians
extract from an article, that they use to change their care, may not be the
major point of the study,” Mark
Kris, MD, Medical Oncologist at Memorial
Sloan Kettering Cancer Center, told IEEE Spectrum. “Watson’s
thinking is based on statistics, so all it can do is gather statistics about
main outcomes. But doctors don’t work that way.”

“We’re doing incredibly better with NLP than we were five
years ago, yet we’re still incredibly worse than humans,” Yoshua Bengio, PhD,
Professor of Computer Science at the University
of Montreal, told IEEE Spectrum.

The researchers hoped that Watson would be able to examine
variables in patient records and keep current on new information by scanning
and interpreting articles about new discoveries and clinical trials. But Watson
was unable to interpret the data as humans can.

IEEE Spectrum reported that “The realization that
Watson couldn’t independently extract insights from breaking news in the
medical literature was just the first strike. Researchers also found that it
couldn’t mine information from patients’ electronic health records as they’d
expected.”

Researchers Lack Confidence in Watson’s Results

In 2018, the team at MD Anderson published a paper in The
Oncologist outlining their experiences with Watson and cancer
care. They found that their Watson-powered tool, called Oncology
Expert Advisor, had “variable success in extracting information from
text documents in medical records. It had accuracy scores ranging from 90% to
96% when dealing with clear concepts like diagnosis, but scores of only 63% to
65% for time-dependent information like therapy timelines.”

A team of researchers at the University of Nebraska Medical Center (UNMC) have experimented with Watson for genomic analytics and breast cancer patients. After treating the patients, scientists identify mutations using their own tools, then enter that data into Watson, which can quickly pick out some of the mutations that have drug treatments available.

“But the unknown thing here is how good are the results,” Babu Guda, PhD, Professor and Chief Bioinformatics and Research Computing Officer at UNMC, told Gizmodo. “There is no way to validate what we’re getting from IBM is accurate unless we test the real patients in an experiment.”

Guda added that IBM needs to publish the results of studies
and tests performed on thousands of patients if they want scientists to have
confidence in Watson tools.

The inability of Watson to produce results for medical uses
may be exacerbated by the fact that the cognitive computing technologies that
were cutting edge back in 2011 aren’t as advanced today.

IEEE Spectrum noted that professionals in both
computer science and medicine believe that AI has massive potential for
improving and enhancing the field of medicine. To date, however, most of AI’s
successes have occurred in controlled experiments with only a few AI-based
medical tools being approved by regulators. IBM’s Watson has only had a few
successful ventures and more research and testing is needed for Watson to prove
its value to medical professionals.

“As a tool, Watson has extraordinary potential,” Kris told IEEE
Spectrum. “I do hope that the people who have the brainpower and computer
power stick with it. It’s a long haul, but it’s worth it.”

IBM Watson Health is “using our presence at HIMSS19 this
year to formally unveil the work we’ve been doing over the past year to
integrate AI technology and smart, user-friendly analytics into the provider
workflow, with a particular focus on real-world solutions for providers to start
tackling these types of challenges head-on,” stated Rhee. “We will tackle these
challenges by focusing our offerings in three core areas. First, is management
decision support. These are the back-office capabilities that improve
operational decisions.”

Clinical laboratory leaders and anatomic pathologists may or
may not agree about how Watson is able to support clinical care initiatives.
But it’s important to note that, though AI’s progress toward its predicted
potential has been slow, it continues nonetheless and is worth watching.