Share this story

In 2015, The BMJ (formerly British Medical Journal) did a thorough audit of online symptom checkers. It found that, on average, the sites listed the correct diagnosis first only about a third of the time. Carl Shen, an ophthalmology resident at McMaster University, has led a team of researchers in a small-scale follow-up looking specifically at eye health and got equally concerning results: the correct diagnoses popped up first only a quarter of the time.

The results, presented at the Annual Meeting of the American Academy of Ophthalmology this week, are early and provisional, but Dr. Shen and his team are planning a larger follow-up study. In the meantime, WebMD has done an update of its algorithm.

Vignettes of unpleasantries

To assess WebMD’s accuracy, Shen and his colleagues compiled 42 eye-health “clinical vignettes” based on the medical literature. A decidedly unpleasant vignette of someone suffering from acute angle-closure glaucoma, for instance, describes a “44-year-old woman present[ing] to ER... with severe pain around her right eye of four-hour duration... She is also nauseated and has thrown up once... Intraocular pressure is extremely elevated.”

These vignettes were then boiled down the barebones symptoms so that they could be easily entered into the symptom checker twice: once by a medical professional, and once by a non-professional. That helped to account both for how non-specialists would interact with the tool and for how different people might describe the same condition in different ways. Because the diagnoses of the clinical cases were already known, the outcome of the symptom checker could be compared to the known, correct diagnosis.

Out of the 42 cases, the correct diagnosis appeared somewhere in the list 18 times—just less than half. The correct result appeared at the top of the list in only 11 cases. And because the main diagnosis was incorrect most of the time, the recommended course of action was frequently off as well. Out of the 18 vignettes that were based on a clinical emergency, the symptom checker correctly recommended the necessary urgent care in just seven of them.

These results are a bit worse than the findings of the 2015 BMJ investigation, which reported the correct diagnosis appearing in the top 20 symptom-checker options 58 percent of the time and appropriate recommendations for emergency care given in 80 percent of the medical emergency cases. But Dr. Shen’s ophthalmology study is also much smaller than the BMJ’s, which looked at 23 different symptom checkers and compiled results from many different people entering the same symptoms. So the ophthalmology study might be less reliable. Shen and his team are also much earlier in the research process than the BMJ investigation, and they are considering crowdsourcing as a way to capture how different people would describe the same condition differently.

Software updates

It’s also important to note that in the time since both of these studies were conducted, WebMD has updated its algorithm, partly as a result of the BMJ research. Where the old symptom checker required users to start by clicking on a body part, the new version just needs a description of the symptoms. “After testing the new version of the WebMD Symptom Checker, our analysis of 77 varied cases showed an accuracy rate of 70 percent for the condition appearing in the top 3,” says WebMD Medical Director Michael Smith.

Independent assessment of that success rate and an exploration of whether it looks the same across different areas of health could follow in future research. As it stands, the picture might be a bit sunnier than Shen and his colleagues suggest, but it’s clear that online symptom checkers shouldn’t be used in isolation to make medical decisions. They’re “intended for informational purposes only,” says Dr. Smith, emphasizing that physicians should “engage their patients in conversation to help them understand how best to use the information.”

Helping doctors to have that conversation from an informed starting point is precisely why Shen was interested in the question. With more patients arriving at the doctor’s office having researched their symptoms themselves, he and his colleagues write in their conference presentation, “it is important for ophthalmologists to be aware of the capabilities and limitations of available health information tools.”

Over and above the fact that you really should not diagnosis yourself using internet sites WebMD is a horrible site to input any kind of medical information unless you are OK with sharing it with the world.

The number of trackers on the site is horrendously high and more than a few of the companies that the site shares info with are on the sketchy side.

Just tested it myself. Woke up with double vision in one eye last week. WebMD suggested low blood sugar, myasthenia gravis, a broken eye socket, a brain aneurysm, retinal detachment, and even multiple sclerosis. Everything was a 'fair' match, except myasthenia gravis, which it thought was a 'moderate' match.

I was paranoid when I discovered what was going on because losing my already-poor eyesight is a phobia, and I was dreading there being something wrong with my retina or lens. Good thing I didn't think to visit WebMD that day.

I visited my ophthalmologist the same day and he found a mild abrasion right across the middle of my cornea. Eyes are otherwise completely healthy. Probably scratched something across it rubbing my eye half-asleep or along a crease in my pillowcase or something. It was healed by the next day. That sort of simple condition wasn't even considered in WebMD, and there were no options for narrowing things down, like "Was the symptom sudden" or "Are you having trouble blinking or talking" to rule things out.

As an eye doctor I could have told you this was true without any research. I can't tell you how many patients I've seen that thought they had a retinal detachment from what they read on the internet. Instead they almost always have much more common and benign issues that have similar symptoms. Docs hate WebMD and Google for this reason. Causes people to panic and even worse self diagnose and treat.

It can also happen with the flip side of the coin too. People will ignore that new floater in their vision and Google told them it was OK. Then they walk in my office a week later with 3/4 of their retina detached.

My favorite shitty WebMD diagnosis was when my wife was plugging in her symptoms (mostly shakiness and confusion) and it suggested that she might have Kuru. I casually asked her if she'd been nibbling on brains lately, but she assured me she had not.

My favorite shitty WebMD diagnosis was when my wife was plugging in her symptoms (mostly shakiness and confusion) and it suggested that she might have Kuru. I casually asked her if she'd been nibbling on brains lately, but she assured me she had not.

I'm happy this research is being done, but simply because a diagnosis is not listed, does not preclude other, more common root causes.

Yes, it should get the 'further evaluation' and 'seek emergency care' responses correct, but by their nature, issues reported in medical journals are more likely to be novel (rare).

Uh, no.

I mean, FUCK no.

When determining a diagnosis, the first thing is to put EVERYTHING on the table. This is SOP for SOAP (a method, probably archaic and updated since, for creating a diagnostic treatment plan starting with Subjective symptoms, Objective observations, Assessment (diagnosis) and Plan (course of treatment)).

As the examination continues, the subjective information helps narrow the focus and the observations (like temperature, blood pressure, lab results, X-rays, etc.) further narrow things down. Once that's narrowed down, anything eliminated is supposed to be SURE it's not involved. If there's any question about it, then it's still on the table.

At the end of the SO part, the A part can be a PITA, because a lot of times, you've not ruled out what else it MIGHT be, but you treat based on what you think it PROBABLY is.

What appears to have happened is that the earlier algorithm was tossing out those "Might be's" based ENTIRELY on the Subjective symptoms. I'd bet it's based on probabilities. After all, how many people would list a tooth decay issue when complaining of chest pain, or a fever with cough? (Yes, you can get infections in the chest, both heart and lungs from tooth decay). Chest pain comes in a variety of issues. One might treat the cough and associated pain from that symptomatically and call it the flu without actually listing the cause.

The probabilities say it's the flu, because that's the most common problem (pneumonia secondary to the flu). The likelihood of a tooth infection is very, very low and was probably below the threshold of likelihood to be listed.

That's bad juju for a medicine man, and worse for the patient.

Diagnosis is a process of elimination and an elevation of probabilities. That which it CAN NOT be is eliminated. That which it MIGHT BE can't be eliminated. So my guess about how they tweaked the algorithm is that they started listing the things that couldn't be eliminated, even if it had a very low probability.

It'll be interesting to see what changes the algorithm tweaks make in the ability to diagnose medical issues based on symptomology alone.

I'm happy this research is being done, but simply because a diagnosis is not listed, does not preclude other, more common root causes.

Yes, it should get the 'further evaluation' and 'seek emergency care' responses correct, but by their nature, issues reported in medical journals are more likely to be novel (rare).

Uh, no.

I mean, FUCK no.

When determining a diagnosis, the first thing is to put EVERYTHING on the table. This is SOP for SOAP (a method, probably archaic and updated since, for creating a diagnostic treatment plan starting with Subjective symptoms, Objective observations, Assessment (diagnosis) and Plan (course of treatment)).

[ ... ]

Diagnosis is a process of elimination and an elevation of probabilities. That which it CAN NOT be is eliminated. That which it MIGHT BE can't be eliminated. So my guess about how they tweaked the algorithm is that they started listing the things that couldn't be eliminated, even if it had a very low probability.

It'll be interesting to see what changes the algorithm tweaks make in the ability to diagnose medical issues based on symptomology alone.

I don't understand the strong language at the start, because it sounds like we partly agree: the algorithm should give better suggestions for 'further evaluation' and 'seek emergency care'. I don't see you disagreeing with that.

My complaint is that starting with issues from medical journals is purposefully sampling for rare conditions, the wrong side of the 80/20 rule. Yes, web diagnostics should return the rare possibilities too, but why should they be expected to be in the top four, as these researchers did? Even though I agree with you about how diagnostics *should* operate, I wouldn't expect most of my doctors (and I suspect they're above average) to cry "zebra" when seeing horses every day.

And yes, SOAP is still taught (as recently in a course I did last week, even). But patients lie, don't know the right questions to ask, include extraneous information. Sure, as you say, the "first thing" is to put everything on the table ... but we as humans don't always know what's relevant and need to discover it, and even when we know it doesn't mean we're able to communicate it effectively.

EDIT: Please indulge a personal story.

My son (age 4) had a persistent, productive sounding cough. We waited a few days, then mentioned it to the doctor while visiting for something else (regular checkup for a sibling, maybe). The answer was the same we'd have given, "Wait and see."

Four weeks later, we return to the doctor. The cough remains. No temperature, no other objective symptoms. S.A.M.P.L.E. history raises no flags. (no allergies or meds, intake and output is normal) Perhaps the only subjective symptom is lower energy. ... Nothing, right? Probably a cold?

Then the doctor uses an oximeter. 85% saturation. That's an immediate ER visit, with a rush to the front of the line. After x-rays show fluid in the lungs, it's an hour-long ambulance trip (a few minutes more and they would have called Life Flight) to the children's hospital. After another scan (MRI or CT, I forget), they diagnose acute pneumonia caused by stage IV neuroblastoma. Terminal cancer of the nervous system causing bleeding in the lungs.

If doctors use this vignette (and our oncologist later asked for permission to publish his case, but I don't know if he ever did), by the protocol described in The Fine Article, we'd expect to see "Cancer" in the web results for "cough". Not a common—nor particularly useful—result. More helpful, I think many might agree—is advice for when to seek medical attention for something more serious, and when it's probably just a cold.

My point at the end of all of this is that although the study is useful and interesting, it has some methodological implications which should not be overlooked in making recommendations for change.

Well you could follow the advice of the poster from the antibiotic article from a couple of days ago. Just wait a week...... (SARCASM ALERT !!!!!!!! DO NOT DELAY EYE TREATMENT OR THERE IS A PRETTY GOOD CHANCE YOU COULD BE BLIND)

I REPEAT DO NOT DELAY EYE TREATMENT OR THERE IS A PRETTY GOOD CHANCE YOU COULD BE BLIND Many other types of issues can be delayed but do not wait for eye issues. If in doubt have it checked out.

I'm happy this research is being done, but simply because a diagnosis is not listed, does not preclude other, more common root causes.

Yes, it should get the 'further evaluation' and 'seek emergency care' responses correct, but by their nature, issues reported in medical journals are more likely to be novel (rare).

Uh, no.

I mean, FUCK no.

When determining a diagnosis, the first thing is to put EVERYTHING on the table. This is SOP for SOAP (a method, probably archaic and updated since, for creating a diagnostic treatment plan starting with Subjective symptoms, Objective observations, Assessment (diagnosis) and Plan (course of treatment)).

[ ... ]

Diagnosis is a process of elimination and an elevation of probabilities. That which it CAN NOT be is eliminated. That which it MIGHT BE can't be eliminated. So my guess about how they tweaked the algorithm is that they started listing the things that couldn't be eliminated, even if it had a very low probability.

It'll be interesting to see what changes the algorithm tweaks make in the ability to diagnose medical issues based on symptomology alone.

I don't understand the strong language at the start, because it sounds like we partly agree: the algorithm should give better suggestions for 'further evaluation' and 'seek emergency care'. I don't see you disagreeing with that.

My complaint is that starting with issues from medical journals is purposefully sampling for rare conditions, the wrong side of the 80/20 rule. Yes, web diagnostics should return the rare possibilities too, but why should they be expected to be in the top four, as these researchers did? Even though I agree with you about how diagnostics *should* operate, I wouldn't expect most of my doctors (and I suspect they're above average) to cry "zebra" when seeing horses every day.

And yes, SOAP is still taught (as recently in a course I did last week, even). But patients lie, don't know the right questions to ask, include extraneous information. Sure, as you say, the "first thing" is to put everything on the table ... but we as humans don't always know what's relevant and need to discover it, and even when we know it doesn't mean we're able to communicate it effectively.

EDIT: Please indulge a personal story.

My son (age 4) had a persistent, productive sounding cough. We waited a few days, then mentioned it to the doctor while visiting for something else (regular checkup for a sibling, maybe). The answer was the same we'd have given, "Wait and see."

Four weeks later, we return to the doctor. The cough remains. No temperature, no other objective symptoms. S.A.M.P.L.E. history raises no flags. (no allergies or meds, intake and output is normal) Perhaps the only subjective symptom is lower energy. ... Nothing, right? Probably a cold?

Then the doctor uses an oximeter. 85% saturation. That's an immediate ER visit, with a rush to the front of the line. After x-rays show fluid in the lungs, it's an hour-long ambulance trip (a few minutes more and they would have called Life Flight) to the children's hospital. After another scan (MRI or CT, I forget), they diagnose acute pneumonia caused by stage IV neuroblastoma. Terminal cancer of the nervous system causing bleeding in the lungs.

If doctors use this vignette (and our oncologist later asked for permission to publish his case, but I don't know if he ever did), by the protocol described in The Fine Article, we'd expect to see "Cancer" in the web results for "cough". Not a common—nor particularly useful—result. More helpful, I think many might agree—is advice for when to seek medical attention for something more serious, and when it's probably just a cold.

My point at the end of all of this is that although the study is useful and interesting, it has some methodological implications which should not be overlooked in making recommendations for change.

Those pulse oximeters that clamp on your finger are cheap. Maybe $20. If you have a Frys, $15 on sale pretty often. Everyone should have one.

The Veridian puls oximeter I use looks similar to the one my doctor uses. I'm not sure if that is good or bad!

As an eye doctor I could have told you this was true without any research. I can't tell you how many patients I've seen that thought they had a retinal detachment from what they read on the internet. Instead they almost always have much more common and benign issues that have similar symptoms. Docs hate WebMD and Google for this reason. Causes people to panic and even worse self diagnose and treat.

It can also happen with the flip side of the coin too. People will ignore that new floater in their vision and Google told them it was OK. Then they walk in my office a week later with 3/4 of their retina detached.

As a 41 year-old eye patient (open-angle glaucoma in both eyes; cause was juvenile angles) I was fortunate enough to be under the care of an excellent LASIK surgeon who monitored my high IOP condition (and treated it with drops) until my pressures leaked into the danger zone (high-20's, low-30's) at which point he referred me to an excellent glaucoma specialist.

WebMD to some degree was a starting point for me to have educated discussions with my specialist about my condition (once diagnosed) and allowed me to actively participate in my care plan and decisions about courses of treatment. My first day in her office, I had turned my several years of IOP measurements into a spreadsheet (along with the detailed records I had brought from my previous eye doctor) and graph the progression of my disease. Without the basic education that WebMD and their ilk provided, I wouldn't have been able to distill the right information and they would have been fumbling through several hundred pages of clinical data in a 30-minute consultation. They noted how useful this was for an initial diagnoses.

That education didn't stop with WebMD, of course, and I went down the deep rabbit-hole of clinical research papers and videos on glaucoma, medications and surgical procedures. Because I was able to talk intelligently about my condition with my specialist, give her the right information about what was going on with my eyes from a symptomatic perspective and make educated decisions about my care, I was able to build a great doctor-patient relationship with good outcomes. She said I was one of the best patients she ever had, and not because I ever disagreed with any decision she made, but because I actively participated in the discussion with good insight.

Doctors don't always have the time to spend with their patients, thinking through all the various details. Sites like WebMD can help some people act as their own "educated" representatives in their health care decisions (granted, depending on the patient, that can be a double-edged sword). It should not be a tool for diagnoses, that is the responsibility of the doctor, but it can be used to supplement medical experience with first-hand patient input and improve overall outcomes.

I would like to have seen the results of the same symptoms being given to a number of suitably qualified professionals and see what their success rate was. I suspect that the professionals success rate would not be 100% but hopefully significantly better than WebMD's.

This isn't really surprising. But I wonder: how would real doctors do when presented only with a textual description of symptoms, without any ability to examine the patient or run supplementary tests? I'd be interested in a similar study looking into this, as a baseline for comparison.

Let your doctor make the diagnosis.Don’t go to the internet, you’ll either think you’ll have cancer, or that its something benign when its not.

If you can't afford medical tests/diagnosis, like myself, WebMD is the next step. I would rather be told something is benign when it's not. Saves money that way.

Thats awful, but it overall does not save money,One day it will be wrong, and it could cost you everything.

The cheapest way is early diagnosis and preventative treatment.

I agree that early diagnosis is much preferred. But it isn't necessarily cheaper. An early diagnosis of, say, diabetes, will lead to many, many years of treatment - and expense. Ignoring diagnosis or postponing it may well lead to death - or life-shortening disability - which, overall, would be cheaper.

The human body is damned complicated. There are a lot of parts & a lot of ways they can fail.

Being right 50% percent of the time may not be particularly useful, but that doesn’t mean it’s not impressive. This is a hard problem.

With our phones and devices we are already collecting a lot of biometric data on people. In theory you could have a dozen pictures a day of someone face, pictures of every meal they eat, where they have been, their heartrate & activity level.

I wonder what kind of insights into human health you might find when you stitch all that data together.

As an eye doctor I could have told you this was true without any research. I can't tell you how many patients I've seen that thought they had a retinal detachment from what they read on the internet. Instead they almost always have much more common and benign issues that have similar symptoms. Docs hate WebMD and Google for this reason. Causes people to panic and even worse self diagnose and treat.

It can also happen with the flip side of the coin too. People will ignore that new floater in their vision and Google told them it was OK. Then they walk in my office a week later with 3/4 of their retina detached.

As a 41 year-old eye patient (open-angle glaucoma in both eyes; cause was juvenile angles) I was fortunate enough to be under the care of an excellent LASIK surgeon who monitored my high IOP condition (and treated it with drops) until my pressures leaked into the danger zone (high-20's, low-30's) at which point he referred me to an excellent glaucoma specialist.

WebMD to some degree was a starting point for me to have educated discussions with my specialist about my condition (once diagnosed) and allowed me to actively participate in my care plan and decisions about courses of treatment. My first day in her office, I had turned my several years of IOP measurements into a spreadsheet (along with the detailed records I had brought from my previous eye doctor) and graph the progression of my disease. Without the basic education that WebMD and their ilk provided, I wouldn't have been able to distill the right information and they would have been fumbling through several hundred pages of clinical data in a 30-minute consultation. They noted how useful this was for an initial diagnoses.

That education didn't stop with WebMD, of course, and I went down the deep rabbit-hole of clinical research papers and videos on glaucoma, medications and surgical procedures. Because I was able to talk intelligently about my condition with my specialist, give her the right information about what was going on with my eyes from a symptomatic perspective and make educated decisions about my care, I was able to build a great doctor-patient relationship with good outcomes. She said I was one of the best patients she ever had, and not because I ever disagreed with any decision she made, but because I actively participated in the discussion with good insight.

Doctors don't always have the time to spend with their patients, thinking through all the various details. Sites like WebMD can help some people act as their own "educated" representatives in their health care decisions (granted, depending on the patient, that can be a double-edged sword). It should not be a tool for diagnoses, that is the responsibility of the doctor, but it can be used to supplement medical experience with first-hand patient input and improve overall outcomes.

Like you, I've only read WebMD to understand medical conditions. Seems fine to me, though I look at Hopkins too. The article is about the WebMD symptom checker, something I never used.

Let your doctor make the diagnosis.Don’t go to the internet, you’ll either think you’ll have cancer, or that its something benign when its not.

I prefer it slightly different.

Get diagnoses from doctors.

The relationship people develop where they have "their" doctor is not actually very healthy. Doctors differ for reasons. One doctor may see you for fifteen years and never notice signs of a condition another doctor notices on your third visit. Just the fact you have seen your doctor a hundred times since you were a teenager does not mean his diagnosis is correct.

I have had a doctor look at me point blank and tell me that men cannot physically get a yeast infection. In actuality, around five percent of men get one at least once in their life, so, it is rare but not so rare a doctor should not be aware of it.

Dropping the doctor for WebMD is not a great idea, though. Just try not to get this idea of "your doctor" being "good." He is good at what he studied, which is a tiny percentage of the medical field no matter how good he is. Move around, it is the best thing for everyone. Doc sees new patients, you get new advice.