Quality control on the medical Internet: proposal for a website registry

Michael Jacobson, MD, MPH

The past few years have witnessed a dramatic expansion in the amount
of publicly available medical information on the Internet. Such information,
often targetting consumers, varies greatly in format (discussion groups,
websites), in sponsorship, and in quality. Concerned with the potential
for misinformation, a number of groups are investigating quality control
methods, to help consumers find appropriate and valid medical information
online (see references). There are two basic
approaches to the problem, which are often combined: specification of a
"code of conduct" and evaluation of medical content.

Codes of conduct (such the Health
on the Net Foundation's code) specify basic criteria that medical websites
should adhere to. These criteria deal mainly with issues of accountability
and documentation: who is providing the information on the website, who
is sponsoring the site, how well information is documented, linked to and
referenced.

The evaluation of medical content is a much thornier problem,
often addressed by "rating" schemes, which attempt to rate and make public
the quality of medical information sources. Apart from subjective
or expert opinion, there are efforts at developing objective criteria for
evaluating the content of medical information on the Internet (for example,
see Mitretek reference).

Codes of conduct are valuable guidelines and should assure a minimum of
accountability. On the other hand, centralized attempts at rating the quality
of medical information on the Internet are unlikely to succeed. Consumers
who turn to the Internet for information (medical or other), are often
looking for independent sources and tend to be skeptical of central authorities.
Even if such a system were to gain consumer acceptance, its implementation
is questionable. With vast sums of money at stake for pharmaceutical
companies and provider organizations, and with so much clinical uncertainty
in medicine, any attempt at evaluating medical content by a single organization
is likely to become mired in acrimonious discussions. One need only think
of the "calcium channel blocker" controversy, and imagine attempting to
rate a pharmaceutical industry-sponsored website that provides consumer
information about the treatment of hypertension.

Nevertheless, the problem of evaluating medical information on the Internet
needs to be addressed. One solution, to be described here, makes use of
the Internet's capacity for interactivity to allow multi-participant evaluation
of medical resources.

Proposal for a website registry system

The backbone of the system would be a database of medical websites,
hosted by an administering organization. Any medical internet resource
wishing to participate would register with this organization and would
be required to abide by a code of conduct (such as the HON
code). The code of conduct would specify issues of accountability and
organization, but would not directly specify matters of content (which
are essentially unenforceable). Failure to adhere to the code would entail
exclusion from the database.

Content evaluation would consist of open feedback from visitors
to the site as well as more structured evaluations from organizations wishing
to participate in the evaluative process (such as medical specialty societies
and evidence-based medicine groups). Feedback would be collated by the
central organization, and made publicly available on the registry's website,
but would not be significantly moderated or censored.

The registry would also function as a searchable database of medical
information on the Internet. A search engine could index all registered
sites and allow extensive searches for medical information, while providing
users with instant feedback and reviews of the sites that turn up. Searches
could also be customized -- for example, a user might specify a search
for "calcium channel blockers" and request that only those sites that score
a satisfactory minimum rating by certain organizations be returned.
Thus, one could search the medical internet and concentrate on those authorities
that one trusts.

The advantage of this system is that it makes use of one of the most powerful
features that the Internet has to offer -- the ability to collate information
and feedback from multiple sources. The lack of reliance on a single rating
authority obviates the need for the development of all-inclusive quality
criteria and decentralizes the rating process (any centralized rating system
would give too much power and authority to the rating organization).
Send feedback for posting
to this page (see comments below)

Feedback

This is an interesting topic. To prepare for a recent presentation
about health information on the Web, I reviewed quite a lot of sites offering
information about immunization and vaccine preventable disease. In
addition to reliable sites offering credible information (HHS's Healthfinder,
American Academy of Pediatrics, CDC, Minnesota Department of Health) there
were many offering information that ranged from self-serving and therefore
skewed (the Advil site) to blatant misinformation (anti-vaccine sites).
I also looked at several review sites for health information and the HONcode
site. I was left with several impressions.

The issues we are dealing with in attempting to regulate the accuracy
of health information are not new. As long as there has been health
information, there has been misinformation. In Britain three centuries
ago, it was decided that an educated populace was best able to deal with
a public health crisis of staggering proportions-outbreaks of bubonic and
pneumonic plague. Bills of mortality were published (with trepidation)
that informed the public how many people were dying in what
parts of London. At the same time, a plethora of misinformation
was disseminated about causes of plague and methods of preventing transmission.
As long as we believe an informed public is a healthier public, we will
have misinformation.

The other impression I had from reviewing Web health resources is that,
although people have to some extent learned to distinguish reliable information
from misinformation, the Web changes the rules. As a general rule,
I believe the public has developed visual cues that they use to decide
whether printed information they read is credible. Does the brochure
or flyer they are seeing look professional; that is, is it attractive,
printed on good paper, and designed and written clearly. This is
the reason our organizations spend money on good communications departments
and printing budgets. On the Web, those same rules don't work to
discern quality information. Very nearly everything looks good.
Some of the anti-vaccine sites I reviewed had the clearest design and writing,
the most intuitive navigation throughout the site, and the best graphics;
certainly better than some of the government sites with minimal budgets.
We can no longer (if we ever could) judge health information by its appearance
and yet that is how most Web users determine quality. (“How users read
on the Web” by Jakob Nielson).

These observations lead to no answers, of course, but it has convinced
me that any effort to educate consumers of health information on the Web
must include an emphasis on criteria to replace the visual cues that no
longer work. JAMA's list (authorship, attribution, disclosure, currency)
is a good place to start although there are other good ones. They
aren't difficult concepts to communicate.

Sorry for the long ramble, but it's an interesting subject. I look forward
to hearing more discussion about a registry. My first reaction is
that it would add one more layer between me and what I want to reach. We
Web users have also become very impatient. Although I used to wait 2 or
3 days for a literature search from the library, a ten second delay at
Medline now seems interminable.

I think the "problem" of how cheap and easy it is to look
professional on the Web is a major one. It is one reason why some sort
of quality control is necessary on the Internet, even though nothing remotely
approaching it is available for consumers browsing the "health" section
of their neighborhood bookstores.

As for the registry adding another layer and delay -- if implemented
properly it shouldn't have to add any time at all to your searches and
browsing (depending on how the registry is implemented and how you decide
to use it). -- mj

>In a nutshell, I believe that centralized "rating" of medical websites
will not be successful because>of both consumer skepticism and the difficulty of impartially judging
medical content.

I am surprised that you believe that it is difficult to impartially
judge medical content. Isn't the "impartial evaluation of medical content"
called medical science?

I imagine that both clinicians and patients usually seek information
from medical knowledge databases that are scientifically-verifiable.

They are looking for therapy and management of conditions that is experienced,
evidence-based, and built on hierarchies of excellence such as consensus
views of specialists and practititioners.

>As a possible solution, I suggest a medical website registry which
would>collect, collate and publicize the ratings of various organizations>along with open viewer feedback.

As a medical practitioner, who is attempting to meet the challenge of
keeping up on diverse guidelines of professional
medicine, why would I go to a registry that combined sites and comments
from faith healers and homeopaths?

If I was a homeopath why would I care about randomized clinical trials
of pharmaceutical products?

>In order to be listed with the>registry, medical internet resources would have to comply with
a basic>code of conduct (such as the HON code).

Unfortunately, the HON code does not acknowledge the hierarchy of knowledge
in medical science. Perhaps medical practitioners can take hold of the
evaluation of web content. I believe that they have an appropriate role
to communicate their experience and study of what works. Clinicians seem
to share a common view of effective medical therapy. I am not sure I see
the difficulty in developing consensus quality rankings for health and
medicine sites on the web with effectiveness, evidence, and excellence
as measurement parameters.

I imagine that Universities, publishers, and societies will support
search engines and sponsor an increasing amount of web content under well
recognized quality "banners" thus making these issues less important.

Thanks, as usual, for your input. I would like to make two general
points before replying to your specific comments.

No matter how the issue of quality control is approached, I believe
it is useful to consider the assessment of content separate from all other
issues (which are primarily issues of accountability and "bookkeeping").
These latter issues are also important but are relatively well-circumscribed,
easy to define and straightforward to apply. Who sponsors a site, who is
responsible for its content, how those individuals can be reached, how
well-linked the references are, how changes in content are documented,
how medical advice is handled -- these are all vital matters. They
represent a floor, a minimum, without which the assessment of content
is fairly meaningless.

The assessment of content, although obviously important, is very different
from the above issues. It is likely to generate heated controversy and
will be more difficult to define. Not impossible perhaps, but clearly more
difficult. As such, content assessment may well benefit from a different
approach than the other issues defined above. This is the reason that I
believe these issues should at least conceptually be kept distinct.
And this is why I consider the Health On the Net code of conduct a very
valuable concept. Not that it is all we need -- of course we need to address
the issue of content as well. But it does address the whole realm of non-content
issues, and does it quite nicely.

As for how we should actually address the quality of medical content
on the Internet, I would suggest that we make use of one of the most valuable
properties the Internet has to offer: interactivity. The ability
to gather and disseminate different viewpoints is something the online
medium is uniquely suited to, and it is something we have not yet begun
to exploit adequately. As Steven McGeady from Intel said, at the Harvard
conference on the Internet and Society, "browsing the Web is a bit like
walking through a shopping mall that's been neutron-bombed: lots of merchandise,
but there's nobody there". Let's put some people and points of view
there.

To respond to your points individually:

>I am surprised that you believe that it is difficult to impartially>judge medical content. Isn't the "impartial evaluation of medical>content" called medical science.>>I imagine that both clinicians and patients usually seek information>from medical knowledge databases that are scientifically-verifiable.>>They are looking for therapy and management of conditions that
is>experienced, evidence-based, and built on hierarchies of excellence>such as consensus views of specialists and practititioners.

In the October 30, 1997 issue of the New England Journal of Medicine
are two articles, under the heading "Clinical Debate. Should thrombolytic
therapy be the first-line treatment for acute ischemic stroke?". The first
article is entitled "Thrombolysis -- not a panacea for ischemic stroke",
the second is "t-PA -- the best current option for most patients".
Even at the top of the evidentiary hierarchy, truth remains elusive.
How are we to rate the content of a medical website that cites exclusively
evidence-based articles, but does so in a biased fashion?

Bear in mind that what I'm talking about is rating the quality of websites
that present medical information primarily to consumers. Can we apply
the same criteria that we use when evaluating clinical trials reported
in peer-reviewed journals? How would you "rate" the content of an
industry sponsored website that cites multiple journal articles and states
that (example made up) "calcium channel blockers have been conclusively
shown to be safe for the treatment of hypertension and are as beneficial
as other agents"? What about if the wording is softened a bit (drop the
'conclusively'; change 'beneficial' to 'effective')? It's not impossible
to come up with criteria, but it certainly represents a quantum leap in
difficulty compared with determining whether or not a website's sponsorship
is clearly indicated.

>As a medical practitioner, who is attempting to meet the>challenge of keeping up on diverse guidelines of professional>medicine, why would I go to a registry that combined sites and>comments from faith healers and homeopaths.>>If I was a homeopath why would I care about randomized clinical>trials of pharmaceutical products.

As a patient with an open mind, if I had a problem that "conventional"
medicine wasn't treating to my satisfaction, I would certainly want to
know what options were available from sites that scored highly on, say,
acupuncture and homeopathy ratings. At the same time, I would want to be
sure that those same sites weren't accused of dangerous practices by respected
conventional medical authorities. As a physician, if I were caring
for a patient with a malignant but localized colon polyp who insisted on
taking coffee enemas, it certainly would be nice to be able to show that
a website advocating such practices was given the thumbs down by some alternative
medicine authorities.

>Unfortunately, the HON code does not acknowledge the hierarchy of>knowledge in medical science. Perhaps medical practitioners can
take>hold of the evaluation of web content. I believe that they have
an>appropriate role to communicate their experience and study of what>works. Clinicians seem to share a common view of effective medical>therapy. I am not sure I see the difficulty in developing consensus>quality rankings for health and medicine sites on the web with>effectiveness, evidence, and excellence as measurement>parameters.>>I imagine that Universities, publishers, and societies will support>search engines and sponsor an increasing amount of web content
under>well recognized quality "banners" thus making these issues less>important.>

See my comments above on the HON approach, and on the difficulty of
judging the content of medical websites directed at consumers.

Bottom line -- I think that a non-content related code of conduct is
crucial and is enforceable. I agree with you that the rating of content
is also possible, but I think it is much trickier, especially when you're
talking about information directed at consumers. Also, it might benefit
from a multi-individual and multi-institutional, open and interactive approach.

Ultimately, what consumers are looking for on the Web site is the content.

An evaluation system that addresses links, currency, sponsorship, etc.
is useful in that it can be used as a general screening to avoid Web sites
that are too cumbersome to use or are more likely to have poor content.

However, the crafty huckster can be sure to meet these criteria and
get the "best" rating, regardless of the quality of content.

Evaluation of content will need to be in place to make using the "evaluated
website database" useful in meeting your ultimate goal of finding content.

A registry using as criterion a HON type code of conduct might be useful
especially in relation to the declaration of financial interests and of
financial sponsoring. The database of medical websites would also be useful.

However, I am worried about the proposed "content EVALUATION" by, among
others, "medical specialty societies and evidence-based medicine groups"
as they may hamper dissemination of some content. Content CLASSIFICATION
might be more useful because at all cost evaluation bias must be avoided
if the Web is to remain useful.

I wonder how well my own webpage about cardiovascular diseases and their
link (i.e. cause and prevention) to foods, supplements and vitamins would
do when "content evaluated" by "medical specialty societies and
evidence-based medicine groups" ( http://www.health-heart.org/
).

These are the same groups that authored the excellent August 27 1998
Heart Failure Supplement of The Lancet (Vol. 352s1) and concerning which
the Editor's discussion admits: "... that despite the advances in
the
management of coronary heart disease there is no clear evidence that
therapeutic advances have made any impact on the overall burden of disease.
([link no longer available]).

Evidence based medicine was an unsuccessful judge as to the effects
of treatment advances. Why would such groups be any better at evaluating
websites relating to "unconventional" techniques.

A few years ago ulcers were not thought to be caused by a bacterium
(H pylori) while the importance of Lp(a) and vitamin E in vascular diseases
and of homocysteine (i.e. B vitamins and betaine) in NTD's and in cardiovascular
diseases is just now becoming generally accepted. Such and other ideas
need and deserve prominent outlets.

Actually, one of the potential advantages of a website registry
with multi-participant content evaluation (including, but not limited to,
medical specialty societies and ebm groups) is the increased likelihood
that non-mainstream medical sites get consideration and exposure.
-- mj