Quality of science sources in Wikipedia and the news

Several weeks ago, I wrote an article on how to decipher the science (or pseudoscience) in popular news articles. It discusses how we should be critical, if not skeptical, of what is written in these articles to ascertain what is or is not factually scientific. We even need to determine the quality of science from the best to the weakest, so that we can determine the level of authority of the science before we pass it along to others. With the social media, like Facebook and Twitter, which provides us with data that may not exceed a few words, then it’s even more imperative that we separate the absurd (bananas kill cancer) from the merely misinterpreted (egg yolks are just as bad as smoking).

Wikipedia is one place which can either be an outstanding resource for science or medicine, or it can just a horrible mess with citations to pseudoscience purveyors. For example, Wikipedia’s article on Alzheimer’s disease is probably one of the best medical articles on the “encyclopedia”. It is laid out in a logical manner, with an excellent summary, a discussion of causes, pathophysiology, mechanisms, treatments, and other issues. It may not be at the level of a medical review meant for a medical student or researcher, but it would be a very good start for a scientifically inclined college researcher or someone who had a family who was afflicted with the disease.

Nearly everything in the article is supported by a recent peer-reviewed journal article. Furthermore, the article does its best to avoid primary sources (ones in which the authors directly participated in the research or documented their personal experiences) while utilizing secondary sources (which summarizes one or more primary or secondary sources, usually to provide an overview of the current understanding of a medical topic, to make recommendations, or to combine the results of several studies). The reason secondary sources are so valuable is that they combine the works of several authors (and presumably locations), eliminating biases of one laboratory or one study. As I’ve said many times, trust your secondary sources over just about anything, and, of course, Cochrane Reviews are nearly the best of the secondary sources.

So when you’re reading a Wikipedia or news article, and it’s making a medical claim, and it’s basing it on some primary research (or worse yet, primary research on animals or cell culture), then it’s a long way to making a conclusion about humans. A very long way.

But it’s also important when you’re reading Wikipedia or science articles, to make sure that you don’t get fooled by what may appear to be valid science, but really aren’t. Many journals, and a lot of research institutions, are sending out press releases when new articles are being published. No one should dispute that these institutions have the right to “market” a new advance in science, but they don’t qualify as a reliable source for information.

Specifically, the results support the hypothesis that university press offices are prone to exaggeration, particularly with respect to animal studies and their relevance to human health and disease, although press releases about human studies exaggerated 18% of the time compared to 41% of the time for animal studies. Again, this seems to make intuitive sense, because in order to “sell” animal research results it is necessary to sell its relevance to human disease. Most lay people aren’t that interested in novel and fascinating biological findings in basic science that can’t be readily translated into humans; so it’s not surprising that university press offices might stretch a bit to draw relevance where there is little or none.

This is very important, something that I see over and over again. There is a tendency to over-dramatize results from animal studies, when only a small percentage of compounds that are tested in animals or cell cultures ever make it into human clinical trials (let alone are approved by the FDA). The National Cancer Institute has screened over 400,000 compounds for treating cancer, and maybe 20-30,000 have even made it to early clinical trials, and of those just a handful are used in modern medicine. You have to be extremely skeptical of reading an article that has source to a press release that might overstate the results (or even if they refer directly to such a primary study).

Gorski continues with a more important issue:

In human studies, the problem appears to be different. There’s another saying in medicine that statistical significance doesn’t necessarily mean that a finding will be clinically significant. In other words, we find small differences in treatment effect or associations between various biomarkers and various diseases that are statistically significant all the time. However, they are often too small to be clinically significant. Is, for example, an allele whose presence means a risk of a certain condition that is increased by 5% clinically significant? It might be if the risk in the population is less than 5%, but if the risk in the population is 50%, much less so. We ask this question all the time in oncology when considering whether or not a “positive” finding in a clinical trial of adjuvant chemotherapy is clinically relevant. For example, if chemotherapy increases the five year survival by 2% in a tumor that has a high likelihood of survival after surgery clinically relevant? Or is an elevated lab value that is associated with a 5% increase in the risk of a condition clinically relevant? Yes, it’s a bit of a value judgment, but small benefits that are statistically significant aren’t always clinically relevant.

Now, as Gorski states, this is a bit of guesswork and instinct, but none of us should probably accept results as meaningful if they are tiny, even if statistically significant. I see this a lot, especially with alternative medicine studies that try to “prove” that they have some benefit beyond placebo. But if there’s a new therapy, or “eat XYZ and it will reduce ABC by 5%”, those results may just be no different than random.

Just as bad as press releases are meeting abstracts (which, of course, are promoted by press releases by research institutions), which are incomplete and unpublished data and undergo varying levels of review; they are often unreviewed self-published sources and these initial conclusions may have changed dramatically if and when the data are finally ready for publication. According to a 2002 paper in JAMA, a total of 252 news stories reported on 147 research abstracts. In the 3 years after the meetings, 50% of the abstracts were published in high-impact journals, 25% in low-impact journals, and 25% remained unpublished. Interestingly, the 39 abstracts that received front-page coverage in newspapers had a publication rate almost identical to the overall publication rate. The authors concluded that, “abstracts at scientific meetings receive substantial attention in the high-profile media. A substantial number of the studies remain unpublished, precluding evaluation in the scientific community.”

Meta

Medical disclaimer

Nothing I write should be construed as medical advice. No one should receive a medical diagnosis or treatment recommendation over the internet in any case.

If you need a real medical diagnosis and treatment, seek out a licensed medical practitioner who has been well-versed in evidence- and science-based medicine. They can provide an appropriate diagnosis, then discuss your particular medical treatment options.

Vaccines are an important point of discussion on this blog. Once again, all decisions about vaccines should be done after a discussion with a licensed healthcare practioner who utilizes evidence-based medicine. Be aware that homeopaths, naturopaths, chiropractors, and other pseudoscience based woo-meisters are simply not knowledgeable about real diagnostic and treatment options, and should always be ignored.

Privacy Policy

This website does not collect any data about any specific user unless using the comment system, in which you log in through WordPress, Facebook or Yahoo. SkepticalRaptor.com does not capture nor store any login information, except when you post comments.

The website does collect information about each visitor's location (by country or state/province), pages visited, amount of time spent on each page, and other information such as in- and out-links. That information is not shared or sold to anyone. Period.

All visitor information is randomized, so no one would be able to tie any activity on this website with any specific IP address. If a law enforcement agency requested my records, they would be incapable of tying any IP address with any activity on this website.

Any threatening or harassing comments would be reported through the providers of the commenting system, whether it's Yahoo, Facebook, or WordPress.

If you have any questions about the privacy policy, you can contact me through the link on the home page of this website.