Wait, wait, wait. Now stop. Think about what you've been told. Now. Does it sound reasonable?

Tuesday, November 2, 2010

A new strange type of Quarkery.

Strange Quarks is a new podcast that has entered the downloadosphere looking at the Ups and Downs of science-based politics in the UK, from the Top of science to the bottom of pseudoscience, with two charming hosts; Martin Robbins off of the Guardian, and Michael Marshall off of Righteous Indignation, Skeptics with a K, Inkredulous and co-organiser of next year's QED conference.

My own report is in there too, and it is probably the campest piece of overacting ever to be seen by this blog. (To be mentioned on the Guardian Online website made it all worth it....)

The podcast is produced in conjuction with Pulse Project and the .mp3 can be downloaded from their website here or you could wait and get it from iTunes (should be up in a few days).

For those of you who prefer their content readable with links, rather than audible, here is a transcript of the report, which may be less fun without my squeaky mixed-up regional accent and intonations. Each to their own :)

--------

Who is responsible for responsible science journalism? An itchy case study.

Earlier this year, Professor Dorothy Bishop from the University of Oxford announced that she was launching a new prize for science journalism which she called the Orwellian Prize for Journalistic Misrepresentation. It would be awarded annually to the English-language national newspaper that has published the most inaccurate report of a piece of academic work.

“Why is science reporting in the media so rubbish?” you’ll probably have heard someone ask.

“That’s because scientists are notoriously bad at engaging the public,“ will no doubt come the media-savvy reply.

“They tend to talk in the language of dry research papers because publication numbers are all that matters”

“But journalists are so lazy they just print any old nonsense and rarely have any real comprehension of the science or its impacts”, someone will chirp, probably a scientist.

In reality, all of that and more is true, and a story which was run recently by a number major news outlets in the UK is perhaps a pretty good case study.

Eczema is extremely common and can be pretty tortuous, so this news will be of interest to millions of people. Could this eczema cream really be making eczema worse?

According the articles, the research had been carried out at the Pharmacy and Pharmacology dept of the University of Bath and published in the British Journal of Dermatology. A respectable institution publishing in a respectable journal – any wonder this story was big. The University had put out a Press Release about the research and had kindly put it on its website, with the headline

The media outlets weren’t exaggerating – here it is from the horse’s mouth (or more accurately the horse’s owner’s mouth, if you want to stretch the metaphor beyond it’s elastic limit)

Incidentally a search for that exact phrase “Creams used to treat eczema could make it worse” comes up over 5000 hits in a google search, mostly syndicated news outlets. Given the media coverage of this story, this research must be top drawer, watertight stuff. I mean, the researchers wouldn’t allow their university to risk both parties’ reputations by putting out a press release which gets global coverage and could impact the treatment regime of millions of eczema sufferers, but was based on unreplicated, duff research? Would they? Can you hear an impending air of disappointment in my voice?

The cream in question is Aqueous Cream BP, a cheap emollient – which is a substance that soothes or softens the skin. It is a light moisturiser which helps stop the outer skin layer from drying out and becoming itchy and flaky. It’s one of the main treatments for general eczema, along with not scratching, avoiding environmental triggers and using steroid cream against more aggressive flare-ups.

The BP in Aqueous Cream BP means it is a British Pharmacopiea standard and anyone selling it must make it to the standard recipe. It’s basically an emulsion of paraffin and water, with some preservative. The emulsion is made using an ingredient called SLS a cheap and cheerful surfactant used in myriad household products, although SLS can be a problem as sensitivity to it is well known and can itself cause reactions which exacerbate the eczema. So perhaps this is why the “Eczema cream can make eczema worse”?

It would be valuable to know what the sensitivity of eczema sufferers is to this workhorse treatment, and given the numbers of people with eczema, it should be cheap and easy to run a test on a large random bunch of healthy-skinned and eczema sufferers, giving them either Aqueous Cream BP or an SLS free equivalent. Any questions?

Let’s have a look at the actual published research – it turns out the researchers in question Tsang and Guy didn’t quite do that:

First of all, rather than testing a large bunch of people there were 6 participants.

And they were all women.

And they were all aged 20 – 36 years of age.

Now perhaps in idle moment of fanciful daydreaming, I like to think that the population of the UK is entirely made up of 20 – 36 year old women and me, but sadly it isn’t.

So this sample is a long way from being representative, quite aside from the point that any study regardless of how rigourous, if done in 6 people is probably not worth a hill of eczema flakes on its own.

So what about the ratio of healthy-skinned people to eczema-sufferers on the trial?

Well they didn’t treat anyone with eczema, they only tested the cream on people with healthy skin. I hope I don’t need to embarrass both of us here by stating the gigantic flaw in research that doesn’t actually test the thing against the thing that you’re claiming it makes worse, but somehow that passed the researchers by.

The researchers also seemed to be doing funny things with the statistics. Each of the 6 women did the same test, twice on each forearm – so four results per person. They then claimed this meant their sample size was 6 times 4 - 24. Still not a large enough sample to carry much weight but that’s not the point. It’s like trying to find out people’s opinions and rather than asking 24 people, asking 6 people 4 times. The accuracy of the individual result should be better, but the sample size won’t change from 6.

So as a quick summary of the research, they concluded that SLS, an ingredient they didn’t control for, can make eczema, a condition they didn’t test on, worse in the population, despite only testing it in an extremely small number of young women.To me , this is the worst kind of research – it’s trying to answer a question it didn’t ask, makes a very definite conclusion despite being incredibly weakly powered, and in actual fact doesn’t add any real knowledge to issue of whether of SLS is a problem in Aqueous Cream.

So the scientists did the research and came to a conclusion which was perhaps bolstered in order to make it to publication, the University took the conclusion and passed it through a media-friendlifier which perhaps bolstered it a bit more , the media outlets then passed it through a science-mangle which tends to bolster claims and suddenly 5000 websites around the world are informing their readers that the cream your doctor is prescribing for your eczema is making it worse!

OMG.

So to go back to Professor Bishop’s award for Journalistic Misrepresentation mentioned at the beginning – I don’t know who it would go to: The newspapers, The Journal, The University or the scientists themselves.-------------

1 comment:

I loved your report! Congrats for it.Uni press depts have rather a lot to answer for, though those researchers (and the journal that published something with such a sloppy dataset & methodology - also happens far too often) want a bit of a slap as well.

Oh and I'm glad you admit to a 'mixed-up regional accent' - It's been bugging me that I couldn't pin it down 100%!! ;)

Keep Libel Laws out of Science

About Me

Who I am is largely irrelevant, and indeed so are most of my thoughts.
Nonetheless, it winds me up that I am supposed to swallow half-truths and untruths relating to scientific claims emanating directly from media sources and indirectly from people who haven't a clue what they're talking about.
Look, you've got me started.
(You can email me at thinkingisdangerousblog AT googlemail DOT com.)