Saturday, 16 April 2011

Lies, damned lies and statistics: How do you turn 61% into 95%?

Edit: 13/05/11 An analysis of the BSA 2007 dataset by Siobhan Farmer, Mark Hawker and myself has been published today in the new publication Lancet UK Policy Matters. You can find it here. We conclude that it is not possible to conclude that 95% of respondents wished for more choice. Using the kind of suppositions given below it may be possible to infer that between 61% and 72% may have thought there should be more choice, but the survey was not designed to answer this question.

Edit: 17/4/11 I and a colleague have independently tried to verify Mark's analysis. We have reached similar conclusions but they don't corroborate Mark's results. Unfortunately he is currently outside the UK. We will update with our own results in the UK. We are in agreement that there is still no justification for claiming that "95% of people want more choice in healthcare".

Page 11 of the leaflet contains the graphic above stating that 95% of those surveyed in the 25th British Social Attitudes Survey wanted MORE choice in the NHS. When Ben looked at the published reports he found that 'Do you want more choice in the NHS?' was not a question in the survey. Instead respondents were asked 'How much choice do you think you should have?' and 'How much choice do you actually have?' But if you could establish how many people thought they should have choice but who currently think that they don't have choice then you might be able to say how many people think that they should have more choice. As Ben points out to answer this you would need to have individual level data. When he asked the DOH for the dataset they unfortunately pointed him to a book chapter.

So how did the DOH manage to get this so wrong? How did they confuse 41% with 95%? Why weren't they able to direct Ben Goldacre to the correct data source? And why have they decided not to fund this survey in the future?

Hopefully someone can help make the correct data look just as pretty as the incorrect infographic in the leaflet. In the meantime I think I'd like to thank Mark for his work, and to agree with this tweet:

1 comment:

Anne Marie, great post. I guess the next question is then, how do we hold them to account? By which I mean, how do we support them to view this as a significant event from which to learn and do better next time...or am I being naive?