Sunday, 28 April 2013

Evidence in policymaking

Edit (30th April 2013): As has been pointed out in the comments, it doesn't look like Ben Goldacre's tweet I mention in the highlighted section below was actually aimed at Jeremy Hardie. I've left it there, though, partly out of transparency but also because the debate between them is still what sets the context for this post, though obviously there's also plenty of others contributing to this too (notably, for me, the British Educational Research Association - BERA).

I’ve been noticing quite a few discussions of evidence and
policymaking recently.There’s been a
bit of a spat between Ben Goldacre and Jeremy Hardie, after the publication of
his book Evidence-based policy: a practical guide to
doing it better co-authored with Nancy Cartwright.At least, I assume that’s who Goldacre was
getting at when he tweeted:

Also, this week has seen the publication of a study
suggesting that the alcohol industry’s evidence to the Scottish Government was
misleading.This led to the inevitable
exchanges on Twitter between Mark Baird
and John Holmes, amongst
others.I’m going to write about that
specific issue separately, but the point here is that there’s much to be
discussed about evidence and policy.

What I want to do here is offer a model for understanding
policymaking that, I hope, helps shed some light on – and cut through – some of
these discussions.

I know that there are plenty of models of policymaking
already available in social policy textbooks, and I’m sure I’m reinventing the
wheel at some level.However, I want to
be clear about what I’m trying to do with the model.This isn’t about modelling how policy is made
in practice and the variety of influences on it (though there are plenty
of models for this).

What I’m trying to do instead is think in a slightly
abstract way about how policy is made: what thinking or ideas underlie
policy?That doesn’t mean that this
thinking is overt, or that policymakers themselves are always conscious of it,
but (even if it’s only implicitly) positions are taken on each of the following
aspects.I would suggest this model is
helpful for looking at existing policies, or policy debates, in order to better
understand some of the currents and influences that are underlying them.

I hope that you’ll be able to see how it’s helpful to
identify (sometimes hypothetical) steps in policymaking, conceived of as a
problem-solving process, from my next post, which will be about the role of the
alcohol industry in alcohol policymaking.

It also helps shed light on my earlier post
about alcohol and neoliberalism.If
we can see that a ‘problem’ and ‘solution’ have been defined in particular
ways, we can start to think whether there is a broader approach to the idea of
government and policy that is affecting how people approach certain issues.

However, I’m keen to stress that this is just some thoughts
about how to think about policymaking that have come to me recently; this post
shouldn’t be taken as an attempt to produce a definitive model of policymaking.I’m sure people will be able to spot gaps in
the argument, or perhaps misunderstandings of certain policy areas.If you do, please let me know, as it will
help my way of thinking about policy. Certainly, there could be more or fewer steps to the model.

1. What is the problem?

‘Problem’ is a bit too specific a word here, really, but any
policy has several aims, which might be understood as attempts to solve certain
problems.One of the things that
fascinates and frustrates me about alcohol policy is that there’s any number of
aims – and that’s crucial for all the stages that follow.However, whether it's stated clearly or not, there will always be some perceived ‘problem’ underlying a policy.

How to judge ‘what works’ is what Goldacre and Cartwright
& Hardie discuss, and with more detail and insight than I can here – or
could even if I had more time and space.The point I want to stress here is that this is not a neutral,
value-free question, because it presupposes an answer to the previous one: what are we trying to do
with the policy in question?You can
only judge whether something ‘works’ if you know what its aim is.A computer doesn’t ‘work’ very well for
cooking a roast dinner, but it might ‘work’ for looking up advice on how to do
this.

There are two issues with this.First, as I’ve already stated, there’s very
often multiple aims of policies – and sometimes these can even be in conflict
to some extent.A drug treatment policy
might want to increase the number of individuals who are free of any drug
dependence, but you wouldn’t want to do this at any expense.There’s a possibility that the same policy
that successfully pushes a lot of people through the system successfully leads
to others feeling alienated, and they therefore go on to disengage and commit
more crime, suffer more health harms and so forth.That is, you can’t judge the policy on one
aim, or on one metric.

This brings me to the second point.The measurement of ‘what works’ depends on
choosing some way of measuring success and failure – ‘operationalising’ your
concepts, as I was taught.This is where
there’s often disagreement about education policy: do rises in GCSE pass rates
indicate a successful policy, or is this metric inadequate?

Ben Goldacre, in his paper on education and research,
suggests that most of the time this operationalising isn’t an issue, as we
should be able to find decent metrics that we can agree on.To a certain extent this is true – we could
perhaps develop better metrics than GCSEs or SATs, or we could contextualise
them to understand what they ‘really’ tell us about the standards of education
in the country.

However, this only brings me back (again) to question 1:
what are we trying to do with education policy?When people disagree about standards slipping, this isn’t simply because
they don’t have faith in the exam system; they’re also often expressing an
underlying difference in opinion about what examination is for and, more
generally, what education itself is for.It is impossible to agree on a metric to measure the quality of
education if we can’t agree what we’re trying to do by teaching someone history,
for example – and there’s any number of things we might be trying to do,
perhaps all in combination (knowledge of dates, an understanding of the human
condition, an understanding of the evolution of the British institutions of
government, an ability to think analytically and assess claims of evidence or causality,
etc etc).

I realise that this makes the idea of this sort of model
questionable: if we can’t have a single measure of ‘what works’, then what is the
help of this concept in a model like this?My point is that by looking at a policy and thinking through what the
phrase ‘what works’ might mean, we can begin to see the assumptions that
underpin the policy area. If we were to
analyse the new curriculum, or the 2003 Licensing Act, or the recent Higher Education
reforms, we’d find implicit answers to these questions (even if they were only:
aim to save money, does it work measured by whether or not the deficit is
technically reduced).These help us
understand what’s actually going on in policymaking, and I’d hope give us some
idea about the process could be better.

3. How does it work?

This is really where Cartwright and Hardie attack Goldacre
for his emphasis on RCTs (randomised control trials).They say that RCTs can’t tell you whether
something will work in a different context.This is part of the more general point that RCTs can’t tell you how something works – that is, what the
precise features are that deliver outcomes.I don’t want to enter into this discussion now, but this is a key stage
in the process, because it adds to the next stage…

4. A model for intervention

This is something that actually comes out as the ‘solution’
to the problem, and depends upon a belief that something ‘works’, with
assumptions about how it works.I’m thinking really of something like NICE
guidelines for the medical profession:

There’s a problem of heroin addiction, and we
want to make sure individuals are no longer physically dependent on using
heroin or other opiates.

We’ve done studies of replacement therapies and
found that in general prescribing methadone seems to ‘work’ towards this aim
reasonably well.

However, in reality you might find that the guidelines
aren’t actually that specific or tailored, and leave plenty of room for
manoeuvre.Certainly this is the case
with the guidelines for the specific
example I’ve just given.This means
that in practice we often have a further stage before there are actually
interventions affecting people’s lives…

5. Application of guidelines by experts

In the methadone example, here we move to treatment staff
actually putting a package of care in place for an individual service user,
which will (we might hope) be based on NICE guidelines, but which we wouldn’t
be able to exactly predict because it also depends on the application of
context and clinical judgement.In
education policy, we might think of the national curriculum being the ‘policy’,
and then the application of this being actual lessons.*

There is then something of a cycle taking us back to the
beginning.Doctors, in Goldacre’s
example, implement guidelines, but on the basis of intuition, or performance
data, identify new or continuing problems, which the NIHR can then investigate,
taking us back to step 1 of the process.Alternatively this might be done by some kind of policy review.

Thinking through these steps, I’m not sure that there’s
always a 5, where guidelines/regulations are implemented by ‘experts’.In terms of MUP, for example, I can only
really think that this would imply some kind of inspection regime to ensure
that prices in shops were indeed at least at the threshold.

Conclusion

I’ll finish this post in the same way I started it: by
stressing that these are only some basic reflections about how implementing
policy entails certain conclusions to have been reached.I’d really welcome responses that critique this
model, or potentially refine it, as something like it is the way I approach
public policy discussions at the moment, and any improvement on my thinking
would be welcome.

* This is where Hardie’s
citing of Hayek’s Use
of Knowledge in Society comes from.(I didn’t really need to mention this, but otherwise there wasn’t going
to be a footnote in this post, and that just seems a missed opportunity when
all the others have them.)

2 comments:

Just for the record I don't think Hardie was the subject of Goldacre's tweet. It came during a seminar that the Alliance for Useful Evidence put on about evaluation and Whitehall and seemed to be in response to this tweet.

About Me

I live and work in Dorchester, a beautiful town in Dorset not far from the south coast.
I’m a Senior Health Programme Advisor within the team that commissions drug and alcohol treatment services through Public Health Dorset, which operates across Bournemouth, Poole and rural Dorset.
I'm also a Visiting Fellow in the Faculty of Health and Social Sciences at Bournemouth University.
How did I get here? Well I did a PhD, also at Bournemouth University, looking at ‘binge’ drinking in Bournemouth and thinking about how class and gender are part of how people understand their own and other people’s drinking. (Search on eprints.bournemouth.ac.uk)
After finishing the PhD I moved to London for a couple of years and worked for the Russell Group of universities, dealing most closely with issues of community engagement, widening participation and, unexpectedly, immigration.
When an opportunity came up to head back down to Dorset and work on drug and alcohol issues, I grabbed it with both hands.
I'm on Twitter: @WilliamHaydock