Well, the ridiculous examples of bad performance measurement keep flooding in. I’ve lost count of whether we should be on Episode 5 or 6 by this point, so I’ll just stick to new titles from now on. This time, I’m going to look at aspects of the Troubled Families agenda, after one of my readers sent me some very interesting documents…

Now, unlike some of my other posts, there is a very serious edge to this subject. If you’re just looking for a giggle about performance measurement in the toilet-cleaning world or on railway networks, then in the words of the character Morrell from Shane Meadows’ film A Room for Romeo Brass, “Why don’t you just leave, before things get really dark?” (It’s a great film by the way if you like that sort of thing).

We’ll start at the thin end of the wedge…

The Trouble with Targets

I’ve done this to death before, so rather than unpick why the following targets are daft, I’ll just show you them and let you work through the checklist that follows this excerpt from a document about troubled families.

“The education outcome is all children having fewer than three fixed-term exclusions and less than 15% unauthorised absences from school in the last three terms; the ASB outcomes are a 60% reduction in ASB and a 33% reduction in offending rates for under-16s in the last six months”.

First of all, let’s look at the significant limitations around the popular performance measurement term ‘outcomes’. There is a growing body of evidence that highlights the difficulties in defining and measuring outcomes. As true outcomes are usually difficult to quantify and measure, outcomes-based accountability programmes tend to rely on proxy indicators which are unsatisfactorily crude and primitive.

Furthermore, when attempting to tackle cross-cutting social issues it is notoriously difficult to establish a causal link between a unit of activity undertaken by a particular agency, and an eventual outcome. External environmental factors can also bear both positive and negative influences, thereby limiting the efficacy of claims that a single agency or specific intervention is the responsible factor in achieving the outcome.

Then there’s the question of who determines when the outcome has been achieved. Studies have shown that measuring the effectiveness of an intervention produces different results depending on when the evaluation takes place – i.e. efforts at determining whether the outcome is successful 6 months, 12 months or 2 years after an intervention will produce conflicting results. Which one (if any), is the correct point to stop the clock and declare success?

The Trouble with Payment by Results

Now here’s the rub. At the opposite end of the scale to the light-hearted jibes directed at silly performance indicators for dog poo, there exists some really dark thinking about how we should approach issues at the serious end of public sector business. In the case of the Troubled Families programme, the whole agenda is undermined by a particularly dangerous pathogen – Payment by Results (PbR).

The problem is that the whole ethos of PbR rests on the assumption that the considerations highlighted in the ‘Outcomes’ section of this post can be surmounted, along with spectacularly misguided confidence that the financial incentives offered under such schemes will not cause dysfunctional behaviour.

As you probably know, I argue that targets alone generate perverse incentives which lead to goal displacement, gaming and other horrible consequences. When bolstered by the prospect of hard cash for ‘results’, it is inevitable that many of the most unpalatable effects of target-driven performance management will be intensified. Extensive research across the social sectors, (in particular education, employment, health and policing) consistently indicates that adverse consequences are ubiquitously borne of public sector PbR programmes. To see examples of this for yourself, I highly recommend the following paper, which explores the phenomena in some detail, utilising a range of evidence:

Examples from within this particular article resonate with many of the very areas that the Troubled Families programme aims to address, such as educational achievement and work placements for the unemployed. It warns of ‘cream skimming’ (i.e. cherry picking those who are most likely to succeed against the PbR criteria; often at the expense of those most in need), as well as other behavioural risks that result from inappropriate performance incentives. It’s not pretty stuff.

The mind boggles at the prospects for the Troubled Families agenda. If previous experience is anything to go by, I fear for those families just below specified thresholds or slightly on the wrong side of definitions. I worry that the best efforts of caring individuals will be hamstrung by distorted priorities and wrong-headed cash-linked performance incentives. I am certain that the PbR-based approach will lend itself to overlooking purpose and marginalising many of those whom the programme is supposed to help.

One more thing – Please understand that this is not a criticism of the aims of the Troubled Families programme (and definitely not a slight on those committed people who are doing their very best to address the important and emotive issues it encapsulates), but rather an emphatic and considered health warning about the associated performance methodology which risks completely derailing those virtuous aims.

So there you have it. Not as pithy and light-hearted as usual, but as Morrell says in that film, “I can’t destroy you today lads – I’m on very serious business”.

8 Responses to Troubled Thinking

Please understand that this is not a criticism of the aims of the Troubled Families programme (and definitely not a slight on those committed people who are doing their very best to address the important and emotive issues it encapsulates)

It will be taken as such, though. There are many out there who are incapable of distinguishing between the question “should we be doing this in this way?” and “should we be doing this?”.

Yes, that target checklist is pure gold.
Looking at the criteria for being a troubled family, wouldn’t the Royal Family fit with most of those criteria? The ginger one dressing as a nazi is surely anti social behaviour. They receive “out of work benefits” I reckon, and they’re certainly a “high cost to the public purse”. They share other similarities with troubled families, they live in housing funded by the state, are a huge clan, have a high rate of divorce and nobody would want them moving in next door to them.
Throw in a racist grandad and it’s basically Shameless without the drugs.

Performance indicators checklist… Finally got an exam question 100% right for once in my life! Does that mean I can get an ‘ology’ now? Am I a scientist? No, but if ‘dim’ people can see it why can’t the others? Because they don’t want to, it serves their purposes to go with the flow for the benefits of their self-interest.

As one of those that have to work to this agenda, I do not feel slighted, I just would like others to understand that unfortunately this is the world that we are made to work in. Targets set by some penpusher working in their ivory towers with no grip on reality, targets that suit the public ear, targets that make CaMORON and his demolition government look good whilst piling the pressure on those on the front line, that my friend is the reality. Slainte