National Institute of Economic and Social Research

Tuesday, 12 June 2012

The Department for Work and Pensions today published an impact assessmentof the Mandatory Work Activity (MWA) programme. The analysis compares participants on the programme with "comparable" (as determined by sophisticated statistical techniques) non-participants. Briefly, what the analysis shows is that the programme as currently structured is not working. It has no impact on employment; it leads to a small and transitory reduction in benefit receipt; and worst of all, it may even lead to those on the programme moving from Jobseekers' Allowance to Employment and Support Allowance.

Sadly, the DWP's press release doesn't mention any of these key points. Nevertheless, one would naturally assume that as a consequence the programme would be redesigned to achieve better outcomes. However, in a Written Statement, the Minister for Employment said:

I am also pleased to announce the Government has decided to expand the Mandatory Work Activity scheme. The expansion will enable Jobcentre Plus to make between 60,000 and 70,000 referrals to Mandatory Work Activity each year, based on the current experience of the scheme, at a cost of an additional £5 million per annum. This decision has been taken as the result of careful consideration of the positive impacts demonstrated within the Impact Assessment.

Unfortunately it is very difficult indeed to reconcile this statement with the impact assessment itself. Before I explain the details, however, I would like to congratulate both DWP Ministers and their officials for performing a detailed impact analysis, subjecting it to outside scrutiny, and publishing the results, even though they are clearly unfavourable. It speaks well of the Department and its continued commitment to evidence-based policy, and contrasts well with the muddle over Work Experience (described here), let alone the continued distortion of data and evidence by the Department for Communities and Local Government.

A team here at NIESR, led by Helen Bewley, and including me, performed a peer review of the analysis. As the DWP publication says:

"NIESR concluded that the methodology was sound, although the
nature of the selection process for programme referrals means that it is very
difficult to identify truly comparable individuals who were not referred. As a
consequence, it is possible that impacts are underestimated. However, subject
to this, the key conclusions - that MWA had a small and transitory impact on
benefit receipt, and no impact on employment - appear reasonable.

The MWA programme is described here. Briefly, Jobcentre Plus advisers can refer claimaints who they think would "benefit" from "experience of the work environment" to a compulsory four week work placement. This is somewhat euphemistic however: as the impact assessment puts it: "Guidance
to advisers indicates that a claimant suitable for referral to MWA is one who
is lacking or failing to demonstrate the disciplines and behaviours needed to
seek out and secure employment."

In other words, MWA is essentially a tool for advisers to use on those they think are not playing the game: blatantly failing to look for work, or perhaps even doing something else on the side. The hope is not so much that participants will be encouraged to get a "real" job, but that they will find the experience of mandatory work so unpleasant (and/or incompatible with other activities they'd rather be doing..) that they will sign off benefit one way or the other. Indeed, for many the prospect of MWA may be sufficiently unappealing that simply being referred to the programme may be enough to get them to sign off. And this is certainly what the international evidence suggests.

So what would expect, based on the programme rationale and the international evidence? Ex ante, I would have said this:

lots of those referred wouldn't show up for the programme

there would be a substantial drop in benefit receipt - some of this would be only temporary, but some would be permanent

there would be little or no impact on employment

And what happened? Well, drop out was indeed very high; only about 55% of those referred to MWA actually started the programme. But the impacts on benefit receipt were disappointing. Among those who did actually start the progamme, there was essentially no impact. And while there was a substantial impact on those who didn't start - as you would expect, given that the programme was compulsory and they could be sanctioned for non-attendance - this was transitory. Overall, the maximum impact was a 5 percentage point reduction in benefit receipt, and only 13 weeks after starting the programme the impact had disappeared completely. On average, someone referred to MWA spent just 4 days less on benefit as a result.

So what appears to have happened is that while some of those referred did leave benefits as result, they drifted back on quickly. There was no lasting positive impact at all. Indeed,quite the contrary: it appears that some of them returned as ESA rather than JSA claimants:

13 weeks after referral, those referred were 3 percentage points more likely to be on ESA. Not to put too fine a point on it, this is a complete policy disaster. ESA claimants are both more expensive and more difficult to get off benefit than JSA claimants. Indeed, the main thrust of welfare-to-work policy under both this government and the previous one has been to try to move ESA claimants closer to the labour market. MWA appears to achieve precisely the opposite. We can only speculate why, although the obvious answer is that the "hassle factor" of being referred to MWA had the unintended consequence of encouraging some claimants to claim a benefit - ESA - where there is not necessarily any obligation to look for work at all. In any case, whatever the explanation, the long-run costs of moving even a few JSA claimants to ESA will clearly outweigh any possible other benefits of the programme.

Finally, what about employment: no surprises, and no impact at all, here:

There are some important caveats. The impact estimates here are significantly less robust than those from the impact assessment of the Work Experience programme, on which I blogged here. It is just much more difficult finding "comparable non-participants" in the data. So there is a lot of uncertainty. It is certainly possible that the characteristics of those referred to the programme were worse, and hence the impacts, especially on benefit receipt, underestimated. But it is very difficult not to conclude that - whatever your position on the morality of mandatory work programmes like these - the costs of the programme, direct and indirect, are likely to far exceed the benefits. [Note: for what it's worth, my personal view is that if such programmes deliver clear and measurable benefits, either in terms of reducing benefit receipt amongst those who shouldn't be claiming, or through increased employment, they are justifiable; that does not appear to be the case here].

To conclude, it is highly commendable that the Department has undertaken and published this analysis. It would be even better if that hadn't been accompanied by a policy decision which seems to fly directly in the fact of the evidence. At at time of austerity, it is very difficult to see the justification for spending millions of pounds on a programme which isn't working.

7 comments:

One thing I think you're missing is the impact on disabled JSA claimants - who are more likely to have moved to claiming ESA due to MWA making their health worse. Goodness knows I would be in that position if I were there - fortunately for me I'm already on ESA, but even my DWP advisor says I should be support-group not WRAG.

"The hope is not so much that participants will be encouraged to get a "real" job, but that they will find the experience of mandatory work so unpleasant (and/or incompatible with other activities they'd rather be doing..) that they will sign off benefit one way or the other" <------ sums up exactly my problem with it.Im finding a lack of employment experience in my field of I.T is probably my biggest impediment, followed by my skills becoming rusty as a second.Apparantly a 4 week unpaid stint (probably in a cafe) is the way to achieve my career aims, and overcome those problems.

Is there any reason you did not use survival analysis for this study? Looking at the report, you seem concerned about conditional independence but your chosen method - analyzing proportions in work by week - induces a huge amount of serial dependence. It also doesn't adjust for the much more serious problem of differential follow-up times, i.e. for the right-censorship inherent in this data.

Furthermore, you seem to have tied yourself in knots trying to find ways to handle claimants' previous benefit history. Again, a basic Cox proportional hazards regression (survival analysis) with time-varying covariate would enable you to include all the information - and you wouldn't have to fiddle around with pseudo-referral dates for the non-referred, which seems like a very dubious way to handle an intervention analysis. You then also wouldn't need a complex propensity-score matching system.

Finally, you make a distinction between referrals and starts, but wouldn't this be simply handled by including an Intention-to-treat assumption in your analysis?

I've never seen a simple retrosepctive cohort study like this handled in the way this report does. It seems unnecessarily complicated and induces a huge amount of bias while ignoring the problem of right-censorship. Is there a reason you didn't use a standard survival analysis?

i) the word Choices with Mandatory work activity and, ii) self-selecting with adviser selecting and,iii) voluntary with at adviser discretion.

So in the absence of an ideally randomised control - can't really say if either scheme is worth the investment.

Taken from page DWP working paper 60 written by IFS in 2009, page 2:

Second and more fundamentally, participation in Choices is voluntary, so it is difficult to know how far different outcomes for participants and non-participants are caused by Choices and how far they reflect pre-existing differences in the type of people who choose to participate. Using propensity score matching techniques, we control for differences between participants and non-participants in a very large set of background characteristics; we thus compare outcomes for Choices participants with those for non-Choices participants who are observably similar in many dimensions. But it remains likely that there are important differences in the unobserved characteristics of the two groups, and it is impossible to know how far the difference in outcomes between the two groups is a result of these unobserved pre-existing differences rather than a result of participating in Choices.

For example, those who are more motivated and more ready to move into employment might be more likely to choose to participate in the voluntary programmes available as part of the Choices package, but they would also be more likely to move into paid work even without participating in Choices. If that is the case, the estimated differences in outcomes between participants and non-participants would be overestimates of the true impact of the programme.

Alternatively, individuals with worse health conditions might be more likely to volunteer for programmes aimed at improving their ability to manage their health problem, but might also be less likely to move into paid work even without participating in Choices. If this were the case then the estimated differences in outcomes between participants and non-participants would be underestimates of the true impact of the programme (and possibly suggest, incorrectly, that the programme had a negative impact on employment outcomes). Another possibility is that individuals assigned personal advisers who more strongly encourage people to enrol in Choices programmes might be either more or less likely to help them move into work in other ways.

Unless one is prepared to make the strong assumption that these unobserved characteristics do not explain both the outcome and self-selection into the programme, it is impossible to provide reliable causal estimates of the impact of the Choices programme. Hence, this study only presents a descriptive analysis of the difference in outcomes between individuals who chose to participate in these programmes and observably similar individuals who did not.

This paper stresses the intrinsic difficulty of evaluating programmes based on voluntary participation when there is no exogenous variation in the availability of the programme. By exogenous we mean no variation in programme participation that is not correlated with other characteristics not taken into account that are also associated with the outcomes of interest. Controlling for a rich set of observed characteristics is unlikely to overcome the fact that participants and non-participants to such programmes might be different for inherently unobserved characteristics, and these unobserved differences might be associated with better or worse subsequent outcomes. Exogenous variations can be used in various evaluation designs, like random eligibility thresholds, piloting based on geographical areas or even more robustly, randomisation at the individual level.

As soon as a potential employer realises that an 'up-to-date' reference was from a MWA placement position they will disregard it immediately. After all, they are going to think the poor sod on the placement was forced to slave away for nothing for four weeks, the least the 'employer' could do was give a reference!