A blog to provoke debate and provide ideas about how to shape policing and the criminal justice system so that we all live in communities free from the fear of crime.

This blog is mainly about the governance and future of policing and crime services. (Police & Crime Commissioners feature quite a lot.) But there are also posts about the wider justice system. And because I am town councillor and political activist, local & national issues are covered a little, as well.

The Moj used Section 22 of the Freedom of Information Act to not answer two of my questions. They say that the information I was seeking is about to be published on July 25. I am prepared to wait until then before deciding to appeal their reply or not.

I asked why they published the results early, they said they did this "to ensure the information was made public as soon as it was available" despite admitting in other answers that it was incomplete. I suspect they wanted to get in some 'good news' before the summer vacation and in advance of CSR negotiations. But in my view, it looks shoddy and is evidence of using data and stats for political purposes. However, I guess all governments do that... don't they?

Their answers do seem to assert that they have sought to compare like with like in terms of cohort comparisons.

Their answer to question 4 does evidence the fact that this data is incomplete and premature, in my view

They originally said that a key difference between the cohort is that in this group "reconvictions only count offences for which the offender was convicted at court, whereas the National Statistics proven re-offending measure also includes out of court disposals (cautions)”. I asked what the impact of that difference was likely to be. They referred me to "Table B3 of annex B from the MoJ’s proven re-offending statistics quarterly bulletin: https://www.gov.uk/government/publications/proven-re-offending--2". I have looked at this table and it is not entirely clear so I think I am going to have go back to them on this and seek further clarification. But do note that they said "We have not produced alternative interim figures on what the impact would be if different rules (such as including cautions) had applied". Which seems a bit sloppy to me. This is a critical difference after all and I suspect that if the data was showing not in favour of the pilot providers, they would be seeking further clarification!

I asked whether the comparison groups (to evidence that the pilot intervention was in fact working) were selected using some kind of randomised selection. They said "The control group will be selected by an Independent Assessor using Propensity Score Matching (PSM), the methodology for which has been published at: Peterborough Social Impact Bond: an independent ... - Gov.uk" So the answer is 'NO': comparator groups will be selected by 'independent' assessor (being paid by the government, I assume). I looked the reference document and here is a quote from it: "It should be noted that, unlike random control allocation, PSM cannot take account of unmeasured differences which may account for variation in reconviction aside from ‘treatment received’" Uh huh. But it goes onto assert that: "However, PSM [propensity score matching] is widely regarded as one of the best ways of matching quasi-experimentally (Rosenbaum, 2002), and it has been increasingly used in a criminological context (e.g. Wermink et al., 2010)." So that is alright then. Excuse me while I give you this new medicine that has been quasi-experimentally tested on people who are sort of similar to you...

I asked "For Doncaster, success “will be determined by comparison with the reconviction rate in the baseline year of 2009”. How will this accommodate national and/or local trends in (say) sentencing practice or levels of crime?". They replied "The five percentage point reduction target was agreed after analysis of historic reconviction rates established that this would illustrate a demonstrable difference which could be attributed to the new system and not just natural variation." That is not an answer to my question, I will need to go back to them on this.

I asked about the 6 versus 12 month comparison and how the headline data (based on six months) was going to look against the usual (12 month) data. They said in reply "The statistical notice made clear the limitations of the information presented and the care that should be taken in interpreting these interim figures." Remind me - was that subtlety in the press releases that went out when this interim data was released...?

They missed the point completely on my question about seasonality...

Please read their answer to my question about why 19 month data. Please let me know what you think. I am thinking 'wool', 'eyes' and 'what do you really mean?!'

The maths question is funny. They said "The figures presented were the rounded versions of the actual figures, which were 68.53 and 79.29". So I have done the calculations again and the result I get this time is 15.7. So they are sort of correct - but why leave out the first decimal point?

I asked about statistical significance (the test of whether a difference is just a chance difference or one that indicates a real effect is in play). This is what they said "We have not carried out statistical significance tests on the interim figures because, when it comes to the final results, neither pilot will be assessed on the basis of whether they have achieved a statistically significant change."

OK. Let me repeat that again in big and bold:

We have not carried out statistical significance tests on the interim figures because, when it comes to the final results, neither pilot will be assessed on the basis of whether they have achieved a statistically significant change.

So, Payment by Results could well be based upon purely random chance events that may just have happened.

No comments:

Post a Comment

Note for spammers

Please... do not bother to leaving comments on any of these blogs with statements such as "wow what an amazing blog, do visit my website on >insert product herewill delete them. These comments will not appear. Your and my time will have been wasted. Thank you.

Jon Harvey

jonsharvey.net
Change facilitator and organisational development practitioner with 25 years experience across the public services and commercial world.
Also a Town Councillor in Buckingham - working with and for local people to make the town even healthier, safer and wealthier for everyone who lives, visits, studies and plays in the place
My passions & expertise centre on:
- Designing and facilitating ‘big’ meetings / whole system processes to shape the future & handle the present
- Coaching & helping people to create robust strategies and lead change & innovation
- Creating away-days where lunch is not the highlight - And questioning - where I have a reputation for challenging people and organisations in ways that makes them ‘wriggle but not squirm’
+44 (0) 1280 822585
Also tweets as @JonSHarvey on wider leadership & organisation development matters

Articles published elsewhere

Please click here for a list of all my articles published elsewhere. My most recent article is:

Staying in touch

Followers

Subscribe To

Follow by Email

The position of PCC

This is a new position and has replaced Police Authorities around the country.

The PCC will set the overall priorities of the police service through creating a local plan, set the annual budget for Thames Valley Police and the police part of the council tax (precept) in consultation with the Chief Constable. The PCC will also have the power to appoint a new Chief Constable and will be able to dismiss one as well.