Contents

Moving Forward: Making Changes - An Evaluation of a Group-based
Treatment Programme for Sex Offenders

This paper summarises the key findings and policy
messages from an evaluation of Moving Forward: Making Changes, an
intensive group-based treatment programme for sex
offenders.

Key Findings

The report provides evidence that most men who completed the
programme (69%) were rated as having a lower risk score by the
end of the programme. Among the men rated as ‘high
risk’ at the start of
MF:
MC, 53%
remained high risk at the end of the programme, 38% had moved
into the ‘moderate’ category and 9% were rated low
risk. Among those initially rated as ‘moderate’ risk,
72% remained in this category, but 26% moved into the ‘low
risk’ category, while a very a small number (3%) were rated
‘high risk’ at the end of the programme. However,
there are serious question-marks over the reliability of some of
the risk data and without a control group we cannot be certain
that these shifts in risk scores were caused by
MF:
MC or by
other factors.

Alongside reduced assessed risk levels, other perceived
benefits identified by staff and men interviewed for this
evaluation included improved ability to sustain healthy social
relationships, regulate their emotions, cooperate with
supervision, and understand and change problematic attitudes. 85%
of men who completed an exit survey on leaving
MF:
MC said
they thought it would stop them reoffending in a similar
manner.

Overall, the programme is being delivered broadly as intended
with respect to length of group sessions, group size, staff
roles, staff participation in mandatory
MF:
MC
training, and practitioner supervision and support. However, in
the community, the programme is typically only being delivered
once a week rather than twice as recommended in the manual. This
was primarily attributed to a lack of staff resourcing.

The main areas identified for improvement were: staff
resources; access to psychological support; the clarity of some
aspects of the
MFMC
manuals; processes for assessing programme suitability; the
programme’s ability to meet the needs of all types of men
who commit sexual offences; and data collection and use.

At least 911 men were recorded as starting
MF:
MC (81%
in the community) although this is likely to be an under-estimate
due to data collection issues. Participants generally met the
target group criteria, with 98% assessed (at the start) as
moderate or high risk of future sexual offending
. However
, some concerns were raised over whether the
current assessment process is always identifying those most
suited to the programme.

Background

MF:
MC was
designed by the Scottish Prison Service and the Scottish
Government’s Community Justice Operational Practice Unit . It
was introduced in 2014, following accreditation by the Scottish
Advisory Panel on Offender Rehabilitation (
SAPOR).
It is aimed at adult (18 and over) male sexual offenders assessed
as medium-high risk of re-offending.
MF:
MC is
delivered in 4 prisons and in the community (currently 11 sites).
MF:
MC’s
implementation and delivery is supported by a set of manuals,
covering the programme theory, structure, content, management
requirements, and framework for collecting evaluation data.

Evaluation aims

The Scottish Government commissioned Ipsos MORI Scotland to
conduct an evaluation of
MF:
MC to
inform considerations on
SAPOR’s
reaccreditation of the programme in 2018. The evaluation was
intended to address three main questions:

1. How does the programme work in practice, from referral to
exit, and to what extent does the programme follow the manuals?
Is programme integrity maintained?
2. Are there difficulties with delivering any aspects of
the programme and are there improvements that could be made?
3. What are the main outcome measures and to what extent
have outcomes been realised?

Methods

The evaluation used a mixed method approach to address the
questions above. Data routinely entered into the
MF:
MCIT system was
analysed. In addition, qualitative interviews were conducted with:
19 participants across 5 case study sites (2 custody and 3
community); 18 Treatment Managers across the 15 sites; 4 Case
Managers; 3 stakeholders from Police Scotland; and 1 psychologist
working with community-based
MF:
MC teams.
The research also drew on evidence from a half-day workshop with 23
Practitioners from 12
MF:
MC
sites.

Main findings

Programme outcomes and improving impact

There are significant limitations to the data available to
quantify the impact of
MF:
MC,
including the lack of a control group (recruiting a control group
was not feasible within the evaluation timescale and resources,
and would also have raised considerable ethical challenges),
perceived issues around inter-rater reliability, concerns around
the validity of some measures, and high levels of missing
data.

However, the monitoring data does indicate a number of
positive changes among participants (though these cannot be
conclusively attributed to
MF:
MC),
including:

A reduction in risk scores over the course of the
programme, and

Positive changes in scores measuring a range of
psychological traits potentially associated with risk of
re-offending.

Overall staff felt that
MF:
MC had a
‘better chance’ of being effective compared with
previous sex offender interventions, as they viewed its design as
being more evidence-based. While this evaluation cannot robustly
assess whether all elements of
MF:
MC are
best practice, in general the programme design appears to be
informed by the current best evidence on ‘what works’
in terms of appropriate treatment targets and approaches to
working with sex offenders.

While staff and stakeholders gave examples where they
believed
MF:
MC had
impacted positively on participants, there was also a strong
feeling among staff that it was very difficult for the current
programme to fully meet the diverse presenting needs of all
participants.

Interviews for this evaluation (particularly with Treatment
Managers) identified a range of suggestions for improvements
aimed at enhancing
MF:
MC’s
impact, including:

providing more external expertise to support delivery

assigning national leads on specific approaches and issues
to ensure the programme is kept continually up to date in terms
of best practice

providing guidance on ‘over-treatment’ (where
offenders have attended programmes numerous times with limited
evidence of progress), and

considering the feasibility of establishing separate groups
for men with lower cognitive functioning.

Improving programme delivery

Overall,
MF:
MC is
being delivered broadly in line with programme design with
respect to: length of group sessions; group size; staff roles;
staff participation in mandatory
MF:
MC
training; and Practitioner supervision and support.

However, the evaluation identifies a number of concerns
around staff resourcing. Most community teams are currently
unable to deliver
MF:
MC at
the recommended intensity due to staffing constraints. Both
Community and Prison-based Treatment Managers identified
significant challenges covering leave or finding time for staff
development. The role of psychological support for delivery
within the community, and any inconsistencies in accessing it,
also requires clairfication.

The Programme Manual was viewed by staff involved in delivery
as ‘a little vague’. There is also a lack of clarity
about the level of deviation from the manuals that is
permissible. In practice, sites were adapting content and
delivery in a range of ways. While these adaptations were aimed
at improving delivery, they introduce scope for inadvertent
deviation from the programme design. The
MF:
MC
manuals should be reviewed, taking account of the various issues
and suggestions for improvement identified in the evaluation
report.

Consideration should also be given to developing an
MF:
MC
‘knowledge hub’, where additional (approved)
materials to support delivery can be shared and added to as new
evidence on ‘what works’ in treating sex offenders
emerges.

While
MF:
MC
appears, for the most part, to be successfully targeting men
rated as ‘high’ or ‘moderate’ risk, some
concerns were raised over whether the assessment process is
always identifying those most suited to the programme. Guidance
on assessing internet offenders and deniers, in particular,
should be updated in light of emerging evidence on managing and
treating these groups.

Sites varied in whether they offered ongoing support to
participants after the group stage of the programme. This
suggests that guidance on ending the programme should be clear,
and resourcing sufficient to ensure a consistent and appropriate
approach to post-programme support.

Improving monitoring and future evaluation

The monitoring data currently available for
MF:
MC has
some significant weaknesses in terms of the robustness of the
outcome measures included, the
IT
system’s perceived usability for recording key data, and
the timeliness of data collection and entry. Suggestions for
improvement include:

A systematic review of the outcome measures included within
MF:
MC,
taking account of concerns raised in this evaluation about: the
utility and appropriateness of the current psychometric
battery; the reliance on self-reported data; and completion of
the ‘significant others’ questionnaire.

Considering whether any further guidance and training is
required around the completion of SA07 for
MF:
MC
specifically, given the concerns about inter-rater reliability
and accuracy raised by interviewees.

Reviewing the structure and content of the
MF:
MCIT system
with a view to simplifying data entry, clarifying data outputs,
and enhancing data usability for both evaluators and
MF:
MC
teams.

Any changes resulting from this review process should be
supported by refreshed guidance and training on how monitoring
data should be entered and used by
MF:
MC
teams, and by the Scottish Government,
SPS and any
external evaluators.

In addition to reviewing and improving the current monitoring
dataset, there should be consideration of how to monitor
longer-term outcomes and whether a control group can be
established. As there is a relative dearth of evidence on the
effectiveness of sex offender interventions, there is an argument
for assessing the feasibility of an experimental or
quasi-experimental evaluation of
MF:
MC.
However, an experimental approach would require sufficently large
sample sizes to be robust and men would have to be randomly
assigned into the programme or into a control group.
Alternatively, a large matched sample of offenders who were not
on the programme would have to be identified. Both approaches
would be challenging to implement in practice.

Conclusions

Overall, the evaluation indicates that
MF:
MC is
being delivered with fidelity to the Good Lives Model on which it
is based. Sessions are focused on helping participants re-evaluate
their life goals and develop skills to achieve those goals without
harming others. It also appears to be delivered in an engaging and
responsive manner, with staff tailoring the content where necessary
to address individual needs. Acknowledging the concerns over the
reliabiity of the risk data and the limitations in being able to
quantify the impact of
MF:
MC, the
available data is generally quite positive in terms of perceived
impact. Overall, level of risk as scored by Stable 2007 decreased
for a majority of men over the course of the programme, while a
significant proportion of men also saw positive changes in scores
measuring psychological traits potentially associated with risk of
reoffending. Men expressed positive views about the
programme’s impact, and staff and Case Managers reported that
feedback from others and their own observations indicated it was
having a positive impact across a range of risk factors. However,
the views of some stakeholders struck a slightly more sceptical
note, reinforcing the need to reconsider what data is collected on
outcomes going forward.

Suggestions for improvement to maximise the potential impact of
MF:
MC and to
assess this more effectively in the future centre around: programme
scope and resourcing – including the expertise required to
deliver
MF:
MC, the
overall level of resourcing, and whether there is a need to
supplement
MF:
MC with
alternative or additional input for groups of men whose needs are
not currently perceived to be fully met; improvements to the
clarity and content of programme manuals; and improvements to the
validity, quality and relevance of monitoring and evaluation.