1. Introduction, Context and Study Method

This report presents the findings of the ex-post evaluation of
the Scotland Rural Development Programme (SRDP) 2007-2013.

The study was commissioned by the Scottish Government in
December 2015 and was undertaken by EKOS Limited in collaboration
with the Rural Development Company, P&L Cook and Partners, and
Prof. Bill Slee.

1.1 Context and Study Objectives

The European Commission requires Member States to evaluate the
assistance from the European Agricultural Fund for Rural
Development (EAFRD) for the programming period 2007-2013.

The programming period incorporated a number of features, such
as a new EC Common Monitoring and Evaluation Framework (CMEF) which
included the intervention logic for each Programme Measure in
relevant Measure Fiches, and an Ongoing Evaluation approach.

The ex-post evaluation was guided by the guidelines for 'Ex Post
Evaluations of Rural Development Programmes'
[1].

As described in the 'EC Guidelines for Ex Post Evaluations':

Ex Post Evaluation is a summative evaluation of a Rural
Development Programme after it has been completed. It is
conducted at a point where it is possible to assess impacts and
the added value of the programme funding. ….Demonstration
of the policy achievements, thus legitimising funding for rural
development measures, is important at European, national and
regional levels, especially when budgets are tight. The Ex Post
Evaluation also provides the opportunity to see whether the
policy was designed and implemented appropriately to address the
most relevant needs in the programme area.

The key objectives of the ex-post evaluation were to assess a
number of factors, including the:

relevance in terms of addressing the most important needs in
the programme area;

effectiveness and achievements towards policy
objectives;

efficiency in terms of receiving best value for money;

results in terms of programme achievements within the group
of direct programme beneficiaries;

socio-economic impacts in terms of programme contributions to
the change observed in the programme area; and

success and failure factors and lessons learned for the
future policy design.

Although the evaluation guidelines include an assessment at
Measure level, the overall impact of the SRDP is assessed at Axis
and Programme level.

In combination with the ex-post evaluation guidance, the CMEF
system provided the basis for the ex-post research design and
assessment criteria.

1.2 Study Method

The study used a mixed-method approach drawing on secondary
research as well as undertaking primary research. The following key
stages were accomplished:

Stage 1 - Inception: The study commenced with an
Inception meeting with the client at which the study scope and
detailed method were agreed. A short Inception Report was produced
and approved by the client.

Stage 2 - Desk-based Research: A number of
comprehensive desk-based reviews of available monitoring
information, research studies, mid-term evaluation (MTE), and
ex-ante workshop findings have been undertaken. Key findings were
presented in the first Interim Report in March 2016.

Stage 3 - Primary Research: The primary research
phase incorporated a comprehensive programme of activities. It was
designed to consult with a range of stakeholders and beneficiary
groups throughout Scotland as well as a range of Scottish
Government officials, Scheme Managers and the Scottish National
Rural Network (SNRN).

A top-up survey of beneficiaries who could not attend the
focus groups.

At the end of Stage 3, a second Interim Report was produced
outlining the key headline findings of the fieldwork.

As the primary research was conducted over the summer period, it
was agreed to conduct the five thematic focus groups by the end of
August delaying the original timeframe of the evaluation by two
months.

Stage 4 - Analysis and Reporting: Following the
completion of the primary research, all desk-based and primary
research findings were analysed and the Topic Guides for each Axis
completed. Further analysis was conducted at the programme level
and a detailed impact assessment was undertaken whereby findings
from previous annual surveys were utilised. Stage 4 assessed the
socio-economic impact of the programme, and the effectiveness and
efficiency with which the SRDP was implemented.

Following the completion and submission of the draft report, we
will hold a learning workshop with the client to reflect on the
findings and draft recommendations following which we will amend
the report and produce the Final Report to the exact specifications
of the client.

1.3 Study Issues

A small number of study issues were experienced, mainly
influencing the depth to which the ex-post evaluation could analyse
data and beneficiary experience:

the ex-post evaluation was commissioned on a limited budget
restricting most beneficiary research to utilising findings from
previous evaluation surveys (with the exception of one postal
survey focusing on Measure 321, and five workshops with a limited
number of beneficiaries participating).

SRDP monitoring data were made available to the study team on
an aggregate basis. Without being able to interrogate and
analysis the raw monitoring data sets, the evaluation team was
unable to analyse the data in detail (type of beneficiary, size
of projects, type of projects).

The above issues placed limitations on the extent to which a
number of the Evaluation Questions and Common Evaluation Questions
posed by the 'Ex Post Evaluation Guidelines' could be answered.

The relatively poor quality of the performance indicator targets
and/or missing baselines further impinged on the delivery of a
decisive ex-post assessment as the extent to which the SRDP was
effective in achieving its targets was difficult to assess.