Evaluation of CUREE: Achievement for All

Introduction

The project to be evaluated is a controlled trial of an intervention called Response to Intervention (RTI), which has shown some promise but with unclear results so far, especially for this age group in the UK. There is evidence of equipoise. RTI is a school-wide multi-tier programme that measures pupils’ response to research-based instruction. The model works in two ways. One is problem-solving by identifying the reasons for underachievement via a case-by-case analysis and tailoring instruction based on these reasons. Another way is the use of a standard treatment protocol which is administered to struggling pupils to prevent failure. RTI is delivered in three tiers of increasingly intense instruction. Here RTI is being offered to Year 6 pupils eligible from schools with high levels of free school meals, who do not achieve Level 4 English at Key Stage 2 (KS2). The treatment will take place in the last few weeks of the summer term 2013 at primary schools.

Impact evaluation

Design

The outline for the intervention proposes a relatively simple waiting list design. All schools will receive the RTI intervention, and all will have previously agreed to be randomised to immediate intervention or the intervention the next school year. Since the second group will be a fresh cohort it is possible to estimate long-term effect sizes for the intervention (although this is beyond the scope of the evaluation proposed here).

Sample size

The project involves school-level randomisation of 80 primary schools, using Year 6 pupils after their KS2 assessments. 40 schools will receive RTI immediately, and 40 schools a year later. A pseudo-random number generator to select the treatment or waiting group for each school, and the result will be revealed after the pre-test for both groups. Assuming an average of 10 eligible pupils per school, this provides around 400 pupils per arm of the trial.

Vaughn and Fletcher (2012) reported an effect size (d) for RTI of 0.16 after two tiers working with struggling pupils in grades 6 to 8. Using Lehr’s approximation for an 80% chance of detecting a presumed effect size of 0.2 with 5% alpha, the minimum sample size needed per arm is 400 individual cases (Gorard 2013). However, this trial involves randomizing schools rather than individual pupils. If we assume an average of 10 eligible pupils per primary school, 40 schools per arm with 10 pupils each will be 400 individuals. If we assume an ICC of 0.2, then the estimated design effect of school level allocation of these 400 individuals will be 2.8. This calculation suggests that something like 1,180 (2.2 times 400) would be needed. Of course, the ICC could be lower and the effect size higher, in reality. Also the correlation between the pre- and post-test scores for individuals will tend to be high which can make the effect somewhat easier to detect. Nevertheless, there is a danger that this trial will not have full power.

Tests

The pre-test scores for both groups will be the GL New Group Reading Test A, administered in classes. The test will be administered by the schools themselves in conjunction with the project team. Because this will take place before randomisation, the process will be ‘blind’ as to treatment group. In order for this sequence to be most practical, all Year 6 pupils in participating schools will take the pre test. The numbers for the pre-test are estimated at 4,000, or 50 per year group in 80 schools. The relevant pupils meeting the selection criteria will then be identified, and then randomised to treatment group or control. In addition, the Mid-May KS2 assessment results in literacy will provide a secondary pre-test score.

The post-test scores for both groups will be from the GL New Group Reading Test B and also the Progress in English Test, administered to those individual pupils identified earlier as meeting the selection criteria. The numbers for the post-test are estimated at 800, or 10 per year group in 80 schools. These tests will be administered by members of the evaluation team and their temporary employees (such as doctoral researchers) who will not know which group each school is in. Schools will be instructed not to disclose the fact to the evaluators. This is to help ensure that the process is ‘blind’ as to treatment group. In addition, the most up-to-date teachers assessment results in literacy can be used as a secondary post-test score.

Other data

The intervention team will prepare a template for data to be uploaded for all relevant pupils at the outset of the trial. The template includes prior attainment plus background characteristics such as FSM, sex and ethnicity. This is the data that will be uploaded from school SIMS or similar to the GL test system, to create a record of individual pupil background data plus pre-test scores. This will eventually be linked via UPN to the individual post-test scores.

Analysis

The primary outcome measure will be the difference in the gain score between the arms of the trial, expressed as an effect size, where the gain is the average difference between individual scores on tests A and B. A secondary outcome measure will be a post-test only comparison of both groups on the Progress in English Test. A further secondary outcome measure will be the average residuals between the actual scores on test B and the predicted (modelled) scores based on prior KS2 literacy assessment and pupil background data (such as FSM).

Process evaluation

The fieldwork forming the light-touch process evaluation has the aim of providing formative evidence on all aspects of the intervention from the selection and retention of schools, through the training of teachers to evaluating the outcomes. This can be used to help assess fidelity to treatment, and the perceptions of participants including any resentment or resistance, and to advise on improvements and issues for any future scaling up.

This will all necessitate the generation of some additional data from observation and interviews with staff, focus groups of pupils, plus observation of training, delivery and testing. These will all be as simple and integrated as possible. Participating schools will understand that being part of the project involves being part of the evaluation.

Results

The study has completed now. Education Endowment Foundation has published the reports on 7th February. For complete reports please click here.

References

Gorard, S. (2013) Research Design: Robust approaches for the social sciences, London:Sage