A. Whether an evidence-based intervention will have a positive effect in
your schools or classrooms may depend critically on your adhering closely to
the details of its implementation.

The importance of adhering to the details of an evidence-based intervention
when implementing it in your schools or classrooms is often not fully appreciated.
Details of implementation can sometimes make a major difference in the intervention's
effects, as the following examples illustrate.

Example. The Tennessee Class-Size Experiment - a large, multi-site
randomized controlled trial involving 12,000 students - showed that a state
program that significantly reduced class size for public school students in
grades K-3 had positive effects on educational outcomes. For example, the average
student in the small classes scored higher on the Stanford Achievement Test
in reading and math than about 60 percent of the students in the regular-sized
classes, and this effect diminished only slightly at the fifth-grade follow-up.23

Based largely on these results, in 1996 the state of California launched a
much larger, state-wide class-size reduction effort for students in grades K-3.
But to implement this effort, California schools hired 25,000 new K-3 teachers,
many with low qualifications. Thus the proportion of fully-credentialed K-3
teachers fell in most California schools, with the largest drop (16 percent)
occurring in the schools serving the lowest-income students. By contrast, all
the teachers in the Tennessee study were fully qualified. This difference in
implementation may account for the fact that, according to preliminary comparison-group
data, class-size reduction in California may not be having as large an impact
as in Tennessee.24

Example. Three well-designed randomized controlled trials have
established the effectiveness of the Nurse-Family Partnership - a nurse visitation
program provided to low-income, mostly single women during pregnancy and their
children's infancy. One of these studies included a 15-year follow-up, which
found that the program reduced the children's arrests, convictions, number of
sexual partners, and alcohol use by 50-80 percent.25

Fidelity of implementation appears to be extremely important for this program.
Specifically, one of the randomized controlled trials of the program showed
that when the home visits are carried out by paraprofessionals rather than nurses
- holding all other details the same - the program is only marginally effective.
Furthermore, a number of other home visitation programs for low-income families,
designed for different purposes and using different protocols, have been shown
in randomized controlled trials to be ineffective.26

B. When implementing an evidence-based intervention, it may be important
to collect outcome data to check whether its effects in your schools differ
greatly from what the evidence predicts.

Collecting outcome data is important because it is always possible that slight
differences in implementation or setting between your schools or classrooms
and those in the studies could lead to substantially different outcomes. So,
for example, if you implement an evidence-based reading program in a particular
group of schools or classrooms, you may wish to identify a comparison group
of schools or classrooms, roughly matched in reading skills and demographic
characteristics, that is not using the program. Tracking reading test scores
for the two groups over time, while perhaps not fully meeting the guidelines
for "possible" evidence described above, may still give you a sense
of whether the program is having effects that are markedly different from what
the evidence predicts.