Friday, April 24, 2015

Thank you for sharing
your most recent report using the 2010 Illinois test score data on the eight
Chicago public schools in which Strategic Learning Initiatives has been
working.

This latest report builds
on the data that served as the basis for our report “Validating the Impact of
Strategic Learning Initiatives’ Focused Instruction Process (FIP Model)” (Steven
Leinwand and Sarah Edwards, American Institutes for Research, July, 2009).

In the 2009 report, we
concluded that “the results that SLI has achieved, and that AIR has validated,
are very impressive and suggest that well before decisions are made to
reconstitute schools under the mandates of NCLB, school districts would be wise
to consider far less drastic, but clearly powerful, interventions such as the
Focused Instruction Process.”

The updated 2010 ISAT data
that you have provided show that the schools supported by Strategic Learning
continue to close achievement gaps and accelerate student learning.SLI continues to make a real difference in the
lives of students and it is clear that its successes can, and should be, broadly
adopted elsewhere.

“American
Institutes for Research” is a registered trademark. All other brand, product,
or company names are trademarks or registered trademarks of their respective
owners.

1000 THOMAS JEFFERSON ST,
NW|WASHINGTON, DC20007-3835|

TEL 202 342 5000|FAX 202 342 5020|WEBSITE WWW.AIR.ORG

Introduction

This
report arises from discussions between John Simmons, President of Strategic
Learning Initiatives (SLI) and the American Institutes for Research (AIR) to
provide an external validation of the impact data in ten Chicago Public Schools
elementary schools that participated in SLI’s Focused Instruction Process
between 2006 and 2008.

AIR
agreed to complete two tasks:

·Rerun and
validate the ISAT-Reading data for the 10 FIP schools; and

·Compare the ISAT
scores between the 10 FIP schools and a matching set of 10 non-FIP schools.

The results
that SLI has achieved, and that AIR has validated, are very impressive and
suggest that well before decisions are made to reconstitute schools under the
mandates of NCLB, school districts would be wise to consider far less drastic,
but clearly powerful, interventions such as the Focused Instruction Process.

Findings

1.How well did the 10 FIP schools do during the first
two years of intervention when compared with the Chicago city average?

Figure 1
shows that for the two-year period from 2006 to 2008 all but two of the 10 FIP
schools had ISAT Reading gains in the percent of students at or above
proficient that exceeded the Chicago city average and that as a group, the
gains in the FIP schools were nearly twice the city average (11.4 percentage
points vs. 6.3 percentage points).

Figure 1

Two-Year
Gains in Percent of Students Meeting/Exceeding Reading Proficiency for 10 FIP Schools During
FIP (2006-08)

School

Gains

Cather

16.1

Faraday

14.2

Goldblatt

10.8

Morton

2.1

Tilton

14.2

Cardenas

18.7

Castellanos

6.0

Finkl

15.3

Gary

8.3

Kanoon

8.5

10 FIP Schools

11.4

ChicagoCity
Average

6.3

Figure 2
shows the annual average gains in the FIP schools during the two-year period
(2006-2008) during which the intervention was in place when compared with the
annual average gains for the four-year period (2001-2005) prior to the
intervention. Once again, the gains made during the period of the intervention
are impressive relative to the gains prior to the invention and relative to
non-FIP schools.

Figure 2

Annual
Average Gains in Percent of Students Meeting/Exceeding Reading Proficiency Before (2001-2005) and During
(2006-2008) the Intervention

Before FIP (2001-2005)

During FIP (2006-2008)

5 Area
7 FIP schools

0.8

5.7

5 Area
10 FIP schools

3.6

5.7

10 FIP
schools

2.2

5.7

Non-FIP
schools

2.1

3.2

2.How well did the 10 FIP schools do when compared with
a set of matching schools?

AIR asked SLI to identity a set of ten matching
schools to the ten FIP schools. SLI used the following process to make these
matches:

1.Data from publicly available data bases from ChicagoPublic Schools
and Illinois State Department of Education was extracted for the 10 FIP schools
and all other elementary schools in Chicago,
Illinois.This data included:

a.Percent low income as indicated by participation in
free or reduced price lunch program.

b.Percent of enrolled students who were identified as
being in any of six specific racial/ethnic groups:White, Black, American Indian, Asian,
Hispanic, Multi-Racial.

c.ISAT Percent Meeting or Exceeding Proficiency in Reading for grade level in
the years 2001-2008.It should be noted
that standards of proficiency were altered between academic years ending in
2005 and 2006.

d.From the ISAT data, Average Annual Gains/Losses in
Percent Proficient were calculated for the 2001 to 2005 (4 year) difference and
separately for the 2006-2008 (2 year) period.

2.A computer program was written to initially screen for
matches to the 10 FIP schools.The
program ignored a criterion that was originally considered:No consideration was given to whether the
schools matched were in the same city neighborhood (“Area”).

3.The program initially searched for schools that had a
Low Income Percent within 1.0 percent of the Low Income Percent of the FIP
school to which they were to be matched.In some cases, this resulted in short list of schools which would be
difficult to match to the FIP school using the additional criteria to be
applied.Accordingly, a second run of
the program was performed to select schools that would match within 2.0 percent
of the Low Income Percent of the FIP schools.This resulted in a longer list.In subsequent matching, preference was given to schools that appeared on
the “within 1.0 percent” match list.

4.Using the schools that matched on the Low Income
criteria, a statistic designated as “stress” was calculated consisting of the
sum of squared differences in racial/ethnic percentages between the FIP school
and each potential matching school.

5.The list of matching schools for each of FIP schools
was sorted low to high on this statistic.The lower the “stress” statistic, the closer the match on racial/ethnic
composition.

6.Further matching was performed manually with the
following criteria:

a.Percent Meeting or Exceeding Proficiency in 2005 was
roughly equal in FIP school and matched school.This criterion was applied because improvement due to interventions will
depend upon starting point prior to the intervention.

b.Average Annual Percent Gain/Loss for the period 2001 to
2005, prior to FIP intervention, was roughly equal.

7.Finally, some schools which would otherwise be
considered as possible matches were excluded because they had participated in
other Strategic Learning Initiatives programs.

Figure 3 shows the results of this matched pair
analysis of FIP vs. non-FIP schools prior to and during the intervention.As would be expected given the selection
process, before the intervention there is a very small difference in the
average annual increase in the percent of students meeting or exceeding reading
proficiency between the FIP and non-FIP schools (2.24 percentage points vs.
2.45 percentage points respectively for the period 2001-2005).However, during the intervention there is a
significant difference in the average annual increase – 5.71 percentage points
for the FIP schools vs. 1.14 percentage points for the non-FIP schools – for
the period 2006-2008. (See Appendix,
Tables 1, 2, and 3 for summary statistics and p-values for the t tests.)Before the intervention, the mean difference
in the average annual increase of percent of students meeting or exceeding
reading proficiency between matched FIP and non-FIP schools was -0.21.The mean difference in average annual
increase between matched FIP and non-FIP schools during the intervention (4.57)
was significantly greater.(See
Appendix, Tables 4 and 5 for summary statistics and p-value for the t-test.)

Figure 3

Average Annual Increases and Differences in Test Scores

for Matched Pair FIP and Non-FIP Schools Before and During FIP

Pair

FIPSchool

Non-FIPSchool

FIPSchool: Average Annual
Increase Before FIP (2001-2005)

Non-FIPSchool: Average Annual
Increase Before FIP (2001-2005)

FIPSchool: Average Annual
Increase During FIP (2006-2008)

Non-FIPSchool: Average Annual
Increase During FIP (2006-2008)

Difference
in Average Annual Increase in Matched FIP and Non-FIPSchool
Before FIP

Difference
in Average Annual Increase in Matched FIP and Non-FIPSchool
During FIP

1

CATHER

DEPRIEST

2.78

2.90

8.05

2.40

-0.13

5.65

2

FARADAY

O'TOOLE

1.25

2.23

7.10

-1.40

-0.98

8.50

3

GOLDBLATT

HERZL

-0.13

0.45

5.40

3.70

-0.58

1.70

4

MORTON

O'KEEFFE

0.10

0.05

1.05

0.25

0.05

0.80

5

TILTON

OWENS

0.20

-0.93

7.10

1.65

1.13

5.45

6

CARDENAS

LARA

2.78

2.45

9.35

2.50

0.33

6.85

7

CASTELLANOS

COOPER

7.30

3.65

3.00

-0.55

3.65

3.55

8

FINKL

STOWE

3.15

4.33

7.65

2.10

-1.18

5.55

9

GARY

MCCORMICK

2.90

4.55

4.15

-2.15

-1.65

6.30

10

KANOON

WHITNEY

2.08

4.85

4.25

2.90

-2.78

1.35

All FIP schools

Matched non-FIP schools

2.24

2.45

5.71

1.14

-0.21

4.57

Figure 4
shows the average change in the percent of students meeting or exceeding
reading proficiency between each year in the 10 FIP schools and the 10 matched
non-FIP schools for the entire 2001-2008 period and reveals graphically the
apparent impact of the FIP intervention.

Figure 4

Note: The test format changed significantly from
2005 to 2006 so the 2005 to 2006 test score change is not included

Conclusion

It is
clear, on the basis of the ISAT Reading scores for the percent of students
meeting or exceeding proficiency in ten Chicago elementary schools for the
period 2001-2008, that the Focused Instruction Process intervention has had a
positive and significant impact on student achievement in the cohort of ten
schools that participated in the FIP model from 2006 to 2008.Whether compared to pre-intervention
achievement, or to the entire set of Chicago
elementary schools, or to a carefully selected set of matched schools, the data
suggest that FIP has resulted in gains that are very unlikely to have occurred without
the intervention.

APPENDIX

Table 1:
Mean Average Annual Increase in Scores Before and During FIP

for 10 FIP Schools

Variable

N

Mean

Std
Dev

Minimum

Maximum

Average annual increase

before FIP (2001-2005)

10

2.24

2.18

-0.13

7.30

Average annual increase

during FIP (2006-2008)

10

5.71

2.58

1.05

9.35

Table 2:
Mean Average Annual Increase in Scores Before and During FIP

for 10
Matched Non-FIP Schools

Variable

N

Mean

Std
Dev

Minimum

Maximum

Average annual increase

before FIP (2001-2005)

10

2.45

2.01

-0.93

4.85

Average annual increase

during FIP (2006-2008)

10

1.14

1.98

-2.15

3.70

Table 3:
t-test for Mean Average Annual Increase in Scores Before and During FIP

for 10 FIPSchools
and 10 Matched Non-FIP Schools

Statistics

Variable

FIP

N

Lower
CL

Mean

Upper
CL

Lower
CL

Std Dev

Upper
CL

Std Err

Minimum

Maximum

Mean

Mean

Std
Dev

Std
Dev

beforeave

0

10

1.01

2.45

3.89

1.39

2.01

3.68

0.64

-0.93

4.85

beforeave

1

10

0.68

2.24

3.80

1.50

2.18

3.98

0.69

-0.13

7.30

beforeave

Diff (1-2)

-1.76

0.21

2.18

1.59

2.10

3.10

0.94

afterave

0

10

-0.28

1.14

2.55

1.36

1.98

3.61

0.63

-2.15

3.70

afterave

1

10

3.86

5.71

7.56

1.78

2.58

4.72

0.82

1.05

9.35

afterave

Diff (1-2)

-6.73

-4.57

-2.41

1.74

2.30

3.40

1.03

T-Tests

Variable

Method

Variances

DF

t Value

Pr > |t|

beforeave

Pooled

Equal

18

0.23

0.8235

afterave

Pooled

Equal

18

-4.44

0.0003

Table 4:
Mean Difference in Average Annual Increase

in
Matched Pair FIP and Non-FIP Schools Before and During FIP

Variable

N

Mean

Std Dev

Minimum

Maximum

Difference in Average Annual Increase in Matched
FIP and Non-FIPSchool Before FIP

10

-0.21

1.74

-2.78

3.65

Difference in Average Annual Increase in Matched
FIP and Non-FIPSchool During FIP

About SLI

The Mission of Strategic Learning Initiatives is to work with schools to dramatically improve classroom practice and district results.

The key to our success in our hometown, Chicago, is that we use the power of research as the guide to improving results in the classroom. Chicago is home to the best long-term research in the nation revealing what works—and what doesn’t—in education. We bring the best of that research and the best research from the world of business that tells us about total quality improvement to each of the schools where we work.

We know the qualities that must be present in a school in order for it to succeed. We work with our schools, principals, teachers and parents to ensure those qualities are present.

In schools where Strategic Learning Initiatives is at work, teachers teach, re-teach, and teach again until every student achieves mastery and the entire school moves forward.

Contact us for more information: 312-738-0022 ext. 1111 Educational Program Asst.