U.S. Review Finds No Proof That Reform Model Works

Study of ‘First Things First’ said to show ‘no discernible effects.’

Despite attracting high-powered backers
such as Bill Gates, the much-touted school
improvement program known as First
Things First has yet to muster conclusive scientific
evidence to show that it prevents students
from dropping out of school, a federal
research review concludes.

The First Things First evaluation was one
of two study reviews posted online Jan. 24 by
the What Works Clearinghouse, which the
U.S. Education Department set up to vet existing
research evidence on the effectiveness
of educational programs and practices.

They found “potentially positive” evidence
showing that the now-defunct model improved
high school completion rates for its
target group of young welfare mothers.

The First Things First review drew criticism,
though, from the program developers.

They contend that the federal reviewers set
too high a bar for evidence and left out studies
that gave the program high marks for improving
student achievement, attendance,
and graduation rates.

“I do not think it’s a fair and accurate representation
of the evidence,” said James P.
Connell, the architect of the program
and the founder of the Institute
for Research and Reform,
the Toms River, N.J.-based
group that houses it.

First Things First is now used
in 30 secondary schools in seven
states—California, Kansas, New
Jersey, Mississippi, Pennsylvania,
Texas, and Wisconsin—and
more than two dozen elementary
schools.

It is also on a short list of potentially
successful high school
redesign programs singled out by
Microsoft Corp.’s chairman, Mr. Gates, as early
as 2005.

Houston Evaluation

The high school-level program has three
pillars: small learning academies that each
keep students together from grades 9-12, a
“family advocate” system pairing teachers
with small groups of students for four years,
and an emphasis on improving instruction.
("‘First Things First’ Shows Promising Results," Mar. 9, 2005.)

Five studies have evaluated First Things
First so far. In its analysis, though, the clearinghouse
includes evidence from only one—an independent evaluation of three Houston
high schools that implemented the program
from 2001 to 2004.

Dropout Prevention

The What Works Clearinghouse dropout-prevention reviews focus on secondary school
and community-based interventions designed to help students stay in school and/or
complete school. These can include incentives, counseling, monitoring, school
restructuring, curriculum design, literacy support, or community-based services to
mitigate factors impeding progress in school.

SOURCE: Institute of Education Sciences

For that study, researchers compared academic
outcomes for the 9th graders who entered
the schools those years with those of their predecessors
three years earlier. The study did the
same calculations for 10 or 11 nonprogram
schools and compared the results.

Based on that evidence, the federal reviewers
concluded that the program showed “no
discernible effects” on keeping students in
school.

One problem with that conclusion, Mr.
Connell said, is that “it’s limited to data from
three schools in one city for one year—basically
from freshman to sophomore year—and
that’s really not a reasonable period of time
for summarizing evidence.”

But Robert Wood, a senior economist at
Mathematica Policy Research, the Princeton,
N.J., group that runs the clearinghouse, said
the review excluded evidence from the other
four sites—Kansas City, Kan.; Riverview
Gardens, Mo.; and Shaw and Greenville,
Miss.—for methodological reasons.

For instance, in Kansas City, where the program
was carried out districtwide, evaluators
failed to find well-matched comparison
schools.

The Mississippi study focused on measuring
different outcomes, he said, and the Missouri
evaluation only included one First Things First
high school and one comparison school.

“You basically can’t separate the school
from the program,”Mr.Wood added.

‘Hardest of Reforms’

The New Chance evaluation, likewise, rested
on one study—a 10-state evaluation in which
2,000 women were chosen by lottery either to
take part in the program or sit it out.

The participants took classes on parenting,
life skills, and preparing for their high school
equivalency exams.

When that phase ended, the women got assistance
with job training and placement and
access to child care.

The two new reports bring to
13 the number of dropout-prevention
programs that the clearinghouse
has evaluated so far.

Only two models—New Chance
and Talent Search, a federal program
that provides college and
career counseling to academically
talented youths from disadvantaged
families—were deemed to
have any evidence of success.

“High school reform is among
the hardest of reforms to
undertake,” noted Steven G.
Seleznow, the program director for education
at the Gates Foundation. Spurring
change may be even harder for First Things
First, he added, because the program works
with existing school staff members rather
than starting over with converted schools.

“In some places where we’re funding First
Things First, we are seeing some evidence of
success,” he said, “and in some places, we are
not seeing evidence of success as fast as we
would like.”

To get a better read on the program’s effectiveness,
the foundation is supporting a
comprehensive, independent evaluation of
the program, not yet complete.

In addition, the federal Institute of Education
Sciences this year launched a $6 million
randomized study of First Things First that
will involve 40 program sites around the country.

“I think I’m going to rely on the findings
that will come from more thorough and focused
research than a summary of other research
projects,” Mr. Seleznow said.

Vol. 27, Issue 22, Page 6

Published in Print: February 6, 2008, as U.S. Review Finds No Proof That Reform Model Works

Notice: We recently upgraded our comments. (Learn more here.) If you are logged in as a subscriber or registered user and already have a Display Name on edweek.org, you can post comments. If you do not already have a Display Name, please create one here.

Ground Rules for Posting
We encourage lively debate, but please be respectful of others. Profanity and personal attacks are prohibited. By commenting, you are agreeing to abide by our user agreement.
All comments are public.