Accountability Studies Find Mixed Impact on Achievement

How effective is accountability in raising student achievement? The
evidence is mixed, according to a set of research papers presented here
last week.

"Most of the evidence is unpublished at this point," noted Eric A.
Hanushek, a senior fellow at the Hoover Institution of Stanford
University, and the answers that exist are "partial" at best.

The dozen papers were presented at a conference called "Taking
Account of Accountability: Assessing Politics and Policy," sponsored by
the program on education policy and governance and the Taubman Center
for State and Local Government, both located at the John F. Kennedy
School of Government at Harvard University.

A review of the existing research, by Mr. Hanushek and Margaret E.
Raymond, a research fellow at the Hoover Institution, found that
schools do respond to accountability systems, but not always in the
ways intended. For instance, several studies indicate that the number
of students excluded from testing, particularly special education
students, tends to rise as new accountability systems are
introduced.

At the same time, an analysis by the two researchers suggests that
state accountability systems may boost student learning. They found
that between 1992 and 2000, states with accountability systems
experienced, on average, significantly higher growth between 4th and
8th grade on math scores on the National Assessment of Educational
Progress than did states without such systems. Based on those results,
Mr. Hanushek said, it appears accountability "really matters."

A study by Thomas S. Dee, an assistant professor of economics at
Swarthmore College in Pennsylvania, similarly found what he called "an
interesting set of mixed results."

Mr. Dee used data from individuals who completed the long form of
the 1990 U.S. Census to examine the effects of two high school
graduation requirements—minimum- competency tests and the
completion of a core set of academic courses—on people who were
age 18 between 1980 and 1988 and born in the United States.

He found that the introduction of such standards significantly
reduced the likelihood that black males, in particular, would graduate
from high school, but increased the likelihood that they would be
employed.

Meanwhile, two papers on test- based accountability in Chicago drew
opposite conclusions.

Anthony S. Bryk, a professor of urban education at the University of
Chicago, examined elementary students' learning gains on the Iowa Tests
of Basic Skills from 1994 through 2001. He found that during the first
half of the 1990s, when Chicago was focused on decentralizing the
school district, students gained, on average, nearly a year of learning
in reading and mathematics for every year of instruction. But as the
high-stakes accountability initiatives rolled out— including
mandatory summer school and possible grade retention for low
achievers—the learning gains flattened.

Dueling Results

In contrast, Brian A. Jacob, an assistant professor of public policy
at the Kennedy School of Government, focused on ITBS scores for
students who took the tests in 1994, before the accountability measures
were introduced, and in 1998, after the changes were in place. He found
that students scored significantly higher on the ITBS following the
introduction of high-stakes testing than their peers did prior to the
new accountability policies.

While gains in math scores came disproportionately in the areas of
computation and number concepts, or basic skills, the improvements in
reading were spread relatively evenly across item types. Moreover,
students made the largest gains on items of moderate difficulty,
suggesting that teachers were not simply teaching easily mastered
skills.

Dale Ballou, an associate professor of economics at the University
of Massachusetts at Amherst, said methodological differences could
account for the two studies' dueling conclusions.

But Ronald Ferguson, a lecturer in public policy at the Kennedy
school, suggested the biggest difference may be in how the researchers
interpreted the jump in test scores in 1996-97.

Mr. Bryk attributed those gains to changes in the first wave of
Chicago school reforms. Mr. Jacob attributed them to the newer wave of
accountability provisions. Both could be right, Mr. Ferguson suggested,
if the jump in scores stemmed from the threat of accountability rather
than the actual implementation of the accountability policies, which
occurred shortly afterward.

Another study, by Julian R. Betts, a professor of economics at the
University of California, San Diego, and Anne Danenberg, a research
associate at the Public Policy Institute of California, based in San
Francisco, examined the effectiveness of California's intervention
program for low-performing schools. The Immediate
Intervention/Underperforming Schools Program provides schools in the
bottom half of the state's accountability index that have not met their
improvement targets with additional money and assistance if they apply
for and are randomly selected into the program.

The researchers found that schools with the lowest test scores were
the most likely to apply for the program, "which I think should be very
heartening to anybody in favor of accountability systems," said Mr.
Betts.

Moreover, schools that applied for and were selected into the
program made substantially greater achievement gains between 1999 and
2001 than schools that applied but were not picked.

But the researchers found no evidence that schools participating in
the program outpaced the total pool of eligible schools, which includes
those that did not apply.

"Is the program working?" asked Mr. Betts. "It's really too early to
know for sure."

Vol. 21, Issue 41, Page 13

Published in Print: June 19, 2002, as Accountability Studies Find Mixed Impact on Achievement

Notice: We recently upgraded our comments. (Learn more here.) If you are logged in as a subscriber or registered user and already have a Display Name on edweek.org, you can post comments. If you do not already have a Display Name, please create one here.

Ground Rules for Posting
We encourage lively debate, but please be respectful of others. Profanity and personal attacks are prohibited. By commenting, you are agreeing to abide by our user agreement.
All comments are public.