Audit questions impact of voucher schools on test scores

A five-year study released this year found students in voucher schools improved slightly more on state tests than those in Milwaukee Public Schools - but a state audit of the study released Thursday said voucher schools' impact on test scores is inconclusive.

State lawmakers in 2005 passed Wisconsin Act 125, which required all private schools participating in the Milwaukee Parental Choice Program to give standardized tests to all students in fourth, eighth and 10th grades and submit the results to the School Choice Demonstration Project, based at the University of Arkansas. That same act required the state Legislative Audit Bureau to analyze the project's annual reports to compare voucher and MPS achievement.

The School Choice Demonstration Project studied 5,454 third- through ninth-grade students - half in voucher schools, half in MPS - beginning in the 2006-'07 school year. It released several reports per year on differences between the two school systems. Its final scheduled report in February on Milwaukee schools found little difference in test scores between voucher schools and MPS.

Beginning in the 2010-'11 school year, voucher schools were required to give the standardized Wisconsin Knowledge and Concepts Examination to all pupils in grades three through eight as well as 10th grade - the same test administered in the public schools. They also had to submit the results to researchers and the Department of Public Instruction, which made the information public.

Studies have shown that students' scores often go up after introducing test-based accountability policies. So after the WKCE was required in voucher schools during the last school year of the study, the Legislative Audit Bureau couldn't determine whether any bumps in scores were due to the newly administered tests or the voucher schools themselves. Therefore, it said it could not conclude the impact of voucher schools on achievement.

Many of the original students in the study transferred between the voucher and public school systems, graduated or were untraceable by researchers for at least one year in the study. Only 17.5% of voucher students in the study remained in a voucher school five years later. Another 18.4% transferred into MPS.

Far fewer MPS students from the sample switched to voucher schools in that time. Only 3.3% of MPS students studied transferred to a voucher school, and 43.5% stayed in MPS schools for the five years.

Nearly a third of the sample students from both types of schools were missing data for at least one year of the study.

Project researchers and the state auditing team varied slightly in how they calculated the data, but reached similar results.

Project researchers followed students in the sample for five years, including students who began the study in voucher or MPS schools but switched between them during the five years. They said this is how pharmaceutical studies work: If a subject on an experimental drug stops taking it, researchers still follow what happens to the subject.

State auditors excluded students who switched between school systems, so that only students who remained in either voucher schools or MPS for the entirety of the study were included in the analysis of test scores. That way they could attribute student outcomes to either the voucher program or MPS, said Program Evaluation Director Dean Swenson, though this reduced their sample size significantly.

Still, both teams found voucher schools to have had a statistically significant increase in math or reading scores in seventh, eighth or 10th graders, but those subjects and grades were different for both teams.

Comparison studies of Milwaukee's voucher program have repeatedly yielded results riddled with caveats, School Choice Wisconsin President Jim Bender said. He said there needs to be a stronger push for better uniformity in studies such as this one so data is accurate.

"We are in support of good data getting to parents, so they can make quality educational decisions for their children," he said. "We are not trying to avoid accountability, we are trying to make sure that when we relay info to parents, we're using reliable data."

MPS spokesman Tony Tagliavia said in an email that both systems have room to grow.

"We certainly recognize that we're in a competitive educational marketplace but the reality is that, as a whole, neither group of students is at the level of achievement where we need and want them to be," he wrote in an email. "We further believe that we're the only educational organization in the city of Milwaukee with the commitment and capacity to educate all children."