Last week, with much fanfare, a study comparing standardized test scores of New Jersey’s charter school students to those of their public school peers was released by the Center for Research on Education Outcomes (CREDO).
As a professor of public policy, a supporter of public education, and a parent of a charter school student, I have four questions that I would like to ask the authors.

The CREDO press release claimed that “New Jersey charter public schools significantly outperform their district school peers.” However, this is not even remotely what the CREDO study found.

First, the CREDO study looked at only about half of New Jersey’s charter schools (46 out of 86).

Second, the study excluded another quarter of the state’s charter school students (23 percent), particularly those from groups that score lower on standardized test scores (students who have to repeat a grade, students with special needs, and students with limited knowledge of English).

Third, the study did not include students who had left charter schools. This is especially problematic given the significant attrition levels at the highest scoring charter schools, with the most academically challenged students the most likely to leave.

So what did the CREDO study find about the performance of the remaining students?

The vast majority of charter school students performed worse or at the same level as students in the traditional public schools from which they came (70 percent lower or same in math and 60 percent lower or same in reading).

The charter school students who performed better were located almost exclusively in Newark, while charter school students in other cities and rural areas consistently and significantly underperformed their traditional public school peers.

The charter school students who performed better did so only for their first two years at the charter school, while their third year performance was actually worse than their traditional public school counterparts.

In other words, the study looked at a limited sample of charter school students, excluding those most likely to be academically challenged, and still found that only a minority of those students outperformed their traditional public school counterparts, and only for some of the years studied.

Which brings us back to the first question: How can an institution that claims to be academically objective put out a press release that is so misleading about the study’s findings?

Question #2: Why did the CREDO analysis largely ignore the dramatic demographic differences between charter and traditional public schools?

Charter schools and traditional public schools serve very different populations of students.

The CREDO study acknowledged this fact in its finding that the traditional public schools it looked at served four and a half times as many students with Limited English Proficiency and one and a half times as many special-needs students as did the charter schools.

However, the CREDO study failed to consider the differences in severity of a student’s language limitations or disability.

Rutgers professor Dr. Bruce Baker has demonstrated that charter school students have much less severe disabilities than their traditional public school counterparts. And more severe disabilities, such as mental illness or autism, are associated with much lower standardized test scores than mild disabilities, such as minor hearing or speech limitations.

The CREDO study did not evaluate the other dramatic demographic differences between charter and traditional public schools: income. Instead, the CREDO data treated all students from families earning up to 180 percent of the poverty line as interchangeable.

CREDO tried to address these demographic differences by factoring in students’ standardized test scores at the time of charter school enrollment. However, as Professor Baker points out, this ignores the post-enrollment impact of differential poverty and disability on students’ learning.

The CREDO study also ignored the positive impact on charter school standardized test scores resulting from the different demographic makeup of student populations in charter and traditional public schools – what academics refer to as the “peer effect.” Going to school only with other motivated and high functioning students may indeed improve educational outcomes for some charter school students, but at what cost?

In Newark, for example, Professor Baker has demonstrated that the growth of charter schools has resulted in an increased concentration of the most challenging to educate children within the traditional public school system. So while some charter schools end up with an easier to educate population, the learning environment for the majority of Newark’s children becomes much more challenging.

Question #3: When will CREDO release their data so that other researchers can verify their findings?

CREDO is a part of the conservative Hoover Institution, and its charter school research is funded by the equally conservative Walton Foundation. In prior publications, CREDO has made clear that it is not a disinterested research institution. Rather, it has a policy agenda of quality “school choice” expansion.

While having an ideological agenda and funding does not preclude CREDO from producing high- quality research, it places particular importance on the transparency of their data and methods, so other researchers can replicate the results.
Unfortunately, CREDO did not release the data from the New Jersey study, although that information was provided to the individual charter schools that were included in their research.

Until CREDO provides this data to other researchers, it is not possible to evaluate fully the accuracy or validity of their findings.

My fourth and final question for the CREDO authors is: What is the objective of this research?

While the CREDO study is an improvement on similar research efforts by the New Jersey Department of Education, the study’s framing makes its findings largely useless.

If CREDO wanted to move the education policy agenda, why not look at what is causing the demographic segregation between charter and traditional public schools and how this segregation can be rectified? Or how about examining which practices at charter and traditional public schools lead to better educational outcomes that can be broadly replicated?

Unless CREDO’s goal is to feed divisive and useless debates about whether charter or traditional public schools are better, the kind of research they released last week accomplishes very little.

Julia Sass Rubin, Ph.D., is an associate professor of public policy at the Edward J. Bloustein School of Planning and Public Policy.