Trends: Charter Vs. Pilot Schools

In the national charter school debate, Boston has special significance. The city has unleashed imaginative teachers to run both independent charter schools and semi-independent “pilot” schools, with much of the rest of the country waiting to see which does best.

Teachers unions and charter opponents have put unusual emphasis on this contest. Boston pilot schools were designed to show that schools with collectively bargained pay scales and seniority protections could do just as well as charters, whose teachers are usually non-union.

Charters, independently operated schools with public funding, were not designed to be anti-union. American Federation of Teachers founding president Al Shanker originated the charter idea. But many conservatives who think unions stand in the way of raising student achievement have embraced the charter school cause, thus politicizing the debate. Their side just won the first round in Boston, and they are not likely to let charter opponents forget it.

A study by scholars from Harvard, MIT, Michigan and Duke, sponsored by the Boston Foundation, shows the Boston charters are doing significantly better than pilots in raising student achievement. This includes results from randomized studies designed to reduce the possibility that charters might benefit from having more motivated students and parents. The study is called “Informing the Debate: Comparing Boston’s Charter, Pilot and Traditional Schools.”

People who see charters as a ruinous drain on regular public schools, and a threat to job security and salary protections for teachers, are not going to accept this verdict. The data come from just one city, with many qualifications. For instance, the randomized results apply only to charters so popular they have more applicants than they can accept. Less popular charters were not included in that part of the study; they could have reduced the charters’ measured gains if their data had counted.

Here is how the scholars summarized their findings. Don’t be frightened by the techie reference to standard deviations, a statistical way of measuring difference that the authors quickly explain:

“We generally find large positive effects for charter schools, at both the middle school and high school levels. For each year of attendance in middle school, we estimate that charter schools raise student achievement .09 to .17 standard deviations in English language arts and .18 to .54 standard deviations in math relative to those attending traditional schools in the Boston Public Schools. The estimated impact on math achievement for charter middle schools is extraordinarily large. Increasing performance by .5 standard deviations is the same as moving from the 50th to the 69th percentile in student performance. This is roughly half the size of the black-white achievement gap. In high school, the estimated gains are somewhat smaller than in middle school: .16 to .19 standard deviations in English language arts; .16 to .19 in mathematics; .2 to .28 in writing topic development; and .13 to .17 in writing composition.”

The scholars add: “The results for pilot schools are more ambiguous and deserve further study. In the elementary grades, the estimated impact of pilots was positive in English language arts (.09), but not statistically different from zero in mathematics. In the middle school grades, the observational results suggest that pilot school students may actually lose ground relative to traditional public school students, with point estimates of -.05 standard deviations per year in English language arts and -.07 in math.”

These results appear at an interesting time. Some union leaders, particularly AFT President Randi Weingarten, have begun an aggressive effort to show they can match the best charters’ reading and math gains with charter or pilot schools run by union members. Teachers at a Brooklyn charter school started by the Knowledge Is Power Program (KIPP) recently joined Weingarten’s union. If they challenge the longer KIPP school day and the power of KIPP principals to fire teachers, a clash is possible between Weingarten and KIPP co-founder Dave Levin. The New York City KIPP education director, Levin has one of the best records in the country for raising the achievement of inner-city middle school students and winning fervent parent loyalty.

Mathematica Policy Research is studying several KIPP schools, using the same method employed in the Boston study. Students who enroll in KIPP in fifth grade are being followed for several years, their records compared with a control group of students who also applied to KIPP but were not selected in the randomized lotteries required when charters are oversubscribed. The Boston study gathered similar data for charter schools and pilot schools with an overflow of applicants.

In Boston, charter and pilot schools have proved popular with parents. Both have performed better than regular public schools in at least some categories. Giving school leaders more independence seems to produce more energetic teaching and better-coordinated programs. It is a shame these two experiments have been put in conflict with each other. But the role of unions in education is a lively issue, often discussed in campaigns and conferences, so Charters vs. Pilots will continue to draw a big crowd, with many rounds to go.

There is a severe weakness in the analysis of this article as there is in the study the article talks about. Until students are assigned to charter schools using a blind, random assignment process, the selection of the students enrolled in charter schools and any study performed on them will suffer from selection bias and any analysis performed will be unable to report valid results. It is not possible to remove selection bias using any post assignment statistical techniques.

If you read the post it sounds like the study compared the academic results of kids who were randomly admitted to a charter school against a group of kids applied to the same charter school but were not admitted(on a random basis) so that takes care of the selection bias.

Hey abird, and others who might be confused about how the study worked. The "lottery" study described above is the more statistically reliable study of the two, as it tracks the results of students who all applied to charter schools and got in against the results that students that applied and didn't get in. In this way, the "selection bias" that is often noted as partly responsible for greater student achievement among charter school student populations is accounted for before any analysis has begun.

That said, the charters included in the pilot study were oversubscribed, as described in the article, and are also some of the best charter schools in Massachusetts. The Pilot schools against which these schools were compared were the only schools that did not have some sort of screening process for incoming students (applications, interviews, test scores, portfolios, etc.), leading one to the argument that they were less academically rigorous than the other pilots.

Taking these results and forming the idea that "Charter schools are better than Pilot schools, period." would be a difficult, and incorrect. However, looking at these results and saying that the specific charter schools used in the lottery study do a better job of educating students that the pilot schools included in the study would be pretty much correct.

It should also be noted that the report makes absolutely no claims as to the reasons why these charters outperformed the pilots.

If anyone is interested in reading the study itself, here's a link (you'll need a pdf reader to open the file): http://www.gse.harvard.edu/~pfpie/pdf/InformingTheDebate_Final.pdf

The big plus concerning charters (and pilots) in Boston as elsewhere are they allow for school choice to poor/minority youngsters previously afforded only to families of wealth. This goes a long way toward explaining why charter students could possibly perform better than youngsters in traditional public schools. Parents of these kids are involved and pay attention to their kids' education. Sadly, this is not always the case with parents of traditional public school students. As you know, parent involvement is a huge positive in determining student success.

Any conversation involving KIPP schools needs to include their failure to retain students who start in grade five and "graduate" from their schools at the end of grade eight. To date, their record on this count is suspect, at best.

In a recent group of discussions with parents of 4 year olds in DC I learned with some distress that not one of the parents in that entire group were going to send their kids to DCPS until Michelle Rhee's proposals were met. Every single parent had their eye on Yu Ying, Capital City and other charter schools.

I am a firm believer in public schools, I send my children to DCPS and I am slowly becoming a school activist. I would never send my kids to a charter school with no resources when DC schools are bursting at the seams with money, corporate donations, improved facilities, small class sizes and everything else except quality teachers in all grades and at all schools and of course bureaucratic central office staffs.

So many of the public vs charter questions are moot in suburban districts, but in DC it's a critical question. The lack of action on the part of the nationwide teacher's industry has allowed too many members in their ranks to become as professional as the postal service when once they were pillars of the community. (Not busting on the postal service in particular, but it's a career with little educational expectations.)

I am EAGER to learn how AFT and WTU plan to re-professionalize their members and get them performing to the level of charters. I want to believe that all DCPS schools can perform as well as Oyster, Murch, Eaton, Banneker, Wilson, Deal, Hardy, etc. But when some friend forwards me an email from a DC principal or teacher that contains typos or grammatical confusion? Oh jeez... the Charters are just going to recruit 90% of the parents in DC so that in 6 years there will be no more public elementary schools.

It's striking to me that charters are not wildly outperforming regular public schools, given the freedom of their leaders and the self-selected group of students (okay, parent-selected). That they are not, speaks to how difficult it is to "raise achievement" however achievement is defined. (It seems vague to me).

Although it seems as if some very good information is coming out of the charter school programs, it is always with sadness and anger that I read articles such as this one. This is not a knock on Jay. This article continues the needed dissemination of information about what it takes to bring kids up to higher achievement levels.

Teachers unions do need to be willing to acknowledge that some of its teachers are not and will not be up to snuff. On the other hand, in the school district in which I work, I would trust only one of six administrators to be really competent at evaluating staff and at helping staff make needed changes.

I think that equating teacher's unions with poor achievement levels is really a conservative agenda smokescreen. What do you think education would be like if unions had not fought for current pay scales and benefit levels? Who would be going to college with the thought of making not much money with few benefits in a job that takes a good deal of effort and skill?

I think that it is clear that the school day and year need to be lengthened. Right now that only way to do this for the vast majority of school districts is by raising local money through property tax increases.

Two independent research studies on Chicago schools were released 2/17/09.
http://www.uic.edu/educ/ceje/resources.html
Both point out that Chicago public schools is using flawed data resulting in bad decisions. One compares charter and neighborhood high school performance on ACT, as well as student enrollment and teacher factors (not so great for charters). The other on Chicago's current proposal to close or "turnaround" 22 school which data show are primarily in communities of color experiencing gentrification or rapidly changing demographics. This study proposes a new method to measure utilization rates – a common reason cited for closing schools.
These studies deserve your attention.

Two independent research studies on Chicago schools were released 2/17/09.
http://www.uic.edu/educ/ceje/resources.html
Both point out that Chicago public schools is using flawed data resulting in bad decisions. One compares charter and neighborhood high school performance on ACT, as well as student enrollment and teacher factors (not so great for charters). The other on Chicago's current proposal to close or "turnaround" 22 school which data show are primarily in communities of color experiencing gentrification or rapidly changing demographics. This study proposes a new method to measure utilization rates – a common reason cited for closing schools.
These studies deserve your attention.