In “The One World Schoolhouse,” Salman Khan presents a simple thesis: We learn best when we learn actively and at our own pace, mastering each new skill before proceeding to the next. What sets Mr. Khan apart from most pedagogical theorists, besides the fact that he’s actually right, is that he’s giving his services away. His website, KhanAcademy.org, hosts thousands of instructional videos and interactive lessons. Millions of people around the world have used them and sing their praises.

Given his growing success, Mr. Khan’s goal is suitably ambitious: “A free, world-class education for anyone, anywhere.” But he seems to want to change the way the world learns without changing the way the world schools.

Mr. Khan’s focus is inside the classroom on instructional practices and tools. He is largely silent on, and seems indifferent to, the ways schools are managed and how students choose or are assigned to them and the way teachers are trained and compensated.

In December of 1993, Bill Clinton remarked that figuring out how to consistently replicate great schools was the central education policy problem of our age. A generation later, it still is.

As someone obsessed with solving that problem, I wanted to know how well our current strategies for achieving it are working. One strategy in particular has attracted the bulk of the funding and attention over the past decade: philanthropists teaming up with what they consider to be the best networks of charter schools, and funding their growth. To find out how well they’ve been picking the winners, I compared the total amount of grant funding received by each of 68 California charter school networks over the past 8 years to the academic performance of those networks. The study is available here.

The correlation between grant funding and performance on the California Standards Tests turns out to be negligible (0.1). In fact, it’s half the size of the correlation between performance and the length of the networks’ names. As a check on those findings, I also ran the numbers on AP test performance. Those results are slightly worse: though the correlations are also negligible in magnitude, they’re actually negative in sign—more grant funding is associated with minutely worse AP performance.

In a nutshell, it’s as if philanthropists have been doling out funding to charter school networks by the same random lottery process by which oversubscribed charters are obliged to accept new students. While this will of course be viewed as a great disappointment by a great many people, it is better to have this information than to continue to labor in ignorance. There are places where excellence in education does routinely scale-up, and documenting them is the subject of my next project….

An anonymous contributor to the NY TimesFreakonomics blog asks “How much does school choice matter?” And answers in much the way you might expect:

Probably less than you think, as [economist Stephen] Levitt has previously argued. Now, in an analysis of seven years of test-score data from 6,000 Los Angeles teachers, the L.A. Times and the Rand Corp. have found teacher effectiveness to be three times more influential than [choice of] school… on student performance.

To his credit, Stephen Levitt did make this distinction in the 2007 posting linked in the blockquote above. But even Levitt refered to the evidence on this subject as “hard to find.” Should anyone else be experiencing difficulty in finding this evidence, they are encouraged to click on the link immediately above, where they will find a peer-reviewed paper digesting the results of 65 studies on this subject.

To the extent education policy commentary actually affects policy, it has the potential to do great good or great harm. Several recent commentaries in this field fall into the latter camp, and it’s important to understand why – so that we can avoid similar mistakes in future.

The one I’ll discuss here is this blog post by Matthew Yglesias, in which he draws broad conclusions about the functioning of education markets from a recent study of a tiny school choice program in Milwaukee as well as from some older unspecified research [for the latter, Yglesias linked here, but the body of that page doesn’t discuss school choice]. The Milwaukee study is part of a vast literature. Over the past quarter century at least sixty-five studies have compared outcomes in public and private schools around the world, reporting 156 separate statistical findings.

The evidence of this literature is starkly one-sided. The vast preponderance of findings show private schools outperforming public schools after all the normal controls. What’s more, when we focus on the research comparing truly market-like systems to state-run school monopolies, the market advantage is found to be even more dramatic (see Figure 2 in the paper linked above). To draw policy opinions from a small, selective handful of those studies while ignoring the rest is policy malpractice, and it is dangerous to children.

Even the recent Milwaukee result described by Yglesias as a failure shows voucher students in private schools performing as well as public school students who receive roughly 50% more government funding. How is a program that produces similar academic results to the status quo at a much lower cost to taxpayers a failure? And what of the research suggesting that students in the Milwaukee voucher program graduate at higher rates than those in public schools?

More importantly from a long term policy perspective, how is a program limited to 20,000 or so children in a single city, being served almost entirely by non-profit entities, a test of market education? Would Apple have spent hundreds of millions developing the iPhone or the iPad if its market were limited to the same customer base? Of course not. The dynamism, diversity and innovation we have come to expect from competitive markets in other fields relies on the prospect of ultimately scaling up to serve mass audiences. Without the prospect of a large-scale return on investment, there is no incentive to invest in the first place.

Andrew Gillen comes to the study’s defense, and I’m delighted that he’s taken the trouble to reflect on it rather than just saying “I like it.” But there are problems with his analysis. First, he faults me for dismissing the “Anti-Lemons” models for being based on false assumptions, citing Paul Krugman:

I am a strong believer in the importance of models, which are to our minds what spear-throwers were to stone age arms: they greatly extend the power and range of our insight. In particular, I have no sympathy for those people who criticize the unrealistic simplifications of model-builders.

Even if we put aside the fact that Paul Krugman is at times less reliable than the Daily Show website, there is an important difference between assumptions that are “unrealistically simplified” and those that are patently wrong. With the former, your model might still huck its intellectual spear somewhere in the general vicinity of the truth, with the latter, you’re just going to put your eye out.

“Anti-Lemons” is in the put-your-eye-out camp. Among other things, it assumes the productivity of all schools is equal. This is both totally false and highly germane – efficiency varies dramatically among schools, and private schools as a whole are consistently more efficient than government schools (as we will see below). Failing to recognize that reality will lead to incorrect results from the model, and this is just one of the false assumptions the paper adopts (see my previous post for others).

Second, Gillen writes that

going by Coulson’s numbers in figure 2 here, we would expect to find a positive impact of markets over government on achievement in slightly less than 2 out of 3 studies (with insignificant findings making up the majority of the others). If the case for free markets over government schools is really so clear cut (and I lean strongly in this direction), than why isn’t this 3 out of 3?

There are many plausible reasons for this result (lack of statistical power, omitted variable bias, other misspecification errors, etc.), but one is particularly worth raising here: government schools in many parts of the world spend several times as much per pupil as their private sector counterparts. This is true in most developing countries, from which a great deal of the inter-sectoral research hails. And when I looked at statewide data from Arizona in 2006 I found that government schools spend roughly 50 percent more than private schools. While it’s true that government school outcomes tend not to improve much as spending rises, the same cannot be said of private schools.

If this is true, you might ask, then wouldn’t the inter-sectoral research on school efficiency be more stark than the research on achievement (that fails to take spending levels into account)? The answer is yes. In fact, if you examine the efficiency bar in the same figure 2 cited by Gillen above, you will see that every single one of the efficiency comparisons between market and monopoly schools is significant and favors the market schools.

So, not only is the “Anti-Lemons” model useless, it is worse than useless: it seems to mislead even intelligent readers into believing that there is some mystery in the literature that needs to be solved by blindly waiving a spear around.

“Anti-Lemons” is neither Camelot, as I said yesterday, nor is it Sparta as Andrew implied. It’s the kid from Christmas Story who nearly puts his eye out by the cavalier application of a potentially powerful tool.