They find that decreased sexual activity is found primarily among females, accompanied by a decreased likelihood of “fecundity.” Decreased cocaine use and arrests were found primarily among boys, who were also less likely to smoke tobacco.

Overall, private schools had little effect on adolescent drinking and marijuana use. This might be due to the fact that while some students were led to avoid drugs and alcohol altogether, students who otherwise would have used hard drugs instead just smoked pot and drank booze – in my view.

Figlio and Ludwig’s data comes from the late 1980s and early 1990s. They find that the effects of private school were focused primarily in two-parent households. They also find that the effects were concentrated in the suburbs, though the definition of suburb in their dataset is overly broad, which unnaturally decreases the accuracy of their estimates in urban areas.

Studies of private schools are typically fraught with selection problems: students who select into private schools might have selected, in the case at hand, to avoid hard drugs and have less sex no matter where they attended school. Figlio and Ludwig eliminate this bias from their study by using a clever instrumental variables model in which transportation infrastructure differences between cities are shown to have an effect on the likelihood that parents will send their children to private religious schools.

Figlio and Ludwig point to character education in religious private schools. I can’t think of a better explanation.

Religious private schools are often organized with family life in mind. They seek to help families raise children who will be good parents and good citizens; their definitions of good parenting and citizenship are, of course, colored by their varying religious beliefs. All religions are not the same. But most religious schools share the overlapping belief that students should abstain from sex and should not use drugs. Figlio and Ludwig show that they’re doing a better job than public schools and non-religious private schools at achieving those goals; is this because religious schools have more rigorous abstinence and drug resistance programs than public schools? I doubt it. Private religious schools largely were established with a private community purpose in mind – which is very different from a public policy purpose. They are able to make broader pleas to their communities.

Perhaps most importantly, private religious schools are less afraid to discuss the consequences of bad, selfish behavior. They encourage kids to think over the long-term, the very long-term.

Figlio employs a regression discontinuity design to analyze the data, and his finding of a small but statistically significant academic gain fits quite comfortably with the larger random assignment literature, which find small year to year gains which accumulate over time.

One of the under-appreciated features of the random assignment literature: the studies usually fall apart after three or four years due to attrition in the control group. Our window into the academic benefits of choice is therefore limited. Figlio’s employment of a different analytical technique provides confirms previous findings, and may (?) open the door to longer term assessment. The challenges with the data described in this paper, however, suggest that it may not be easy.

Money quote from the study, with a definite echo of previous random assignment studies:

These differences, while not large in magnitude, are larger and more statistically significant than in the past year’s results, suggesting that successive cohorts of participating students may be gaining ground over time.

So, the Step Up for Students program has now been found to help improve public school results, help improve participant academic gains, generates high levels of parental satisfaction. Sounds like a rock solid justification for expansion to me.

The reaction of New York Times reporter, Sam Dillon, and LA Times reporter, Jason Felch, to my post on Monday about erroneous claims in theircoverage of a new Gates report could not have been more different. Felch said he would look into the issue, discovered that the claimed negative relationship between test prep and value-added was inaccurate, and is now working on a correction with his editors.

But more importantly, Dillon failed to check the accuracy of the disputed claim with independent experts. Instead, he simply reconfirmed the claim with Gates officials: “For your information, I contacted the Gates Foundation after our correspondence and asked them if I had misquoted or in any way misinterpreted either Vicki Phillips, or their report on their research. They said, ‘absolutely not, you got it exactly right.'”

He went on to call my efforts to correct the claim “pathetic, sloppy, and lazy, and by the way an insult.” I guess Dillon thinks that being a reporter for the New York Times means never having to say you’re sorry — or consult independent experts to resolve a disputed claim.

If Dillon wasn’t going to check with independent experts, I decided that I should — just to make sure that I was right in saying that the claims in the NYT and LAT coverage were unsupported by the findings in the Gates report.

Just to review, here is what Dillon wrote in the New York Times: “One notable early finding, Ms. Phillips said, is that teachers who incessantly drill their students to prepare for standardized tests tend to have lower value-added learning gains than those who simply work their way methodically through the key concepts of literacy and mathematics.” And here is what Jason Felch wrote in the LA Times: ““But the study found that teachers whose students said they ‘taught to the test’ were, on average, lower performers on value-added measures than their peers, not higher.” And the correlations in the Gates report between test student reports of test prep and value-added on standardized tests were all positive: “We spend a lot of time in this class practicing for the state test.” (ρ=0.195), “I have learned a lot this year about the state test.” (ρ=0.143), “Getting ready for the state test takes a lot of time in our class.” ( ρ=0.103). The report does not actually contain items that specifically mention “drill,”work their way methodically through the key concepts of literacy and mathematics,” or “taught to the test,” but I believe the reporters (and perhaps Gates officials) are referencing the test prep items with these phrases.

I sent links to the coverage and the Gates report to a half-dozen leading economists to ask if the claims mentioned above were supported by the findings. The following reply from Jacob Vigdor, an economist at Duke, was fairly representative of what they said even if it was a bit more direct than most:

I looked carefully at the report and come to the same conclusion as you: these correlations are positive, not negative. The NYT and LAT reports are both plainly inconsistent with what is written in the report. A more accurate statement would be along the lines of “test preparation activities appear to be less important determinants of value added than [caring teachers, teacher control in the classroom, etc].” But even this statement is subject to the caveat that pairwise correlations don’t definitively prove the importance of one factor over another. Maybe the reporters are describing some other analysis that was not in the report (e.g., regression results that the investigators know about but do not appear in print), but even in that case they aren’t really getting the story right. Even in that scenario, the best conclusion (given positive pairwise correlations and a hypothetically negative regression coefficient) would be that teachers who possess all these positive characteristics tend to emphasize test preparation as well.

Put another way, it’s alway good to have a caring teacher who is in control of the classroom, makes learning fun, and demands a lot of her students. Among the teachers who share these characteristics, the best ones (in terms of value added) appear to also emphasize preparation for standardized tets. I say “appear” because one would need a full-fledged multivariate regression analysis, and not pairwise correlations, to determine this definitively.

Another leading economist, who preferred not to be named, wrote: “I looked back over the report and I think you are absolutely right!” I’m working on getting permission to quote others, but you get the idea.

In addition to confirming that a positive correlation for test prep items means that it contributes to value-added, not detracts from it, several of these leading economists emphasized the inappropriateness of comparing correlations to draw conclusions about whether test prep contributes to value-added any more or less than other teacher practices observed by students. They noted that any such comparison would require a multivariate analysis and not just a series of pairwise correlations. And they also noted that any causal claim about the relative effectiveness of test prep would require some effort to address the endogeneity of which teachers engage in more test prep.

You’re certainly correct here. A positive pairwise correlation means that these behaviors are associated with higher performance on standardized tests, not lower performance. The only way that it could be an accurate statement that test prep is causing worse outcomes would be if there was a negative coefficient on test prep in a head-to-head competition in a regression model — though even then, one would have to worry about endogeneity: maybe teachers with worse-performing students focus more on test prep, or maybe lower-performing students perceive test prep to be more oppressive (of course, this could go the other way as well.) But that was not the purpose or intent of the report. The report does not present this as a head-to-head comparison, but rather to take a first look at the correlates between practice measures and classroom performance.

There was no reason for this issue to have developed into the controversy that it has. The coverage contains obvious errors that should have been corrected quickly and clearly, just as Jason Felch is doing. Tom Kane, Vicki Phillips, and other folks at Gates should have immediately issued a clarification as soon as they were alerted to the error, which was on Monday.

And while I did not know where the error occurred when I wrote the blog post on Monday, the indications now are that there was a miscommunication between the technical people who wrote the report and non-technical folks at Gates, like Vicki Phillips and the pr staff. In other words, Sam Dillon can relax since the mistake appears to have originated within Gates (although Dillon’s subsequent defensiveness, name-calling, and failure to check with independent experts hardly bring credit to the profession of journalism).

The sooner Gates issues a public correction, the sooner we can move beyond this dispute over what is actually a sidebar in their report and focus instead on the enormously interesting project on which they’ve embarked to improve measures of teacher effectiveness. An apology from Sam Dillon would be also nice but I’m not holding my breath.

A new study by David Figlio links higher gains among Florida public schools with higher levels of competition from the Step Up for Students tax credit program. You can read the St. Pete Times story by Ron Matus here. Matus wrote:

Figlio emphasized the boost was significant, but modest.

“Anybody looking for a silver bullet has to keep looking,” he said. “What we find is certainly positive and statistically strong, but it’s not like public schools are revolutionizing overnight because of this, either.”

So it turns out that the public school gains associated with a state program with an initial statewide cap of $50m in a state with a multi-billion dollar public school budget were statistically significant but modest. Would it be reasonable to expect anything more from such a modest program? I suggest we scale this public school improvement program up to say a cool billion per year and then measure the impact.

My favorite line in the story comes from a hostile academic:

Another researcher remained skeptical. Stanford labor economist Martin Carnoy, who has studied the impact of vouchers and reviewed the latest study, said Figlio and Hart did “an honest job with the data.”

“But here is the real story: even after several years the effect size is TINY,” he wrote in an e-mail. “They are so small that even small downside effects would nullify them, leaving vouchers as mainly an ideological exercise.”

This is one of the more unintentionally hilarious statements I have read in some time. The field of education reform battle is covered with the dead bodies of reforms that show nothing in the way of a statistically significant impact. Increasing per pupil funding, Head Start, teacher certification, almost everything studied by the “What Works” clearinghouse so far, etc. All of these failures cost a great deal of money and deliver nothing in the way of sustained academic gains.

So the state of Florida passes a small law that actually saves the state money and shows a statistically significant and small result of improving public schools, and we are supposed to wring our hands and despair because something bad could come along and nullify the gains? Ummmmm, no.

First of all, nothing bad did come along and nullify the gains- quite the opposite. This program was only a part of the strategy to increase parental choice in Florida. That strategy also includes charter schools, McKay vouchers and virtual schooling- all of which either already are or soon will be much larger programs than Step Up for Students.

Second, the parental choice strategy was itself a part of a larger effort to improve Florida public schools. Parental choice reinforced the central K-12 reform of grading schools A-F. Transparency, rewards for success, consequences for failure formed the core of the Florida strategy.

Did it work?

The Step Up for Students program played a contributing role in Florida’s symphony of success rather than “destroying public education.” This is what Milton Friedman argued all along. Bravo- the obvious conclusion to draw is to push both parental choice and public school reform still further in Florida and elsewhere.

David Figlio’s study (with Cassandra Hart) on how the Florida Tax-Credit Scholarship program impacts public schools is finally out. Guess what? His detailed statistical analysis finds that competition from school choice improves public schools. (Here’s some local news coverage.)

But that was no surprise to anyone who’s been following the research. Early last year I counted up the studies and here’s what I got:

Removing the double-count for studies that had findings in multiple locations, that made it 16 studies finding school choice improves public schools to zero finding they hurt public schools. (The one null finding was in DC, where the program pays enormous cash bribes to the public system – apparently on the princple that children are the chattel property of the government school system – in order to deliberately neutralize its effect on public schools.)

As always, critics are trying to make hay out of the fact that in the Figlio/Hart study, a tiny, population-limited, regulation-cramped choice program produces only moderate-sized benefits. Well, geniuses, if the benefits of a tiny, population-limited, regulation-cramped program are too small for you, can you think of any way you might make the program’s impact bigger?

The editors of the Tampa Tribune have decided not to join the misinformed rush to judgment on Florida’s tax-credit scholarship program:

It’s too early to accurately gauge the students’ academic progress, as the University of Florida economics professor who oversaw the report emphasized. It measured only first-year test gains. Researcher David Figlio was handicapped by incomplete data for a baseline.

I’m shocked to see that in print. A newspaper actually checked the facts!

I do have to quibble with the editorial’s assertion that the Figlio study shows students who select into the program are among the most “academically challenged.” We don’t, in fact, know that. We know that they are more likely to come from schools that are among the most academically challenged. But school characteristics and individual student characteristics can vary considerably.

This matters because choice opponents have relied upon unsupported assertions about selection bias to wave away the consistent empirical research consensus showing that school choice works. In fact, the Figlio study doesn’t allow us to address this question, as the study itself explicitly says. Other research that does examine this question has not turned up any serious evidence that vouchers either “cream” (selecting high performers) or “dredge” (selecting low performers).

But the editors are back on solid ground when it comes to finances:

The program is a good deal for taxpayers.

Attending public school costs more. When local, state and federal costs, plus capital costs, are factored in, the average cost per student in public school is $12,000.

In the voucher program, the maximum scholarship is $3,950, about 57 percent of the roughly $7,000 the state pays per public school student.

And a scholarship parent pays on average $1,000 a year for their child to attend the private school. The program requires the parents and child to be motivated.

By taking challenging students from poor-performing schools, the Tax Credit Scholarships are easing the burden on the public school system, not diverting resources.

Kudos to the Tribune for checking the facts rather than rushing to judgment!

David Figlio’s latest report on Florida’s Corporate Tax Credit (CTC) Scholarship program was released yesterday. I can’t find the report online but Ron Matus of the St. Pete Times sent it to me and you can read about it in his article.

I agree with almost everything said in the article. I even agree with Mark Pudlow, the spokesperson for the teacher union when he said: “There is no quick fix for struggling students.”

The problem is that the standard for success when it comes to school choice is that it has to produce a quick fix or critics deem it a failure and declare: “we really ought to reconsider why we’re doing it.” No one demands that every other education policy produce huge gains in a single year or they should be “reconsidered.” Yes, promoters of policies may make unrealistic promises to get them adopted, but the standard for success should be long-term progress, not promises made by politicians.

So let’s slow the rush to judgment and review what we really know about the CTC program. First, Figlio finds that 92.7% of all CTC students in private schools provided a usable standardized test to the evaluation. This shows widespread compliance with the legal requirements for those students to be tested to satisfy political concerns for accountability.

Second, Figlio finds that the CTC program has largely targeted students who are significantly more disadvantaged than students remaining in Florida’s public schools — even significantly more disadvantaged than public school students receiving subsidized or free lunch. So, concerns that the program would cream off the best students appear unfounded.

Third, and most importantly, Figlio’s report does not make any claims about whether students benefited academically from participating in the CTC program. He simply provides descriptive information on the academic achievement of CTC students as well as subsidized lunch students in Florida public schools. But we know that CTC students are even more disadvantaged than those public school students and Figlio makes no attempt in this report to control fully for those disadvantages.

Figlio makes these points explicitly and repeatedly in the report: “it is important to recognize that they are not causal estimates of the effect of program participation on student outcomes. Causal comparisons require more complete modeling of the selection decisions into the scholarship program and fuller data from a baseline than is afforded using the 2006-07 school year test score collection. More compelling causal estimates of program participation will be possible following the collection of the 2008-09 school year’s test score data. The comparisons in this subsection should be interpreted as purely descriptive in nature.”

Unfortunately, most people never pay attention to these warnings and rush ahead as if descriptive information is causal. Folks wrongly conclude that if CTC students make year to year test score gains that are about the same as subsidized lunch public school students, then they must not be benefiting from the program. Nothing in Figlio’s report supports that conclusion.

To know whether CTC students are benefiting we would have to know how they would be doing had they remained in public schools. The best way to judge that is with a random-assignment study where students admitted to the CTC program by lottery are compared with students who lose the lottery and remain in public schools. Unfortunately, that research design is not possible because there was no lottery. The next best thing would be to use a research design that approximated random-assignment (like a regression discontinuity) or a rigorous quasi-experimental design that controlled for all observed differences between the two groups. But Figlio didn’t do that in this report. He just provided descriptive statistics while promising a more rigorous research design next year.

Of course, we might wonder why Figlio bothered reporting this descriptive information without a more rigorous analysis. I suspect that he was required to produce a report each year by the legislature, so he complied even though he didn’t have the information he needed for a causal analysis.

And the descriptive information is useful. It suggests that choice was no miracle cure since the raw differences between CTC and public students in academic progress were not huge. Again, miracle cure is the wrong standard for judging a program’s success.

The CTC program may well have attracted students who had been on a downward trajectory before they switched to a private school. And the CTC program may well helped those students level-off and may, over time, enable them to make significantly greater progress than they would have made had they remained in public schools. This is what we’ve seen from rigorous evaluations of other choice programs, including the most recent evaluation of the DC voucher program. But these things require careful research designs and time to show themselves. Let’s give David Figlio more time to use a better research design so that we can actually say something about the academic effects of the CTC program.