Comments on: Estimated effect of early childhood intervention downgraded from 42% to 25%http://andrewgelman.com/2014/08/08/estimated-effect-early-childhood-intervention-downgraded-42-25/
Wed, 13 Dec 2017 21:32:13 +0000hourly1https://wordpress.org/?v=4.9.1By: How does a Nobel-prize-winning economist become a victim of bog-standard selection bias? - Statistical Modeling, Causal Inference, and Social Sciencehttp://andrewgelman.com/2014/08/08/estimated-effect-early-childhood-intervention-downgraded-42-25/#comment-527029
Thu, 20 Jul 2017 13:59:34 +0000http://andrewgelman.com/?p=23378#comment-527029[…] models of selection bias, but he doesn’t see it when it’s right in front of his face. See here, for […]
]]>By: When good governments (or any governments) base policies on bad research - The Washington Posthttp://andrewgelman.com/2014/08/08/estimated-effect-early-childhood-intervention-downgraded-42-25/#comment-240562
Wed, 09 Sep 2015 12:45:54 +0000http://andrewgelman.com/?p=23378#comment-240562[…] good news, whether it be the effectiveness of the latest cost-saving teaching innovation or the claim, based on a study of 130 kids in Jamaica, that early childhood intervention can raise children’s […]
]]>By: I didn't say that! Part 2 - Statistical Modeling, Causal Inference, and Social Science Statistical Modeling, Causal Inference, and Social Sciencehttp://andrewgelman.com/2014/08/08/estimated-effect-early-childhood-intervention-downgraded-42-25/#comment-195704
Wed, 15 Oct 2014 00:24:39 +0000http://andrewgelman.com/?p=23378#comment-195704[…] a phenomenon that is important if real. For another example, these problems have arisen in studies of early childhood intervention and the effects of air pollution. I’ve mentioned all these […]
]]>By: Elinhttp://andrewgelman.com/2014/08/08/estimated-effect-early-childhood-intervention-downgraded-42-25/#comment-184253
Mon, 11 Aug 2014 13:55:24 +0000http://andrewgelman.com/?p=23378#comment-184253Andrew did you read the supporting materials? The 25% in the abstract and results table shorter paper is for all jobs but the text reports “substantially larger” impact on full time non temporary work which is shown in the original paper and the supporting materials for this paper. Still above 40% in that scenario in the supporting materials. Of course the college attenders more often have work study , part time or student type temporary jobs.
]]>By: Elinhttp://andrewgelman.com/2014/08/08/estimated-effect-early-childhood-intervention-downgraded-42-25/#comment-184252
Mon, 11 Aug 2014 13:31:50 +0000http://andrewgelman.com/?p=23378#comment-184252That’s very interesting; basically you’re providing an actual explanation of why scale up often doesn’t work. In a way it reminds me of the “ready to change” factor in the evaluation of addiction treatment. What’s interesting is the idea that you might be able to model what settings a given intervention is likely to have the biggest impact in. It doesn’t necessarily mean “nothing works” and it’s not some mysterious consistent false positive internally. At the same time, a lot of times you’ll see interventions criticized for not being effective for the hardest cases, like school interventions working with parents are motivated/have some college or with kids who are already relatively successful in school. On the one hand, nothing wrong with having a program that works for them especially where they are in awful schools where despite their advantages they will still likely end up not graduating from high school or without college skills. On the other, the kind of program in the article discussed is likely only to be most helpful in very high need cases with diminishing returns as the situations improve even mildly. These children had stunted growth and were ages 0-2 in a very high poverty and high violence country. They were already behind their peers. They are probably among those children most likely to benefit from almost any intervention. 22% still in school versus 4%, that’s totally astonishing to me.

Very interesting. A similar phenomenon is how when running certain experiments to show a treatment leads to “activation” (eg phosphorylation) of a protein/ signaling pathway they do it under serum starved conditions which lowers the activity of a cell in general.

“To me, the experience of early childhood intervention programs follows the familiar, discouraging pattern …small-scale experimental efforts staffed by highly motivated people show effects. When they are subject to well-designed large-scale replications, those promising signs attenuate and often evaporate altogether.”

From the Abstract:
“Site selection bias” occurs when the probability that partners adopt or evaluate a program is correlated with treatment effects. I test for site selection bias in the context of the Opower energy conservation programs, using 111 randomized control trials (RCTs) involving 8.6 million households across the United States. Predictions based on rich microdata from the first ten replications substantially overstate efficacy in the next 101 sites. There is evidence of two positive selection mechanisms. … While it may be optimal to initially target an intervention toward the most responsive populations, these results show how analysts can be systematically biased when extrapolating experimental results, even after many replications

From the Abstract:
“Site selection bias” occurs when the probability that partners adopt or evaluate a program is correlated with treatment effects. I test for site selection bias in the context of the Opower energy conservation programs, using 111 randomized control trials (RCTs) involving 8.6 million households across the United States. Predictions based on rich microdata from the first ten replications substantially overstate efficacy in the next 101 sites. There is evidence of two positive selection mechanisms. … While it may be optimal to initially target an intervention toward the most responsive populations, these results show how analysts can be systematically biased when extrapolating experimental results, even after many replications

]]>By: gwernhttp://andrewgelman.com/2014/08/08/estimated-effect-early-childhood-intervention-downgraded-42-25/#comment-184078
Sat, 09 Aug 2014 15:35:19 +0000http://andrewgelman.com/?p=23378#comment-184078Speaking of Headstart, a new analysis is out suggesting the gains on cognitive tests were hollow as far as fluid intelligence goes: https://pdf.yt/d/i5PsjJ3sNIKHL-JG / https://dl.dropboxusercontent.com/u/85192141/2014-nijenhuis.pdf
]]>By: Some random links | Science.xcuz.mehttp://andrewgelman.com/2014/08/08/estimated-effect-early-childhood-intervention-downgraded-42-25/#comment-184073
Sat, 09 Aug 2014 14:50:08 +0000http://andrewgelman.com/?p=23378#comment-184073[…] Gelman has an interesting analysis of the Jamaica pre-school […]
]]>By: Elinhttp://andrewgelman.com/2014/08/08/estimated-effect-early-childhood-intervention-downgraded-42-25/#comment-184009
Fri, 08 Aug 2014 23:40:19 +0000http://andrewgelman.com/?p=23378#comment-184009The analysis is pretty complex but I’m guessing that the change in handling of the missing migrants (the treatment group was more likely to migrate, and among migrants the controls were more likely to be lost to follow up) may have led to the change. I would not be surprised if the migrants had salaries far higher than non migrants based on what I have heard from the Jamaican students I have; even working as off the book nannies in New York they are better off than at home. I’m also not surprised that so many of the treatment group at age 22 are both working full time and in college. It makes me wonder if the earlier analysis handled salaries of full time students differently than the final analysis did (22% of treatment still in school versus 4% of control). Twenty two is still too young to characterize these data as being about adult earnings if there are still over a fifth of of subjects in one group still in school. The payoffs from college attendance for even a small portion of either group are still in the future, but likely to be meaningful (and not just monetarily either, as as we saw in Changing the Odds and Passing the Torch). It will be extremely interesting to see where they are in five years.

I really wish they discussed the distribution of the 9 deaths across the groups. Among other things Jamaica has a very high homicide rate; not so high that you’d expect a homicide among 127 subjects but still, they were probably higher risk than others of the same age. From what I can tell lost to follow up and dead were treated as the same for the purposes of imputation. Maybe that is something that changed also.

In general, school teaching can be thought of as a very unglamorous form of show biz, which involves stand-up performers (teachers) trying to make powerful connections with their audiences (students). We are not surprised that some entertainers are better than other entertainers, nor are we surprised that some entertainers connect best with certain audiences, nor that entertainers go in and out of fashion in terms of influencing audiences.

In other words, if you think of entertainment as social science experiments, they have poor replicability. But that doesn’t mean that the original Elvis (the initial experiment) didn’t have a big impact on fans, just because all the Elvis impersonators (the replication experiments) seem pretty ho-hum these days. Similarly, just because it’s hard to replicate the results of successful educational and early childhood interventions doesn’t mean they didn’t originally actually happen.

It’s also not generally the case that large-scale early childhood interventions do not work. The Chicago Child-Parent Center program is large scale, as are the various other state and local pre-K programs that have been studied, such as programs in Tulsa, Boston, and North Carolina.

]]>By: Andrew Sabiskyhttp://andrewgelman.com/2014/08/08/estimated-effect-early-childhood-intervention-downgraded-42-25/#comment-183949
Fri, 08 Aug 2014 13:53:03 +0000http://andrewgelman.com/?p=23378#comment-183949Well, I haven’t been to Jamaica, but from a brief check of the stats the poverty rates look fairly bad. So it’s certainly not implausible that outcome heritability is much lower over there and that family environment accounts for a much greater percentage of the variation in adult outcomes than is the case in the US or Europe. If these programs are going to work anywhere, I’d imagine Jamaica would be the place.

The Abecedarian Project is a strange one. As I understand it, it was originally supposed to investigate whether or not educational & social intervention could prevent mild to moderate learning disability. The problem is that both the control group (mean IQ 89) and the intervention group (mean IQ 93) had IQs perfectly within the normal range, especially for the African-American population that the project was aimed at. So over time, it seems to have switched focus towards seeing what happened to fairly normal kids if you gave them extra stimulation anyway. Apparently the final test score difference between the groups was exactly the same as that at 6 months of age (though measuring intellectual development at that age is a very dubious business). This has led some to suggest it’s just an example of unhappy randomisation, helped out by small sample size (see Bacharach & Baumeister, http://libres.uncg.edu/ir/asu/f/Bacharach_V_2000_Early_Generic_Educational_Intervention.pdf)

Regardless, I think it’s fairly clear that results from Abecedarian should be taken with a huge pinch of salt. Anyone who places too much weight on its findings is probably selling something.

]]>By: greghttp://andrewgelman.com/2014/08/08/estimated-effect-early-childhood-intervention-downgraded-42-25/#comment-183946
Fri, 08 Aug 2014 13:37:54 +0000http://andrewgelman.com/?p=23378#comment-183946Just a minor thing about the quote “the press release did not link to the published article”. I can’t find the link in your post either… :)
]]>