Head Start Impact Study - A Flawed Assessment of Head Start Outcomes

Overview
Late last year, a new study found that Head Start children were better prepared for kindergarten in every aspect that the study measured compared to peers who did not attend Head Start. Debate around the study, however, has taken on a political tone, as critics focused on another finding that advantages ”fade out” through 3rd grade. This focus and the Head Start Impact Study are concerning. Both threaten to create a misleading and false debate about Head Start that ignores the many independent studies over the last fifty years that show that Head Start works.

The impact study control group is “contaminated”
Typically, a study assessing program benefits would compare a treatment group (those who attended Head Start) with a control group, who did not receive treatment, but met the same demographic characteristics. Dr. Edward Zigler, a Yale University professor, and one of the most well known early child development experts raised concerns about the study noting that “the groups in the Impact Study were badly contaminated.” He further notes that “many of their parents enrolled their children in another Head Start center that was not part of the study…[in the first year] 18% of the 3 year olds and 14% of the 4 year olds in the supposed control group actually attended Head Start.” In other words, the study ended up comparing ‘apples to apples’ according to Nobel Prize economist James Heckman. The overlap was even more extreme the second year – 50% of the 3 year olds in the ‘control group’ were enrolled in Head Start at age 4. [link]

Bottom line: students from both groups attended early education programs, and many in the control group enrolled in Head Start, raising questions about how big a difference there would be in their academic performance once they were in grade school.

Short term vs. long term benefits
Much of the discussion about the Impact Study revolves around the concept of ‘fade-out’. The term ‘fade-out’ is deceptive, since it measures only very short term cognitive benefits, and to a large extent represents other children catching up, rather than Head Start children losing the cognitive readiness that they were shown to have in kindergarten. It does not address the benefits of more active parenting and health, or the social and emotional development that is credited for much of the long-term benefits of early childhood programs. The Impact Study was not designed to and does not capture intermediate and long-term benefits of Head Start. Many other rigorous studies have done so, including studies more up-to-date than the Impact Study. Those studies found that Head Start reduced criminal activity, child mortality rates, special education costs, grade repetition later in school, and increased child achievement test scores, high school graduation rates, and immunization.
Some researchers have suggested that the long term benefits demonstrated by Head Start simply do not show up on in the point-in-time assessment used by the researchers in the Impact Study. Joshua Sparrow, a Harvard University researcher and a member of the Advisory Committee On Head Start Research and Evaluation said “most researchers looking at this data believe that this is because Head Start provides long lasting improvements in skill areas such as self-control, motivation, and social skills that are critical for long term economic and life success that the 3rd grade test simply does not pick up on.”

Despite the impact study’s flaws, if you dig deeper into the data there are sizable benefits measured for the most at-risk children served by Head Start – foster children, children of color, English language learners, etc. The benefits of intervention with those children and families are still clear at 3rd grade.

Fade-out vs. Catch-up, and the quality of public schools
As Nobel Prize-winning economist James Heckman wrote, “It (the fadeout argument) also overlooks the fact that many Head Start children move from a nurturing early education environment into a low-quality elementary school. Gains made in early childhood education must be sustained with quality education.” What we’re seeing in the Impact Study is a ‘flattening’ effect, as Head Start children don’t progress as fast as expected, and control group children catch up. Unfortunately, most of these children end up in low-quality public schools.

Head Start prepares children for school, and it does it effectively. Once schools take over it is their responsibility to keep the momentum going. Professor Rucker Johnson, a researcher from UC Berkley found that Head Start children who attend better quality schools end up with lasting gains compared to children who attended low-quality schools. [link]

Head Start Works
There is a lot of politically-charged discussion around this flawed study, despite dozens of other studies that show very different results over the last 20 years. A few of the most recent include:

David Deming, Harvard 2009: Head Start programs realize 80% of the benefits of the model Early Childhood Program studies (Perry, Abecedarian, Chicago Parent-Child Centers) at half the cost. [link]

Parisi/Grice, Mississippi State University, 2013: The MSU study shows significant short term gains, tracking more than 2300 Head Start students. It found that compared to the control group Head Start students were 18% less likely to be retained and 26% less likely to need special education services. In terms of cognitive skills, the Head Start children were 2.34 times more likely to be proficient in language and writing than the comparison group, and 2.1 times more likely to be proficient in math. [pub. pending]

Zhao, Huafang; Modarresi, Shahpar, 2009: Children that enrolled in a full day Head Start program were 66 percent less likely to need special education services in kindergarten than their peers. The Head Start children averaged 3.7 hours of special education services per week in kindergarten, compared to 9.8 hours for students that did not attend full day Head Start. [link]

Head Start Continues to Improve: CLASS Scores Consistently Get Better
Head Start is continuing to improve. The Obama Administration has implemented a new accountability initiative to replace poorly performing grantees and current Head Start agencies are expected to do even more for children and families. And the results are already paying off. Head Start recently adopted a new assessment tool called CLASS to measure teacher-child interaction and we are happy to report that CLASS scores continue to steadily improve, indicating greater child outcomes.