Question raised about new KIPP study (with update)

By
Valerie Strauss

UPDATED AT BOTTOM
No sooner had a new study been released saying that middle school students in the charter school network called Knowledge Is Power Program significantly outperform their public school peers on reading and math tests than critics raised an important question about the report.

Mathematica Policy Research issued the study yesterday, my colleague Bill Turque reported, about the KIPP network, which now has 82 schools that serve children from low-income households. Seven are in the District, including three middle schools -- KEY, WILL and AIM academies -- which are among the highest performing on the DC-CAS standardized tests.

The study looked at 22 KIPP middle schools including AIM and KEY, and discovered that by seventh grade, half of them showed growth in math scores equal to an additional 1.2 years of school. Reading gains for KIPP were not as dramatic but still significant, the researchers reported, reflecting an additional three-quarters of a year of growth.

However, an initial analysis of the study conducted by Professor Gary Miron of Western Michigan University found that attrition data appears to have been misrepresented.

A key finding of the study is that attrition at KIPP schools is not much different from attrition at comparable conventional public schools. This finding is important because past research about KIPP suggests that selective attrition -- struggling students disproportionately leaving, with more successful students staying and then scoring well on tests -- may give KIPP a substantial boost.

However, an initial analysis of the report by Professor Gary Miron of Western Michigan University concludes that this initial study report misrepresents the attrition data.

According to Miron, "While it may be true that attrition rates for KIPP schools and surrounding districts are similar, there is a big difference: KIPP does not generally fill empty places with the weaker students who are moving from school to school. Traditional public schools must receive all students who wish to attend, so the lower-performing students leaving KIPP schools receive a place in those schools."

In contrast, Miron explains, "The lower performing, transient students coming from traditional public schools are not given a place in KIPP, since those schools generally only take students in during the initial intake grade, whether this be 5th or 6th grade."

The KIPP study’s description of attrition only considers half the equation, when comparing KIPP schools to matched traditional public schools. The researchers looked at the attrition rates, which they found to be similar - in the sense of the number of students departing from schools. But they never considered the receiving or intake rate. Even though the researchers agree that the students who are mobile are lower performing, they do not take into account the reality that KIPP schools do not generally receive these students.

Professor Miron conducted his own quick analysis, using the Common Core database, and concluded that there is a 19% drop in enrollment in KIPP schools between grades 6 and 7 and a 24% drop in enrollment between grades 7 and 8. (This analysis only included KIPP schools that had enrollments in all three grades). In comparison, traditional public schools in these grades maintain the same enrollment from year to year.

While Miron’s review questions about the validity of this report’s particular findings, this is solely because of this single problem. In other ways, he found the study to be rigorous and high quality, promising to be even better in subsequent years of the evaluation. Those future reports can, and Miron hopes, will address the questions raised here and also about students retained in grade.

Importantly, Miron is also not saying that the KIPP schools do poorly. Those schools provide about 50% more instructional time and place rigorous demands on students and their families.

"We have every reason to believe that KIPP likely does a great job with the low-income students of color who wish to attend and who have relatively supportive parents who can do things like drive them to Saturday school," Miron says.

But he does question whether this is a viable model for larger numbers of students, and he also wonders whether the different departure and receiving policies may make matters worse for students who are left behind or who later leave KIPP schools. How would the KIPP model work if students who cannot handle the rigorous KIPP demands could not move to conventional public schools?

In a separate e-mail, Professor Kevin Welner, director of the Education and Public Interest Center (EPIC) at the University of Colorado at Boulder, told me:

I hope Gary’s raising of this important issue is received as it was intended. This was a difficult study to undertake, and in many ways the Mathematica researchers did an excellent job. Gary’s comments below make that clear. There seems to be a tendency among advocates for school models such as this to react very defensively instead of welcoming criticism and finding ways to address genuine issues. In this case, Gary focused in on the validity of a key claim -- an attrition claim that has been trumpeted by KIPP’s advocates -- which is substantially undermined by the researchers’ failure to account for the differences in "receiving" policies. This was almost surely a source of bias, resulting in a more positive result for KIPP.

This doesn’t mean that KIPP educators are not doing a good job. It doesn’t mean that parents shouldn’t choose to send their children there. It doesn’t mean that donors should not support KIPP. It just means that this interim evaluation report has a weakness that should be addressed in later reports, and it means that the tentative findings presented in the report should be taken with a grain of salt.

UPDATE:
Welner sent this update today:

I want to correct something I wrote yesterday. "This was almost surely a source of bias, resulting in a more positive result for KIPP." I had misunderstood Prof. Miron, so "bias" was the wrong word to use here. As your commenters "amr11" and "jane100000" correctly noted, the measurements were not directly affected by the "receiving" differences described by Prof. Miron. His contentions are, instead, that:

(1) Researchers should be aware of the substantial indirect effect of these receiving differences -- on a school's ability to create a healthy learning environment. Selective attrition has a net impact on the capacity of sending and receiving schools. The research should account for the apparent situation whereby KIPP schools are only on the 'benefiting' end of that impact.

(2) Grade retention differences may also have played an important role, and the researchers would be wise to consider their impact on test scores and on selective attrition. A strong finding from education research is that grade retention is strongly associated with later dropout rates. Among comparable students, a holding one back in grade will, on average, increase his relatively likelihood of later dropping out. Also, grade retention changes the makeup of takes a given test (e.g., the 7th grade state standardized exam) in a given year.

Regarding the attrition issue, one suggestion that Prof. Miron noted to me was that the researcher may want to analyze the variance in test score data for the different grade levels at the sending (KIPP) and receiving (traditional public) schools. If selective attrition is occurring in the way he suspects, the KIPP variance will be substantially reduced (with the lower-achieving students largely disappearing), while the traditional public school variance will hold basically constant (with leaving students who leave generally being replaced by comparable entering students).

A final suggestion: Almost all high-quality studies (and this one certainly qualifies as that) include a section clearly explaining the study's limitations. For some reason, this one does not include such a section. If it had, these sorts of concerns could be expressed in the study/report itself. The Mathematica team is working with a good, if not great, dataset and has the talent to answer all these questions or clearly explain when their ability to answer them comes up short.

All educational research should be taken with a "grain of salt." The sponsoring agency has a vested interest in proving a point to their donors/supporters. Bias is very difficult to completely eliminate.So thanks to those who choose to critique these studies and point up their defects/limitations. It's our duty to be skeptical in a democracy!

"Professor Miron conducted his own quick analysis, using the Common Core database, and concluded that there is a 19% drop in enrollment in KIPP schools between grades 6 and 7 and a 24% drop in enrollment between grades 7 and 8."

So how great a drop in enrollment is there between 6th and 7th - 45%? I don't know if it is accurate to add the drop to each other directly to find the attriction rate.

Another thing to remember is that KIPP does not take over an existing school and keep its population.
As noted in the article, one must apply.

It would be interesting to see what would happen if KIPP was to take over and run Stanton Elementary in Southeast. They would have to keep the current population, take any student who moves within its boundaries, and not be able to expel a student unless they engaged in physical assault.

And the KIPP schools in Indiana score at or below the state average:

Go to this web site to see the results:
http://www.doe.in.gov/istep/2010/index.html

jane100000- you're right, this criticism seems misguided and possibly based on a misunderstanding of the analysis. The analysis follows a set of students over time. Neither the KIPP group nor the comparsion groups has students added in to the analysis sample over time.

Prof Miron is probably right that this would be an issue that affects school-level performance if a district tries to transfer the KIPP model to public schools, but this article is written as if this concern invalidates the analysis. That's sloppy reporting.

@aby1: You can't add the percentages together. (It would be like a sale where something already 50% off gets marked down 50% again. You still pay 25%.) Imagine a KIPP school with 100 students starting 6th grade. With 19% attrition, there would be 81 students left to start 7th grade. With 24% attrition the following year, there would be about 62 students left to start 8th grade, for a 38% total attrition rate after two years. Not as bad as 44%, but something I'm sure KIPP schools want to significantly improve.

Whether you're for or against charters, the fact is that they exist and we need to continue to evaluate their performance and be prepared to replicate their methods when they show positive results. This to me is the big question about KIPP schools: would their emphasis on extended seat time, including Saturdays and summers, work for a school where the participants weren't opting into the program? Some KIPP schools are doing quite well, but the transferability and scalability of their results is still questionable.

Anyone who teaches in a school with a high mobility rate can attest to the effect of new students entering within the school year. This is a real problem when one considers the use of test scores to determine teacher effectiveness. I remember a few years back that my school had 35 new kids in the third grade (the first year of the MSA tests). The total number of third graders was 85. That's a lot of new kids--and that was at the beginning of the year. We continue to get new kids throughout the year right up to the last week of school. How can their learning be attributed to our school and their new teacher if they haven't been there? Of greater concern is the fact that some of these students move multiple times in the year. We have a new family this year with students in Kdg and 1st grade. Those kids had already been in 4 schools. How disruptive to their learning is that?

It would interesting to know how the rest of the curriculum at KIPP is set up; do these students have a good balance of other studies and activities such as social studies, P.E.,foreign language, art, music, field trips? Additionally, how else is the learning atmosphere fostered?
I've read about the discipline aspect,longer days, committed attitude and supportive parents.

Middle school students are notoriously changeable human beings due to that particular stage of their development, so it is important to know what sustains their ability to handle a longer day in schooled circumstances.

Michael,
I just finished reading Outliers by Malcolm Gladwell and he writes about KIPP and follows a middle school girl who attends KIPP and lives in poverty through long days and Saturday classes. It gives you a flavor of what goes on in that system. I too would like more detail. Right now I am reading Teach Like a Champion which gives a clear picture of the techniques used in another charter system: Uncommon Schools. This is quite an amazing book which has distilled the classroom techniques of Master teachers into short usable bites. I am a 20 year plus teacher - most in middle school and these techniques are wonderful, some familiar and some new, and immediately usable in my classroom. This is an example of the contributions that I think Charters should be making along with alternative schools for at risk students who cannot function well in a regular classroom.

My objection to Charters is that they have been taken over by Wall Street as a way to extract public school money through real estate investment and the development of private management companies whose members often also sit on the charter boards. There is rampant fraud going on here and little oversight. I read about it constantly in the papers here in Florida in the St Pete Times and there is even a blog that tracks Charter abuses nation wide at http://charterschoolscandals.blogspot.com/

I don't want to throw out the baby with the bath water but Charters need more oversight and since they only serve 5% or less of our children they cannot be the ony "answer" to the so called failure of the public schools.

Everything both professors say about the KIPP study sounds perfectly reasonable to me.

I have no doubt the KIPP model has value - it's such common sense that its features would result in higher academic achievement. Also, I think it's important to understand how a successful program works and not to misrepresent it.

How well publicized will this new finding be? I bet yesterday's press is all most people remember.

KIpp does sound good. Although if I understand this information correctly, KIPP does not accept new students to replace students who move away or who otherwise don't attend school. Do I have that right?

That's correct. The KIPP schools in the study do not accept new students beyond 6th grade. That means that their upper grade level students are those who have made it through 5th and 6th. Remember, these schools retain all students who are not at grade level.

I would love to see an analysis of achievement size effects as a result of retention. It would also be important for those considering moving this charter program to a district-wide setting to address how many students drop out due to the retention policies.

Downclimb math says we're looking at around a 38% drop from 6th grade to 8th if the percentages are correct...at least for this particular school. That's a pretty large drop ....meaning the 8th grade class is 38% smaller then 6th hence much easier to individualize and help catch up.

So if they started with 100 6th graders in 4 classes of 25 each by 8th grade it's 15-16 students per class. Thrity-eight of the 100 are gone. Hmmmm ....and these are 15-16 left are motivated with parents that helped them stick to a rigoruous, disciplined schedule of learning.

Please don't tell me they even consider comparing this to a regular low preforming public school. NOT!!!!...even close.

Stop the spread of Charter school management abuse. The Gulen movement now manages and operates 97 US Charter schools. Fetullah Gulen is described as the most dangerous Islamic Imman in the world. He operates over 600 schools worldwide. They are bringing uncredentialed teachers from Turkey to the USA to teach under HB-1 Visas that the US Tax payers are paying for. The indoctrination of American into the Islamic way of life with language, dance, food, etc. is pushed very hard. The mismanagement of money is HIGH with the numerous Gulen foundations and institutions money laundering and paying for trips to Turkey for the children to perform. DO YOUR RESEARCH, be a part of your child's education. You decide about the worldwide Gulen Movement.
http://www.charterschoolwatchdog.com
http://turkishinvitations.weebly.com/

lyn122 - KIPP wouldn't have classes of 15-16 8th graders. They wouldn't waste money paying extra teachers. They would combine the students into 2 classes of 30+ students.

KIPP schools typically don't accept students for the upper grades because the students can't keep up unless they've had a quality education in 5th and 6th grade. At the KIPP school I taught at, we would accept older students, but the principal interviewed them carefully and gave them entrance exams to make sure they could work at grade level. There's no sense in setting students up for failure.

"At the KIPP school I taught at, we would accept older students, but the principal interviewed them carefully and gave them entrance exams to make sure they could work at grade level"

really? this seems to go against KIPPs policy of not being selective on the front end. I don't blame the principal for doing this and you're right - it's only fair to the kids -- but then it's a bit unfair to compare KIPP to public schools who can't use this practice.