So in 2007, DCPS officials excluded 14% of students from 4th Grade NAEP testing, and in 2011 that figure fell to 3% (the inclusion for all students standard in 2011 was 95%). In 2007, DCPS stood far out of compliance, but came well within compliance in 2011. This is all well and fine, other than the fact that it complicates our ability to assess the recent history of DC NAEP gains.

In order to get a clearer picture on this, I decided to run 4th Grade NAEP scores for students outside of ELL or special education programs. This should minimize the impact of inclusion policy changes. Examined in this fashion, you get the following results:

Recall that the unadjusted total scores for 4th grade reading jumped from 197 in 2007 to 202 in 2009 but dropped back a point to 201 in 2011. That is a four point gain in four years, which ranks in meh territory. Given Figure 1 above, I am not exactly inclined to trust those scores, and in fact out second table tells quite a different story: general education students in DC made a 10 point gain between 2007 and 2011 on 4th grade reading. Ten points approximately equals a grade level worth of progress, so it is fair to say that DCPS general education 4th graders were reading approximately as well as 2007 general education 5th graders. Ten points ranks as the largest reading gain in the nation during this period for these students. Mind you, a 209 score for non-Ell and non-special ed students is still terribly low. Only gains will get DC out of the cellar, however, and DC banked solid gains during this period.

If you combine 4th and 8th grade reading gains for general education students, and only look at Free and Reduced lunch eligible students for a bit of socio-economic apples to apples, here is what you find:

DC students had the largest general education 4th grade reading gains in the country, and tie for first in the combined 4th and 8th grade reading gains. The District of Columbia, in short, made very substantial reading gains during the 2007-2011 period.

Share this:

Related

This entry was posted on Thursday, January 12th, 2012 at 6:54 pm and is filed under NAEP. You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.

NAEP is a low-stakes test that evaluates random samples of students in entire states (and in this case DC) but not individual schools and certainly not individual teachers. Incentives for the sort of cheating alleged in the link you provided don’t exist, and I haven’t seen any allegation of cheating on NAEP in DC.

Glad to see you are familar with the NAEP data explorer. If you will take a look at figure 1 above (or delve into the appendix of the 2011 NAEP reading report) you will see that the 2007 13% of students were excluded from the 2007 8th Grade reading NAEP, while only 3% were excluded from the 2011 exam. The samples, in short, were very different.

If you will go back to your data explorer and track the 8th grade reading progress for DC general education students from 2007 to 2011, you’ll find that such students scored seven points higher in 2011 than in 2007. If you check the national average for those same sorts of students, you’ll find the progress was three points. We wouldn’t expect the ELL and Special Ed students to show better scores in 2011 than in 2007 given that approximately four times as many students were excluded in 2007.

Exactly- vanishingly few of the 14% of students excluded on the basis of ELL and SPED in 2007 were likely to be high income Anglos in DC. The high exclusion rates before 2011, especially the rate in 2007, probably played a substantial role in inflating test scores for low income students.

Look at your Hispanic numbers between 2005 and 2007- a 13 point gain in two years? I suspect that the doubling of the exclusion rate between 2005 and 2007 played a substantial role in driving that gain.

This is from the NAEP report:
Score Gaps for Student Groups
 In 2011, Black students had an average score that was 64 points lower than White students. This performance gap was not significantly different from that in 2002 (60 points).
 In 2011, Hispanic students had an average score that was 51 points lower than White students. This performance gap was not significantly different from that in 2002 (55 points).
 In 2011, students who were eligible for free/reduced-price school lunch, an indicator of low family income, had an average score that was 46 points lower than students who were not eligible for free/reduced-price school lunch. This performance gap was wider than that in 2002 (25 points).

Your NAEP report references would be even more impressive if they were relevant to the topic at hand. Just as a reminder, that topic is not whether DCPS continues to have massive academic problems (it does) but rather whether or not it has been improving, and what data should be used to make such judgements.

Matt,
“A Closer Look at DCPS NAEP Scores”, one “positive trend is that: In 2011, Black students had an average score that was 64 points lower than White students. This performance gap was not significantly different from that in 2002 (60 points).

Also, looking at your second graph, we see the same upward trend in the NAEP, excepting from 2002-2003.

And, if I am correctly reading between your lines, the unstated supposition is that the general lackluster performance in DCPS under the stewardship of Michelle Rhee is due to the increase in the percentage of SD and ELL students taking the test

If you will click the first link, you will see that DC came in 4th place nationwide in combined gains on the four main NAEP exams for the 2007 to 2011 period for FRL kids even before any considerations of the substantial changes in exclusion rates.

In all of the debates back and forth over Rhee’s record, what effect does it have on DCPS that so many kids have migrated to charter schools during the 2000s? We’re always told that charter schools are getting the “most motivated” kids, and if that’s true in DC (the people making that claim never feel the need to provide evidence for it), wouldn’t that be a huge negative pressure on DCPS scores on NAEP?

You are correct, if the assertion were true. Since we’ve seen very large improvements in DCPS scores since charter schools were introduced, I’d dare to guess that there either isn’t much to the assertion and/or the competitive effects drown them out.

Of course there are plent of other things going on, but one would struggle to find evidence that charter schools have had a negative impact on the academic performance of DCPS.

[…] Ladner says, essentially, that increased numbers of illegally excluded ESL students brought heroic DC chancellor Rhee to grief – that is, when NAEP no longer allowed them to be excluded. Making their scores count made his heroine look bad, he concludes. […]

[…] based on a few years of inconclusive data we can determine that a reform has “failed.” There is mixed evidence regarding the results of the changes made in D.C., but as Matt DiCarlo pointed out in a takedown of […]

[…] progress is finally being made in 8th grade reading by both low-income and black students. Perhaps Jay Greene is correct and gains were slow to register because, in 2007, the NAEP sample started to include […]