Through the lens of social science, eduwonkette takes a serious, if sometimes irreverent, look at some of the most contentious education policy debates. (Find eduwonkette's complete archives prior to Jan. 6, 2008 here.)

A Course in Statistics at Columbia: $3186. The NYC DOE's Comment on Confidence Intervals: Priceless!

You have to hand it to the New York City Department of Education's Department of Assessment and Accountability. You really do.

Yesterday morning, the NY Times reported that the DOE will now distribute teacher value-added reports to teachers and principals.* Here's the thing - the value-added reports don't just report that a teacher performs at the 65th percentile, the 25th percentile, etc. Instead - as they should - the DOE reports a confidence interval around each teacher's value-added to represent the uncertainty of the estimate. And unsurprisingly, these confidence intervals are quite wide. A 65th percentile teacher in this example has a confidence interval ranging from the 46th to the 84th percentile. In providing this range, the DOE is formally acknowledging that we do not know if this is a below average, average, or above average teacher.

So imagine my surprise when DOE Accountability Czar Jim Liebman popped over last night to criticize skoolboy's post showing that the report cards are flawed because they don't address the uncertainty of the estimates of school progress. To Liebman, this is so much sound and fury.

Morning Jim Liebman, it's my pleasure to introduce you to Evening Jim Liebman. You two are going to be BFF. Or frenemies. It's hard to tell. Though you are the same person, you have remarkably conflicting views about what role uncertainty should play in accountability reports. Maybe it's the coffee.

What beats me is why the DOE reports confidence intervals in their teacher reports if uncertainty is just a concern of pesky bloggers like eduwonkette and skoolboy. By providing confidence intervals on the new teacher reports, the DOE basically concedes that their school grading system is bunk. Oops!

* Sidenote: I question the wisdom of giving teachers and principals information that is likely to be inaccurate, based on all of the reasons articulated here (in short, New York's testing schedule is problematic), but that is for another post.

Categories:

4 Comments

I understand the human face of the whole thing. I believe that the biggest problem with value add is that it is most useful and least reliable at the teacher/classroom level. In my limited understanding this makes sense because the n size is so small.

On the other hand, it seems to exist because so many at the teacher level have wanted to prove, or to have someone understand, that they are really doing a super job, but their kids just start off really far behind. Now this is unlikely to be universally the case when we look at the accrued gap by the end of high school (or where kids are when they drop out).

So, while the building level may be the point of best balance between utility and reliability, I can understand the desire for folks in a building to know how things look at the micro level (didn't you want to know your IQ score?) But I would be loath to put the things out without an indication of the size of the possible variance.

But the wider the confidence interval, the higher the probability that the teacher's job performance actually falls in the reported range. So we know from the example you present in this post that the employee in question is a teacher, and that his or her performance has been subjected to an in-house quantitative accountability assessment. To paraphrase Terry McAuliffe, excellent news for the DOE!

As I've been saying for years: Sure, there are a lot of good principals, and they are critical players in school improvement, but I'm skeptical about giving principals a free hand to hire & fire teachers at will. What divine envoy came down from Heaven & gave principals the gift of omniscience?