SHANKER BLOG

Underlying Issues In The DC Test Score Controversy

In the Washington Post, Emma Brown reports on a behind the scenes decision about how to score last year’s new, more difficult tests in the District of Columbia Public Schools (DCPS) and the District's charter schools.

To make a long story short, the choice faced by the Office of the State Superintendent of Education, or OSSE, which oversees testing in the District, was about how to convert test scores into proficiency rates. The first option, put simply, was to convert them such that the proficiency bar was more “aligned” with the Common Core, thus resulting in lower aggregate proficiency rates in math, compared with last year’s (in other states, such as Kentucky and New York, rates declined markedly). The second option was to score the tests while "holding constant" the difficulty of the questions, in order to facilitate comparisons of aggregate rates with those from previous years.

OSSE chose the latter option (according to some, in a manner that was insufficiently transparent). The end result was a modest increase in proficiency rates (which DC officials absurdly called “historic”).

I don’t have a particularly strong opinion about how OSSE should have proceeded. I can see it going either way. Setting cut scores is as much a political and human judgment call as anything else.

The controversy surrounding this decision, however, is both ironic and instructive. My understanding is that the actual scale scores are comparable between years (albeit imperfectly), even with the changed tests. The issue, again, is where to set the minimum score above which students are called advanced, proficient, etc., which in turn determines the rates for each of these designations (with proficiency rates getting the most attention). Unfortunately, OSSE doesn’t report anything except the rates. This severely limits the value of their annual testing results as reported to the public.

In other words, much of this controversy is about choosing between two cut score configurations, both of which present the data in an incomplete manner. Changes in the actual average scores, though still cross-sectional and therefore not "progress" measures, arguably provide a better sense of the change in performance of the typical student than either of the cut score proposals that are on the table. Now, to be clear, OSSE would have had to make the cut score decision regardless -- i.e., for the purposes of their accountability systems -- and the rates can be useful too (particularly for looking at performance in a given year).

Nevertheless, had the scale scores been available, the public would not have been forced to rely solely on the rates, which depend heavily on the specification of cut scores. This whole situation is a perfect illustration of why OSSE should release scores and rates every year.

In addition, I have to take issue with part of district officials’ justification for the decision. As mentioned above, their first claim was that changing the scale would have complicated accountability systems. This is basically true; there would have been easy ways to address it, though DC may have had to submit an amendment to its ESEA waiver.

But the other justification for the decision was described in the story as follows:

The OSSE made an unwritten commitment years ago to maintain that trend line as a way to judge progress and the effectiveness of reform efforts, said Jeffrey Noel, who oversees testing at the agency.

These types of statements are truly disturbing. In short, to whatever degree the decision was based on this “commitment," it is based on false premises.

As we’ve discussed countless times, changes in cross-sectional proficiency rates are not measures of “progress” (they don’t follow the same group of students over time), nor are they valid as evidence that any policy or policies are “effective." And the situation is even shakier for DCPS, where the student population is highly mobile, both for “natural reasons” (e.g., moving) and due to charter school transfers.

Every year, DC officials, as well as their counterparts in other states and districts, make these kinds of claims. And every year they are wrong in doing so. The reforms may be working, or they may not be. This is not evidence either way.*

And the same would have been true had OSSE decided to use the alternative cut scores, which, again, would have required their reporting lower proficiency rates in math. No doubt this would have resulted in more public outcry that the DC reforms are a failure.

Such an outcry would have been both unfair and inaccurate even if the tests hadn’t changed, but this year, it would have been especially ridiculous. If the tests are scored with higher standards, then aggregate results are going to look worse than they would had the standards remained constant. And, in either case, it is not valid policy evidence.

In other words, DC officials seem to be trying to evaluate their reforms using inappropriate evidence, and, perhaps, avoid criticism that is no less inappropriate.

Until the general public understands that cross-sectional proficiency rate changes are truly poor measures of student progress, and that raw changes, whether cross-sectional or longitudinal, cannot tell us much about whether policies are “working," officials and advocates on both “sides” will keep firing empirical blanks at each other, in a battle that neither can actually win.

- Matt Di Carlo

*****

* It bears mentioning that the rates have been largely flat for three years, and increased only modestly this year. Thus, by OSSE's own standard of "evidence," the conclusion would have to be that the reforms are failing. Fortunately for them, this is not a valid approach.

Authors

Comments

It always comes back to the correction factor in things like this and charter schools. You have to have a correction factor which makes things equivalent or you have nothing to compare. Otherwise how do you follow through time. Kind of like let's change to a 5.0 from a 4.0. Stupid and senseless as now there is no real comparison without a correction factor to make all even in reality not fantasy of smoke and mirrors.

Upcoming Events

Elected officials seeking to diminish the pensions of public sector employees have argued that they are responding to a fiscal crisis. Is this crisis real or contrived? March 11, noon-2. More information and registration.

Teaching and learning are not primarily individual accomplishments but rather social endeavors that are best achieved and improved through trusting relationships and teamwork. Yet, most policies focus on improving the individual capacities of teachers. What is to be gained from approaches that strike a balance between human and social capital?

There is concern that, as the U.S. population and student body is growing more racially and ethically diverse, the teacher workforce does not yet reflect this diversity. In fact, diversity should go beyond having more black and brown teachers in front of students. Diversity is also about equipping all teachers (regardless of race) to work with heterogeneous classrooms and diverse schools.

Our Mission

The Albert Shanker Institute, endowed by the American Federation of Teachers and named in honor of its late president, is a nonprofit, nonpartisan organization dedicated to three themes - excellence in public education, unions as advocates for quality, and freedom of association in the public life of democracies. With an independent Board of Directors (composed of educators, business representatives, labor leaders, academics, and public policy analysts), its mission is to generate ideas, foster candid exchanges, and promote constructive policy proposals related to these issues.

This blog offers informal commentary on the research, news, and controversies related to the work of the Institute.

DISCLAIMER

This web site and the information contained herein are provided as a service to those who are interested in the work of the Albert Shanker Institute (ASI). ASI makes no warranties, either express or implied, concerning the information contained on or linked from shankerblog.org. The visitor uses the information provided herein at his/her own risk. ASI, its officers, board members, agents, and employees specifically disclaim any and all liability from damages which may result from the utilization of the information provided herein. The content in the Shanker Blog may not necessarily reflect the views or official policy positions of ASI or any related entity or organization.