Monday, August 16, 2010

Identifying Good and Bad Teachers--By Name

A Times analysis, using data largely ignored by LAUSD, looks at which educators help students learn, and which hold them back...

Yet year after year, one fifth-grade class learns far more than the other down the hall. The difference has almost nothing to do with the size of the class, the students or their parents.

It's their teachers.

With Miguel Aguilar, students consistently have made striking gains on state standardized tests, many of them vaulting from the bottom third of students in Los Angeles schools to well above average, according to a Times analysis. John Smith's pupils next door have started out slightly ahead of Aguilar's but by the end of the year have been far behind...

The Times used a statistical approach known as value-added analysis, which rates teachers based on their students' progress on standardized tests from year to year. Each student's performance is compared with his or her own in past years, which largely controls for outside influences often blamed for academic failure: poverty, prior learning and other factors.

Though controversial among teachers and others, the method has been increasingly embraced by education leaders and policymakers across the country, including the Obama administration...

Many teachers and union leaders are skeptical of the value-added approach, saying standardized tests are flawed and do not capture the more intangible benefits of good instruction. Some also fear teachers will be fired based on the arcane calculations of statisticians who have never worked in a classroom.

The respected National Academy of Sciences weighed in last October, saying the approach was promising but should not be used in "high stakes" decisions — firing teachers, for instance — without more study.

No one suggests using value-added analysis as the sole measure of a teacher. Many experts recommend that it count for half or less of a teacher's overall evaluation.

I didn't know this was going on. Very interesting. So what did they find out?

On visits to the classrooms of more than 50 elementary school teachers in Los Angeles, Times reporters found that the most effective instructors differed widely in style and personality. Perhaps not surprisingly, they shared a tendency to be strict, maintain high standards and encourage critical thinking.

But the surest sign of a teacher's effectiveness was the engagement of his or her students — something that often was obvious from the expressions on their faces...

Aguilar, a stocky 33-year-old who grew up in the area, is no showman. Soft-spoken and often stern, he doles out praise sparingly. It only seems to make his students try harder.

"Once in a while we joke around, but they know what my expectations are," he said. "When we open a book, we're focused."

It seems to work: On average, his students started the year in the 34th percentile in math compared with all other district fifth-graders. They finished in the 61st. Those gains, along with strong results in English, made him one of the most effective elementary school teachers in the district...

It was only 11a.m., and already it had been a tough day: Three of Smith's students were sitting in the principal's office because of disruptive behavior. All were later transferred permanently to other classrooms.

In an interview days later, Smith acknowledged that he had struggled at times to control his class.

"Not every teacher works with every kid," said Smith, 63, who started teaching in 1996. "Sometimes there are personality conflicts."

On average, Smith's students slide under his instruction, losing 14 percentile points in math during the school year relative to their peers districtwide, The Times found. Overall, he ranked among the least effective of the district's elementary school teachers...

Even at Third Street Elementary in Hancock Park, one of the most well-regarded schools in the district, Karen Caruso stands out for her dedication and professional accomplishments.

A teacher since 1984, she was one of the first in the district to be certified by the National Board for Professional Teaching Standards. In her spare time, she attends professional development workshops and teaches future teachers at UCLA.

She leads her school's teacher reading circle. In her purse last spring, she carried a book called "Strategies for Effective Teaching."

Third Street Principal Suzie Oh described Caruso as one of her most effective teachers.

But seven years of student test scores suggest otherwise.

In the Times analysis, Caruso, who teaches third grade, ranked among the bottom 10% of elementary school teachers in boosting students' test scores. On average, her students started the year at a high level — above the 80th percentile — but by the end had sunk 11 percentile points in math and 5 points in English.

Calling teachers out this way, by name, is brutal. Oh, I'm not saying it isn't necessary or important, but publishing their names like this in the paper is brutal. Joanne says that "[t]he teachers named as ineffective — which had to be humilating — said they’ll look for ways to improve." Is humiliation really what it takes?

I've been on the value-added bandwagon for years. I hope it takes off and is used as a tool to help teachers hone their craft. Combined with what Superintendent Mike Miles is doing in Harrison District 2 in Colorado Springs, this is how you make teaching a profession, and this is how you give children the best opportunity to learn.

I don't think that humiliation is required, but it does seem like the teachers may not have know how well they were doing. From the article (as quoted by Joanne), "On average, her students started the year at a high level — above the 80th percentile — but by the end had sunk 11 percentile points in math and 5 points in English.

Caruso said she was surprised and disappointed by her results, adding that her students did well on periodic assessments and that parents seemed well-satisfied."

Surprised? This sounds like she didn't *know* that her kids were going relatively backward. *Someone* at that school isn't doing their job ... either Ms. Caruso or her boss.

But ... we don't know what a typical range of performance is going to be. If these tests are norm-referenced (I don't think the article says), then for every point one student goes up, another student is going to lose a point. That's how norm-referenced tests work. I'd expect some "noise" in such a system, and we really need to know how noisy things are. We also need to know how random the student assignments are.

So ... I would like to see data like this, but I'd like it to be statistically valid. It seems like the school either doesn't collect or doesn't do anything with the data. Which is very sad. But we also don't know quite what to do with the data that the newspaper has gathered, either :-(

Mark .. . that's actually not how norm based rest work. It's a common misconception. They actually establish a norm when the test is written, based on the reference groups the test is given to. Once that norm is established, it stays until a new test is written. So, it is possible, in theory, for every student to score in the 99th percentile *compared to that original reference group.* won't happen, but it is possible for everyone to improve.My problem with this study is that very few, if any, of the standardized tests are of any importance to the students. They can opt out of them, as many top end students do, or they can fill in random answers without penalty. As reasonable as this approach may seem, and as obvious as the conclusions may be, you can't use an instrument that has no meaning to students to measure teachers. It would be an easy legislative fix to change this, and it would shut me up . . . except for my additional belief that standardized tests aren't a good measure of student learning.

It is/was my understanding that these sorts of standardized tests come in two basic flavors: 1) Norm referenced, which is a fancy way of saying "curved", and 2) Criterion referenced, which is what you described (define some constant target and see how you do relative to it).

In type (2), everyone can improve at the same time. In type (1), they can't.

You are actually correct about the terms; the problem is that the people who administer the tests are wrong in their description of them. I know this only because I asked the same question you brought up, because we were being fed the "everyone needs to improve" line. The answer is that there is an initial norm that is established, which IS a curve. That norm then BECOMES the criterion on which future test takers are judged. At least that's how California's are done.