This entry was posted
on Thursday, October 21st, 2010 at 7:18 am.
You can follow any responses to this entry through the RSS 2.0 feed.
You can skip to the end and leave a response. Pinging is currently not allowed.

Yes, it’s extremely “ironic” that the unions believe both that value-added ratings should not be the basis of teacher evaluations AND that these ratings should not be reported publicly. You’ve really exposed them!

Of course, this isn’t their “own mousetrap” at all, but two bad choices imposed upon them. A better analogy is that teachers are being charged protection money: “Nice little reputation you have there, teacher. Shame if anything happened to it. Maybe you should agree to let us determine your compensation and job security based on this inaccurate metric…”

Excellent! I’m a rodent! Putting this information in the paper is going to do SO much towards improving education. I wish my fellow teachers would remember that test scores are the only thing that matter, and teach accordingly. The TDRs are 100% accurate- if it says you’re lousy, you’re lousy.

Why do you think this entry is an example of “contempt and ill will towards teachers”? Be explicit, 100 words or less.

Guy:

I think the irony is that the teacher unions are clamoring in opposition to both, and will likely succeed in preventing neither. They could instead be reasonable and realize that value-added is not going away, and support teachers with that in mind.

Your analogy is also much worse. The reputation a teacher has currently has not been gained through any kind of stringent, objective measure. Anything that helps change this aspect in the slightest, and helps to better understand teacher effectiveness, should be a goal at this point.

Rodent:

“I wish my fellow teachers would remember that test scores are the only thing that matter”

Chris: I do not think that word (‘irony’) means what you think it means.

Do you have any evidence at all that a typical NYC teacher’s reputation among his/her peers, students, and parents is a less accurate measure of teaching effectiveness than his/her value-added rating? Be explicit and specific, but you may use more than 100 words if needed.

There’s nothing ironic about it (and thanks Linda for stealing my line). The union already AGREED to include the VA data in evaluations, as long as the data remained confidential. So, yeah, if I were the union, I’d be a little peeved.

And Linda, you have to remember that according to Andywonk, it’s always the union’s fault. But here’s another example of him twisting the facts to fit his own version of reality.

When superintendents are willing to subject themselves to an “objective” performance measure in which the odds of being mis-identified as effective or ineffective is between 30% and 50%, then teacher unions should agree to value-added scores being used and published.

Chris, maybe no one has ever explicitly argued that test scores are the only thing that matter, but Mayor Bloomberg and Chancellor Klein should just admit that’s the only thing that matters to them. Their actions show it. They wanted 100% of our evaulations based on test scores, not the 40% that was eventually agreed to. That, to me, sounds like they put scores above everything else.

If you’re going to try to correct me, at least get your definitions straight:

“Irony: incongruity between what is expected to be and what actually is, or a situation or result showing such incongruity”

This is an example of irony, since the teacher unions might have a better chance at protecting their teachers’ interests if they were to focus on preventing their members’ personal information to be shared along with their evaluations, instead of becoming bitterly embroiled in a war against value-added data as a whole, one that they will inevitably lose.

“Do you have any evidence at all that a typical NYC teacher’s reputation among his/her peers, students, and parents is a less accurate measure of teaching effectiveness than his/her value-added rating?”

The onus is on you to prove that the reputation these teachers have is “accurate”, but sadly it is categorically impossible that you (or anyone) knows the accuracy of any measure of a teacher’s effectiveness. We probably will *never* get a perfect understanding of teacher effectiveness that is 100% accurate, and even if we were to suspect we’ve reached that point, we’d never be able to double-check the numbers since we don’t have a standard for every teacher to compare them to.

This is the problem associated with employee evaluations, and one that you will find in EVERY workplace on the planet.

Fortunately, the goal was never “100% accuracy”, as you incorrectly attempt to attribute to me. The goal is a more “stringent, objective” type of measure of teacher effectiveness. It doesn’t exist now in districts that presume 2 teacher evaluations a year adequately qualifies as either “stringent” or “objective”, and by definition it absolutely CANNOT exist within the evaluative power of “peer/student/parent reputation”, since there is no such metric that all peers/students/parents stringently grade their teachers on.

“When superintendents are willing to subject themselves to an “objective” performance measure in which the odds of being mis-identified as effective or ineffective is between 30% and 50%, ”

The confidence intervals I’ve seen approach 30%, and higher typically when there’s very little data to work with. Given the fact that the only arguments for using VA data suggest they be used as an added lens to an evaluation system comprising other methods, and that everyone in the eduworld already knows that these data are estimates, this counterexample is not very helpful. Yes, they should be willing, although they probably have it hard enough since they are generally blamed for the test scores of students throughout the entire district regardless of whether VA data is utilized or not, and sometimes regardless of if there were improvements or not. Go ask Rhee how that worked out for her.

*****

One mouse among many:

“Chris, maybe no one has ever explicitly argued that test scores are the only thing that matter”

Then don’t imply otherwise. We have enough straw man arguments in these comment threads as it is.

*****

Anne:

Yes, I have read all of those and more. The trouble with this discussion is always the premise that either side is arguing from. If we both accept the premise that, “Value-added analyses give an objective and exact measure of teacher effectiveness,” then yes, you would have a point with suggesting that analyses of these data do not work well. 30% error can be quite problematic in this regard. The premise itself, however, is incorrect, and one that is absent in all arguments I’ve seen for value-added analysis.

The correct premise is the following: “Value-added analyses give an objective estimate of teacher effectiveness”. An uncertainty of 30% is low enough to determine which teachers tend to be great, and which teachers tend to be below average. Principals could then use this data *along with other evidence of teacher effectiveness* to evaluate their teachers.

Every pro-VA argument I’ve seen has reasoned from this latter premise to suggest that VA data be used to supplement an evaluation system, and yet I’ve seen no compelling counterarguments to this reasoning. Is there any you can offer?

This is the terminally lame argument of Chris and the entire blame-the-teacher crowd. OK, now we admit that “value added” metrics don’t work. Never mind. Nothing else works either. We NEED to undo the power of the unions . . . and put it in the hands of people like me (who clearly have something else than the common good in mind.)

“The union already AGREED to include the VA data in evaluations, as long as the data remained confidential. ”

I can’t find a citation for this one, can you please provide it? The Union’s tone when addressing VA data is not so pleasant to suggest they’d agree to use it for evaluations:

““We have invalid test scores, going into an unreliable formula, which equals a bad result,” [UFT president Michael] Mulgrew said.”

I’ve found some articles on 20-25% of the evaluation based on state tests, but have not seen any mention of the unions agreeing to use VA data.

[3 links of mine removed; see comment above for links after it gets moderated]

*****

The Anti-Chris:

(Why do all my nemeses on this blog have boring names?)

“When superintendents are willing to subject themselves to an “objective” performance measure in which the odds of being mis-identified as effective or ineffective is between 30% and 50%, ”

The confidence intervals I’ve seen approach 30%, and higher typically when there’s very little data to work with. Given the fact that the only arguments for using VA data suggest they be used as an added lens to an evaluation system comprising other methods, and that everyone in the eduworld already knows that these data are estimates, this counterexample is not very helpful. Yes, they should be willing, although they probably have it hard enough since they are generally blamed for the test scores of students throughout the entire district regardless of whether VA data is utilized or not, and sometimes regardless of if there were improvements or not. Go ask Rhee how that worked out for her.

*****

One mouse among many:

“Chris, maybe no one has ever explicitly argued that test scores are the only thing that matter”

Then don’t imply otherwise. We have enough straw man arguments in these comment threads as it is.

*****

Anne:

Yes, I have read all of those and more. The trouble with this discussion is always the premise that either side is arguing from. If we both accept the premise that, “Value-added analyses give an objective and exact measure of teacher effectiveness,” then yes, you would have a point with suggesting that analyses of these data do not work well. 30% error can be quite problematic in this regard. The premise itself, however, is incorrect, and one that is absent in all arguments I’ve seen for value-added analysis.

The correct premise is the following: “Value-added analyses give an objective estimate of teacher effectiveness”. An uncertainty of 30% is low enough to determine which teachers tend to be great, and which teachers tend to be below average. Principals could then use this data *along with other evidence of teacher effectiveness* to evaluate their teachers.

Every pro-VA argument I’ve seen has reasoned from this latter premise to suggest that VA data be used to supplement an evaluation system, and yet I’ve seen no compelling counterarguments to this reasoning. Is there any you can offer?

To go back to the post, I think Eduwonk’s point is pretty simple, from a legal standpoint. If the value added data were part of teacher evaluations, then when newspapers and others make FOIA requests, they would not be able to get the data. This is because personnel and evaluation records are exempted from public records requests.

But as long as the data is not part of the teacher evaluation process, then there is no reason newspapers can’t get the information from a FOIA request; there is no legal exemption.

The irony, then, is clear: Mulgrew doesn’t want value-added data to be used in teacher evaluations. Because of this position, the data is likely to be totally accessible by newspapers and made completely public.

The further irony: unions rightfully view test scores as only one piece of relevant information for evaluating teachers. But instead of seeking to have it be one piece, Mulgrew and others assail the use of it altogether. The end result: newspapers wind up printing it as the ONLY information about teacher performance. (yes, I know that the unions in a number of places have cooperated in making this part of teacher evaluations; but these FOIA requests will no doubt light a fire under the holdouts).

Anyway, as I have become more educated about teacher evaluations, I am in favor of having test score gains be a piece – but no more than 50% until they are shown to be more accurate than current. Here’s the tricky part though: as a parent and a taxpayer, I want newspapers to print the data. Right now, I have very limited visibility into whether my kids’ teachers are doing a good job. In a world in which 95% of my kids’ teachers are rated as outstanding under the current ridiculous evaluation system, I would love some actual data. And yes, if my kids’ teachers are in the bottom decile of performance, I am going to pay the principal a visit.

Until value-added scores are shown to be very accurate (and I believe they won’t be, due to the myriad of variables that affect the performance of any given class of students), I don’t think they should be used to evaluated teachers, and I especially don’t believe they should be published in newspapers for the consumption of the general public.

As a former teacher, I have seen how student performance can be affected by multiple issues, and these issues are not always spread evenly among the teaching staff. For instance, disruptive students will make it more difficult for the rest of the class to learn. However, principals often can choose which teacher will be assigned these students. If a principal assigns the five most disruptive 2nd graders to the class of the new third grade teachers, she’ll be at a disadvantage when it comes to teaching her class (and the test scores might show this). Similarly, if the principal gives veteran teachers their own classrooms, but makes the new teachers change classrooms each period (pretty common), those teachers will be at a disadvantage. These are just two of the MANY variables that affect each teacher’s ability to raise the scores of their students (putting aside the students themselves — their families, abilities, and motivation).

Given all the above, it’s not surprising that test scores for individual teachers show so much variance from year to year. Therefore, it is inaccurate to state that a teacher with a 15% increase in test scores in a given year is “better” than a teacher with a 10% increase, because you’re never going to be able to control for the vast multitude of variables which affect student performance.

Hopefully jurists in New York will be knowedgeable enough to understand that these value-added tests are not able to assess the effectiveness of a teacher, for the reasons given by AttorneyDC.

We all seem to agree that teachers, like everyone else, should be carefully evaluated but this needs to be done by examing the progress of each child through observation and individual testing; and by observation of the teacher by skilled professionals. People want to evaluate a teacher with one ten-dollar group test that isn’t even professionallly proctored, but obviously it can’t be done.

Linda, you are missing the point. Jurists don’t decide whether the tests are an effective tool or not. They simply have to decide whether or not a news agency has a legal right to access the information. And from a legal perspective, it seems like they do have that right.

The NY unions have a major pr problem, just as the LA union did. Because they have not come up with a reasonable metric for evaluating teachers that demonstrates what we all know – i.e. most are decent or pretty good, some are outstanding, and some are terrible – they have a very hard time telling the public that test score gains don’t matter. There is a massive public credibility gap because the unions have done everything in their power to keep poor performing teachers on the payroll in NYC.

This credibility gap is demonstrated by the fact that the NY Daily News is on the side of getting the data. Not the republican-oriented NY Post or the WSJ, but the Daily News.

Again, I think it’s reasonable to say test scores should be just one factor among several. But that’s not what the union in NY has been saying – it has been saying you can’t use test scores in evaluations. And, ironically, that means that the media will likely win its case and will get access to the data which they will print and which will become the only publicly available commentary on teacher performance. In effect, the union’s position on test scores has exposed its membership to a pretty unfortunate level of public scrutiny.

Chris, a better irony lesson would be to acknowledge that it is ironic that you are lecturing Guy on the definition of irony when your example suggests that you really don’t understand it after all.

How do you know the unions will “inevitablely lose” the value added battle? Your example is like me saying I knew the Yankees would beat the Twins in the first round of the playoffs and thus, it is ironic that the Twins even played the games and tried to win. Sorry, but that is not irony. You may be correct that the unions will have to make cocessions and allow value added models to become part of teacher evaluations programs, but that won’t necessarily make their actions “ironic”. They may in fact expect to lose that fight but still fight the battle in the hopes of at least mitigating the negative impact value added aproaches can have on their teachers. In short, an “objective and stringent” analysis of your reasoning shows you don’t understand irony and that that particular part of your comments in this thread added little to no value to the world.

I think it is within the province of jurists to decide whether this information is accurate or not. If they decide that it’s not accurate, they might conclude that it’s not public information. There is also a question of privacy. Of course, I’m not an attorney so I’m just guessing.

I’ve said this before but it’s worth saying again: Principals have long had the right to evaluate teachers in any manner that they wish. They can visit classrooms frequently, observe students, evaluate student progress, look at test scores and compositions and so forth. They can pull student files and look at standardized tests without informing the teacher. What they CAN’T do is state in a formal evaluation that Miss Smith is “ineffective” based on her test scores. However, and this is important to note, there is nothing to prevent her from basing her evaluation on test scores or anything else so long as she doesn’t announce her reasons publicly. She is free to state that “Miss Smith is ineffective due to poor student progress., as demonstrated by student work and principal observation.” In my state and many others, the content of a principal’s evaluation cannot be challenged. The unions can only challenge procedural problems or evaluations that are arbitrary or capricious. For example, my friend was given a poor evaluation the day after she spoke to the Board of Education. This was reversed by the courts and the principal was transferred. When he did this to another teacher, he was fired.

The unions are fighting evaluations based on test scores because it quite possible for an excellent teacher to have low “value-added” scores while a poor teacher could have high scores. Also, if the public hears that a teacher is “less effective” based on standardized test, many will view this as the sole determinant of her worth as a teacher.

So why have principals been giving almost everyone “highly effective” ratings? I’m guessing here as well, but I think it’s because in good economic times not many people want to be seventh grade teachers in urban schools. So principals just want to keep anyone who shows up each day and can control the class. Another reason is the fact that principals don’t have time to truly evaluate each teacher. It’s expensive and time-consuming.

I believe that this current push to evaluate teachers based on a ten-dollar test is related to the recession. It’s a quick and easy way to get rid of expensive veterans. Hopefully the jurists in New York will see the injustice involved and prevent it from happening.

Everyday millions of Americans go to work and report to a boss and are held to some standard of performance. Teachers are lossing all credibility with the public that some how they should be immued to what everone else is required to do.

“The irony, then, is clear: Mulgrew doesn’t want value-added data to be used in teacher evaluations. Because of this position, the data is likely to be totally accessible by newspapers and made completely public.”

This is a tortured use of “irony”. If I have only enough money to but food or pay the rent this month, and choose food, is my eviction “ironic?” If I’m a principal forced to slash my budget, and I choose to keep my art teacher, is laying off the music teacher “ironic?” These are choices between two bad outcomes, and choosing (hopefully) the least bad one — and that’s the UFT’s situation, at least as they perceive it.

Perhaps it seems ironic to you and Eduwonk because you think it’s right to use value-added in teacher assessments, but wrong to publish individual teacher ratings. So the union’s avoiding a good outcome — in your view — has resulted in a worse outcome. But the union likely doesn’t see it as a worse outcome — a bad one, yes, but not worse than enshrining value-added as an appropriate factor in teacher advancement and compensation. If you actually try to understand the union’s perspective, there is no irony in it’s actions.

Teachers are professional people, who like other professionals, are trusted to do the job without much supervision. For 42 years I opened my classroom in the morning and closed it in the late afternoons. Sometimes I’d go weeks before the school principal or another administrator entered the classroom. Once I thought to myself, “This is like being self-employed, only without the overhead.” Perhaps this is not “right” but it’s been this way for a long, long time. This situation is part of our culture and is not the fault of teachers or “the unions.” Personally I always enjoyed having another adult (teacher or principal) to work with, but this seldom happened. Placing two teachers in an inner-city classroom (as they do in many elite private schools) would be huge for the children, but taxpayers would probably not agree to this.

50% of all teachers leave during the first five years. Many of these people leave of their own volition and others are counseled out by the principal. Teaching is a “gentle profession” where “bosses” are very unlikely to say “You’re fired.” Instead the teacher is usually told “We don’t have a position for you next year.” She’s released at the end of her contract and listed as a “resignation.” Teaching (K-12) is actually the most self- selective of all the professions and that’s a fact.

Is it difficult to dismiss a teacher who has achieved permanent status? Yes, but all government workers are granted due process. It’s also difficult to fire a firefighter, police officer, city librarian, social worker, city nurse, attorney or physician. Do you want less for teachers? If so, why?

As you can tell from this blog, there is a huge push right now to discredit schools and teachers for the purpose of appropriating school tax money meant for poor children. Only teachers and their associations stand in the way.

Guy – I think that is a totally fair point on the definition of ironic. Agree that it seems ironic to me because I think data should be a part of the eval process, though not the only part and not public. Thanks for your argument – think it is totally fair and I will keep that in mind!

Linda – your ongoing assertions that the motivations of people who disagree with you are greed and desire to steal from poor kids have moved from ridiculous to outright offensive. Consider – for just a teeny tiny second – that people who disagree with you on policy may actually genuinely, deeply believe that their views will result in better outcomes for poor kids. Questioning the motives of people you don’t know: genuinely bad way to engage in the discussion, and you do it repeatedly.

No, there is a push to move money out of failed institutions that charge the taxpayer $25,000 per child while delivering no performance or hope and mandating by law poor children attend such institutions dooming them. There is a push to recognize that most teachers are hard working competent and underpaid and under supported. That little of the $25,000 per student ends up with the teacher.
That FedEx has three layers of management between the top and the guy who hands you your package. Yet in education, you have legions of bureaucrats between the teacher and superintendents, none of them helping the teacher as you attested yet sucking up volumes of funds.
If you want good schools you need good teachers, it is that simple. To get good teachers you need to pay them, motivate them and yes hold them accountable. Tenure, peer review and union protection only discredit the profession.

You will never be able to show that *ANY* evaluative method is accurate. You are confusing “accuracy” with “precision”, and the trouble with doing so amounts to more than just semantics.

Given that accuracy is not a useful way to evaluate a method of evaluation, the best way that we could do so is to determine if the method combs objective data to define teacher effectiveness. VA does, and thus we can calculate its uncertainty. Classroom observations and teacher “street cred” do not, and so we know neither the latter’s accuracy nor its precision.

*****

Jokefest:

“How do you know the unions will “inevitablely lose” the value added battle? ”

They already are, considering that newspapers are combing through and publishing VA data with personal information attached to it. As Eduwonk and Edconsumer (and I) have tried to explain, the alternative would be for the unions to accept VA data as a piece of the teacher effectiveness puzzle, and further argue that it cannot be made public.

“They may in fact expect to lose that fight but still fight the battle in the hopes of at least mitigating the negative impact value added aproaches can have on their teachers.”

The negative impact that VA analyses will have will fall on the lowest-performing teachers in any given district, assuming that everyone still remembers that a 30% uncertainty still allows for the data to be used as an objective estimate of effectiveness.

*****

Linda:

“The unions are fighting evaluations based on test scores because it quite possible for an excellent teacher to have low “value-added” scores while a poor teacher could have high scores.”

It is even more possible at this time for an excellent teacher to have a low “classroom observation” score while a poor teacher could have high scores. It depends on the observer’s subjective take on the classroom.

*****

Guy:

“This is a tortured use of “irony”. If I have only enough money to but food or pay the rent this month, and choose food, is my eviction “ironic?” If I’m a principal forced to slash my budget, and I choose to keep my art teacher, is laying off the music teacher “ironic?” These are choices between two bad outcomes, and choosing (hopefully) the least bad one — and that’s the UFT’s situation, at least as they perceive it.”

Bad analogy, since in this case the unions are choosing one bad outcome and subsequently helping cause the other bad outcome to occur. It’d be like if you choose to buy food instead of pay the rent, and then your landlord comes over and takes the food.

If the unions wanted to avoid teacher evaluation data being made public, they should have agreed to it being part of their teachers’ formal evaluations. Instead, they chose the “least bad” option in their perspective, and prevented VA data from being incorporated into formal teacher evaluations. Because of that strategy, not only will the VA data be made public, but their school communities will now informally evaluate their teachers with the only data available to them. Who wants to bet that the public will soon be asking why this data isn’t also being officially used? Such questioning has already started: the LA Times argued that LAUSD was sitting on this goldmine of data they wouldn’t utilize, and subsequently a discussion focused on whether or not VA data should be utilized broadened to whether it should be utilized *and made freely available to the public*, and it’s growing into a discussion had by more than just the education community.

Whether or not the union wins here doesn’t change the fact that its own strategy of choosing the “least bad” option in essence helped increase the opposition on both rhetorical fronts.

If you read my post again I was referring to “points made on this blog” but not necessarily by the posters. Specifically I am referring to the frequent celebrations of any or all losses for teachers, along with the juxtaposition of “edujobs” which always seem to indicate an education job that pays well but doesn’t involve any teaching. This is what is meant by “Poor Schools, Rich Targets,” not my words. Surely you must be aware that many people are suspicious of a “reform” movement that denigrates teachers. How can this possibly help children? For an excellent view of this position, see Diane Ravitch’s book “The Death and Life of the Great American School System. ” Yes, many of us see a great institution being attacked and that is offensive to us. We are fighting to preserve schools that were good to us and to our children. We want to see them improved, but not destroyed.

Chris:

I agree that classroom observations can be very subjective. In my post I described a comprehensive review of teachers that would include classroom observations, examination of assorted tests throughout the year, and a general awareness of how children are progressing. This is what is supposed to be happening now, but principals are generally too busy to do a thorough job. However, common sense should tell us that the job can’t be done with a ten-dollar group test. It’s difficult to believe that anyone would think so.

We DO need an improved method of evaluating teachers. I’d like to see peer evaluation and review or retired teachers and administrators hired for the express purpose of evaluating teachers. The real problem with teachers, though, is attracting and retaining very bright and well-educated people. How do we do that, especially with all the teacher-bashing going on?

TaxPayer: All professionals (doctors, nurses, dentists, accountants, college professors etc.) have peer review, strict due process, and strong associations that fiercely protect them. I think you’re putting teaching in the same category with jobs that are not considered professional. Many people do, and that’s probably one of the main reasons it is difficult for us to attract the “best and the brightest” to the K-12 teaching profession. Not that many people want to pursue five years of college, only to be treated like at-will casual employees.
.

**You can’t even measure the error rate in current principal and supervisor evaluations of teachers can you? And if you can’t measure the error rate it must be higher than any error rate you can measure!**

So, what Chris is saying is that the unobserved error rate in one system is necessarily greater than the observed error rate of another – even if we have no way to quantify it – in fact, because we have no way to quantify it!!

Unobserved error rate of ‘status quo’ > measured error rate of VAM

As Bruce Baker states (and yes, I stole some of his verbiage here)–such an argument is patently stupid.

Chris: I guess I just don’t understand all the energy spent on trying to evaluate teachers these days: Does anyone think this will help the schools? As Linda noted, other than in a recession, inner city schools are not going to have many teachers itching to teach in their classrooms. In my experience teaching in several schools, I really think that students will perform well when they are motivated, have supportive families and cultures (e.g., Asian students are advantaged by a culture that highly values academic performance), and when schools have the power to enforce appropriate discipline, preventing one or two misbehaving students from interrupting the rest of the class. Nowadays, most schools are hamstrung by federal and local laws (and political pressure) to avoid issuing any punishment, even for students who seriously disrupt classes on a continual basis. Letting teachers TEACH, and not have to play security guard in their classes, would do a lot to help overall student learning, in my opinion.

“In my post I described a comprehensive review of teachers that would include classroom observations, examination of assorted tests throughout the year, and a general awareness of how children are progressing.”

All of the above are examples of evaluative methods that suffer from subjectivity, so while they may be avenues of evaluation that ought to be included in defining teacher effectiveness, they are also very problematic in usage.

“However, common sense should tell us that the job can’t be done with a ten-dollar group test.”

If you’re suggesting that the given alternatives to your evaluative methods are only as good as a “ten-dollar group test”, you need to give evidence to support that suggestion.

“’d like to see peer evaluation and review or retired teachers and administrators hired for the express purpose of evaluating teachers.”

Still doesn’t solve the problem of subjectivity

“The real problem with teachers, though, is attracting and retaining very bright and well-educated people. How do we do that, especially with all the teacher-bashing going on?”

Begging the question is your favorite thing to do on this blog, isn’t it.

Well, neither does value-added since subjectivity is built into every statistical model. Analysts make hundreds, if not thousands, of decisions when building a data base and analyzing the data and these are subjective choices. Clearly you have not done this work, otherwise you would understand this.

“I guess I just don’t understand all the energy spent on trying to evaluate teachers these days: Does anyone think this will help the schools?”

Yes! That’s the whole point. There needs to be a better evaluation system in place that is more capable of resolving the effectiveness of different teachers, as there are many positives associated with such a concept. Why would you assume there is instead an ulterior motive?

*****

The Anti-Chris!!:

“So, what Chris is saying is that the unobserved error rate in one system is necessarily greater than the observed error rate of another – even if we have no way to quantify it – in fact, because we have no way to quantify it!!”

…

Look, if you want to carry on with a tone, at least know what you’re talking about.

Given the subjective nature of supervisions, “calculating an error” for observations doesn’t mean anything at all. You can’t effectively calculate error for qualitative data, unless you subjectively rank such data by a metric, although this doesn’t change the nature of the data itself. Saying we have no way to quantify the error for observations is missing the point: the error itself is undefined.

“Chris–”Still doesn’t solve the problem of subjectivity”

Well, neither does value-added since subjectivity is built into every statistical model.”

Ponder what you’re suggesting here. Value-added analyses, which utilize quantitative data through a mundane set of calculations, are as subjective as classroom observations? Really? You can’t be serious. Test scores and observational data are complete opposites on the “What kind of data am I?” scale. Why would you think otherwise?

“Citation about the agreement between the unoin [sic] and NYC DoE. SOunds like DoE is a bunch of liars. Pretty much par for the course for ed deformers.”

You’ve linked to a blog that opines on this issue, but doesn’t cite any sources for its claim. This is not a “citation”, but yet another red flag that readers should observe before believing you.

Linda – Your m.o. is to say things that are really outrageous – i.e. claims that the reformers are aiming to kill the system so that they can profit personally from it – and then back away from them.

I am fine with your vehement belief that people are doing things that harm the system. I may disagree with your views but you are certainly welcome to express them.

I am not fine with the repeated efforts directed not just at this blog but at all “reformers” – whatever that means – to claim that the people with different viewpoints actually want to personally profit from the system, and don’t care about kids. That is not OK.

In essence, I think you are doing exactly what right wing republicans do when they want to smear Obama. They say dog-whistle things like “I take him at his word when he claims he is American/Christian…” and pretend that they aren’t participating in an outrageous smear campaign. That is what you do when you claim that people are seeking education reforms so that they can turn a personal profit – and then you back away.

Who is trying to turn a personal profit? Stand up and say what you think. Are the people running KIPP doing it to turn a personal profit? Are the school superintendents who disagree with you purusing their policies so that they can turn a personal profit at the cost of kids’ lives?

This is what you imply repeatedly. Don’t back down when you get called out. Do you think these people care about kids, or do you think they simply are trying to make money?

That is what has to stop, Linda – the outrageous smears of the many people who have different policy views than you.

I was evaluated daily by students and parents. In addition to that I was formally evaluated every other year. Most of the time the evaluation was of a cursory nature, but that was not the fault of the teachers or “the unions.” The only time in my career I was carefully evaluated was when I applied for Mentor Teacher and had to be evaluated by my colleagues as well as administrators. Obviously teachers don’t want to be evaluated on the basis of a test that is not designed to measure teacher effectiveness.

Poll after poll shows that citizens support the teachers in their own communities. What we saw in DC will soon spread across the country as “reform” comes to local district.

Taxpayer, the objection is to being “graded” based upon one flawed metric. Don’t get me wrong, there are plenty of teachers who, in my opinion, misuse the power grading affords them. If a teacher were to tell parents and students that one timed multiple choice bubble test would determine whether students in his course pass or fail for the year there would be plenty of justified objections. In most classes grades reflect a semester’s worth of assessments: quizzes, in-class practice responses or labs, tests, homework, essays, projects, class participation, etc. should or could all be factors. I think your point, however, is a good one in that it demonstrates how educators need to be more precise about why we object to the value added approach becoming a leading factor in teacher evaluations.

You cannot calculate the error rate on qual, but you can on quant. Therefore, people like you state:

“Unobserved error rate of ‘status quo’ > measured error rate of VAM”

precisely because you cannot calculate the error rate on the status quo of qual teacher observations. I guess you cannot reflect on how you think and why you think things. Argghhhhhhhhhh. Man, whats the point of even arguing with you when you cannot comprehend a simple point???

Yes, I am serious about subjectivity. Have you been trained to observe teachers and calculated VAM scores? Wrll, I have and I can tell you both are subjective and can actually be equally subjective. Observers can be trained to reduce subjectivity.

Why do you think quant researchers describe methodology? Because the choices are SUBJECTIVE and the reader needs to know what subjective choices are made. But even more subjectivity is inherent in all the decisions about how data is collected, what is collected, how it is merged, what data are errors and removed from the analysis, etc.

You obviously have NO experience in either area, so perhaps you might want to keep quiet about things you don;t know very well and let the experts discuss.

And about two ends of the scale–the scale ends are closer than you think. Again, training of observers dramatically reduces subjectivity. Thats the point of the training you get in grad school in principal prep programs (which business leaders will NOT get which makes their observations be more subjective.

Edconsumer: I think a presumption of good faith on the part of opponents is a reasonable standard to aspire to. But please recognize that the new wave of reformers who routinely question the motives of those they disagree with. The assertion that schools are run for the benefit of adults rather than children is now routine. It is the central theme of Waiting for Superman. Michelle Rhee cannot speak or write for more than 250 words without claiming to speak for the best interests of children, and asserting or implying that anyone who disagrees with her is subverting those interests to a selfish agenda. Indeed, reformers now seem to assume automatically that any disagreement with their agenda must, almost by definition, reflect ulterior motives. Linda’s comments notwithstanding, this assertion of bad faith is actually much less common among those labeled “defenders of the status quo.”

So, if you are going to hold Eduwonk and his allies to this same standard — and call them out when they impugn the motives of teachers, union leaders, and ed school administrators — then I will applaud your efforts. If not, well then it’s hard to take your outrage very seriously.

Guggenheim loves to chant corporate slogans of all inner-city schools being “dropout factories,” as though teachers and principals and other employees work steadfastly—intentionally—to push students off the attendance list. Reform is needed, Guggenheim repeatedly insists, and thankfully, as he explained to MTV News on September 7 this year, the last decade has ushered in “a new generation of reformers who are doing an amazing job, in every city across the country. They’re starting to break the code on how you can educate kids, even in the toughest neighborhoods. So there’s a lot of hope if we focus on these reforms and smart reforms, and put away all the adult problems, we can actually start helping kids.”

No one here suggested that VA be the only metric. And what is flawed about a source of data that is precise enough for helping to differentiate between great and terrible teachers?

****

The Anti-Chris:

I don’t know why I’m bothering to respond to someone arguing in what looks like text message format, but here:

1) No one has ever argued the simple-minded inequality you posted above. Stop attributing it to me.

2) You are not paying attention: I’m arguing that the ability to actually calculate an error makes VA analysis more reproducible, more objective, and altogether more useful if it were included in teacher evaluations than the alternative. This is not about which one has the lower error; it’s about which one has error, and which one sits in a black box of subjectivity.

3) You may or may not have been trained to observe teachers and calculate VAM scores, but you’ve never learned the difference between the data you analyzed. That much is clearly apparent now.

4) Given what you’ve offered here so far as the anonymous upstart, I’m going to assume you’re not an expert on this topic. Even if you were, that doesn’t make your arguments infallible, and I will continue to reply to them.

5) “Why do you think quant researchers describe methodology? Because the choices are SUBJECTIVE and the reader needs to know what subjective choices are made.”

That the choice to choose a given experimental design is subjective does not imply the data collected within the design or the subsequent analyses themselves are subjective. Again, big difference between raw testing data and observational evidence. This is a key point in the usefulness of the two types of evaluative methods.

6) “Again, training of observers dramatically reduces subjectivity.”

Do you have a “citation” for this, too?

*****

Guy:

“But please recognize that the new wave of reformers who routinely question the motives of those they disagree with.”

That sounds like a trend. Can you name a few influential ones with instances of when they’ve done this?

“The assertion that schools are run for the benefit of adults rather than children is now routine. It is the central theme of Waiting for Superman.”

That assertion doesn’t question anyone’s motives, but it does question the system in place. Suggesting that some schools suck is not an attack on anyone’s motives, as it doesn’t imply that anyone intentionally wants those schools to suck.

“Michelle Rhee cannot speak or write for more than 250 words without claiming to speak for the best interests of children, and asserting or implying that anyone who disagrees with her is subverting those interests to a selfish agenda. ”

You should be specific here. Rhee has never argued that those that disagree with her are intending to destroy education and/or profit off of children. There’s a world of difference between implying that one’s method of change is the best method, and implying that all others operate on evil intentions.

“So, if you are going to hold Eduwonk and his allies to this same standard — and call them out when they impugn the motives of teachers, union leaders, and ed school administrators — then I will applaud your efforts.”

This is a rather passive-aggressive jab at Eduwonk. Can you give examples of where Eduwonk has similarly denigrated the motives of his rhetorical opponents?

Linda: I’m a lifelong democrat. Sorry to tell you. And no, you aren’t just making a point about public schools. You are maligning the motives of people you don’t know, doing it maliciously and publicly, and then pretending you are on higher moral ground.

Guy: I hear your point. I think everyone would do well to not impugn the motives of people who disagree with them. I don’t think the “well the other side does it even more” argument is an acceptable retort though.

Edconsumer: it’s not that “the other side does it more.” It’s that impugning the motives of the other side is the very heart of the argument made by reformers today. Reformers’ have adopted a message frame that posits a fundamental conflict of interests between children and educators (especially teachers), and claims that they are speaking for the best interests of children while those they disagree with are defending the parochial self-interest of adults. If someone argues against merit pay, they are “afraid of accountability.” If someone opposes value-added, they are “protecting the jobs of bad teachers who destroy children’s future.” At this point, I think this assumption is so automatic for some reformers that they don’t even consider the possibility an opponent is arguing from a good faith interest in children.

It’s a powerful rhetorical weapon. I understand why Rhee et. al. repeat it endlessly. But those who choose to use this weapon shouldn’t be surprised if those on the other side feel their motives are being impugned and respond accordingly. I find Chris’ naive response fascinating. I wonder if it’s common among reform supporters to not even recognize that they are doing this. Claiming the moral high ground is now such an ingrained habit, perhaps reformers don’t even realize their rhetoric routinely demeans others? If so, fascinating…..

Linda,
Why would you fight for the status quo after describing how pathetic the evaluation process was during your career? Do you support the Gates Foundation conclusion that merit pay with VA being 40% is optimal?
Concerning your contention that communities across the country will follow DC you are grossly mistaken. These communities are forced to be responsible and balance their budgets. They cannot spend $30,000 per student and be ranked at the bottom in performance. Everyone is being forced to do more with less and as public education is rightly viewed as massively inefficient the DC model will stay only in DC.

If you’ve read my posts, here and elsewhere, you will see that I do not support the status quo regarding evaluation. During my tenure I observed, and was part of, thorough evaluations, so I know what they look like. Here’s how it’s done:

Skilled professionals (teachers and/or administrators) observe the teacher’s lessons. They note the behavior of the students. Are they engaged? Are they learning? The evaluators look at student tests (benchmark, standardized, teacher-made) as well as student work and compositions throughout the year. The evaluator needs to have a good understanding of the progress of each child. It should be obvious to everyone that this can’t be done by looking at a single test. I support a committee of professionals as opposed to a single administrator because this will ensure more objectivity to the process.

At this time, most testing experts agree that value-added is not ready to be used as an instrument to evaluate teachers. There are many reasons for this, but in the simplist language this process is not able to accurately measure teacher effectiveness at this time. As I’ve written before, an excellent teacher could get a low score, while a poor teacher could get a high score (if she drills the kids on the exact items, for example). Remember that these tests are not professionally proctored.

A teacher CAN and should be evaluated, but it can’t be done on the basis of a test that is neither valid nor reliable for that purpose. So to answer your question, I don’t think VA should be used at all at this time.

As to merit pay, every teacher I know works to capacity and spends from $500 to $5000 a year on books and supplies for the classroom so if they want merit pay, they can award it to themselves by avoiding Borders and Teachers Supplies on the weekends. I feel certain that this action will not help children, but I don’t think it will hurt them either.

I DO support a career ladder (assistant teacher, associate teacher, teacher, Mentor teacher etc.) similar to what college professors have (with increases in pay with each promotion) but that’s probably well into the future. These promotions would be decided by committees of mentor teachers and would go a long way in helping teachers to become fully professional. This is something that WILL help children because full professional status for teachers will attract the best and the brightest to our nation’s classrooms. A career ladder will help retain teachers because they will have the opportunity to advance without leaving the classroom.

As you can see, not only am I opposed to the “status quo” but I am much more of an advocate for radical and significant change than most of the “reformers.”

Regarding DC, we saw that teachers are generally highly regarded by the people they serve. There are exceptions, yes, but in general these teachers are often leaders in their communities.

This is the way it is in almost every community in our country. Right now I’m thinking about my own. I am surrounded by excellent schools, even though I’m in a large city. In fact two of the top schools in the United States are within a two mile radius (Oxford in Cypress and Whitney in Cerritos, CA). Both my sons received world-class educations at local schools that prepared them for Harvard and Stanford. Almost EVERY person I know is satisfied with his OWN schools and teachers (supported by polls) and this is why we’ll see DC repeated. Just wait and see.

Don’t be an ass and throw in a passive-aggressive taunt my way. Be specific:

Why do you assume that “a fundamental conflict of interests between children and educators (especially teachers)” posits that “educators are motivated to only help themselves”? Surely the unions can rightfully respect that there are interests of their members that do not coincide with children, but at the same time can share a motivation for helping kids through supporting other student interests, as well?

Linda claims that “there is a huge push right now to discredit schools and teachers for the purpose of appropriating school tax money meant for poor children.” This not only assumes that the opposition’s policies are bad but that the opposition’s motivations include ruining schools. Find a similar claim by a reformer that suggests that teachers or unions *WANT TO* ruin schools with their ideas for policy. Rhee has never implied that her detractors want what’s worst for kids. Linda continually does so.

There was even a thread not too long ago where several(!) commenters reasoned that KIPP was run by white supremacists trying to indoctrinate poor minority kids to accept their inferior social status. And this is of course the same thing as arguing that someone is “afraid of accountability” if they don’t agree with merit pay. Please.

And if you snidely suggest that Eduwonk does this, show some examples. If Linda seldom bothers to back up her opinions with facts, at least you will, right?

Career ladders and peer review are the same as what we have, is not radical and will not change a thing. This will amount to pay based on tenure not performance. The reviews need to be by principals, students and parents. Colleges are a disaster and their systems avoided not mimicked.

Again, for a profession that spends its time grading other people who usually feel the process is capricious and not representative for teachers to say everyone else needs to be graded but not us because the test are unfair seems incredibly hypocritical. The Gates Foundation research shows VA at 40% is optimum. I suggest you spend your time on refining VA because it is coming.

Teachers are usually held in high esteem and I spend much time on how to pay the good ones more. The biggest obstacle is other teachers and the union.

But I too believe change is coming and “Waiting for Superman” is the new consensus not the failed bloated systems we are suffering with.

Without intending to do so, you have given us the primary reasons the “reformers” will fail; many are operating without knowledge of the system.

Teachers do NOT have a career ladder. What they have is an automatic salary schedule. A teacher advances automatically and is paid according to college credits and years of experience. I am talking about a career ladder where promotion would be determined by a committee of professionals and would be based on performance. It is basically the same as that used in higher education.

As for peer review, the only time I experienced it was when I applied for the job as mentor teacher. It was the only time in my career that I was thoroughly evaluated. The change that I propose is so radical that you can’t even understand what I am suggesting. Basically I want to see fully professional teachers who are in charge of the schools, just as doctors are in charge of their clinics and attorney are in charge of their law firm. Peer evaluation and review is used in some districts, but it is not widespread.

Our colleges are not “disasters” but are considered among the best in the world. Affluent people from every country send their sons and daughters to our colleges and universities. Our graduate schools, along with those in England, are considered to be without rival.

Many of us do not agree with your assessment of our schools as “failed bloated systems.” To me they are among the greatest of American institutions and the bedrock of our democracy. They are the reason for the spectacular achievement of the American people, most of whom attended public schools. They need to be improved so every child can profit from them.

Polls consistently show that the majority of Americans agree with my point of view, at least in regard to their community schools and colleges and that is why the “reform” movement, as it is presently conceived, will fail.

If value-added can be improved so that it really can tell us who is an effective teacher and who is not, then I would have no trouble supporting their use. No one wants an ineffective teacher in the classroom. Until that time, we must support evaluation methods that are tried and true.

Chris, I don’t know if you purposely misread what others have stated or you have a reading comprehension problem. You seem too bright for the latter.

Please reread my 12:03 post. I favor a thorough and meaningful evaluation of teachers. This would include an evaluation of the progress of each child through various means, including examination of test scores. Classroom observations would constitute one component. I am NOT in favor of a perfunctory method, whether it’s a single observation by the principal or a peek at a “value-added” test score.

As I’ve said before, you remind me of my son when he was 13 years old. If I said, “It’s day” he would argue, “No, it’s night.” That kind of “argument” can become very tedious for adults. That’s why I don’t like to respond to your posts but once in a while I feel that I should tell you why.

* The evaluators to look at student tests (benchmark, standardized, teacher-made) as well as student work and compositions throughout the year, although “look at” doesn’t mean anything as dastardly as “VA analysis”, so I don’t know what “look at” entails other than some form of “this teacher is improving because we think he’s improving!”

In short, to solve the problem of subjectivity inherent in classroom observations, you propose we form committees to subjectively determine if test scores and student work adequately define if a teacher is doing a good job or not. It’s such an amazingly astute solution, one wonders why no one else thought of it.

Your name and your words reinforce my strong belief that this “reform” is about money and the tremendous resentment many people feel regarding the education of “those children.”

Did your children get a good education, as mine did? I suspect that they did even though I don’t know you. Well, let’s share that good fortune with the children of the poor. Let’s make sure every child has access to a high-quality education at public expense.

I do agree with you about the “reform bus coming.” Now that teachers and other citizens are taking their rightful place at the forefront of the reform, I think we’ll start to see significant improvement for our children, such as preschool, highly qualified teachers for low-performing schools and increased professional status for teachers.

The old “reform” died in DC and will be hauled to the dump on November 2. Eduwonk is smart enough to see that coming (See October 22 post) so I’ll give him credit for that. Let’s see what happens.

Research on teacher evaluations performed by principals found that they were highly correlated with VAM. Assuming you could invest in better training of principals or observers. Observations are not nearly as subjective and inaccurate as some think. And VAM is much more subjective than people think as well. There is plenty of research out there that shows you get different results with the inclusion or exclusion of different variables.

One thing VAM proponents fail to admit is that we need to have better measures of student poverty, parental support, student effort, working conditions, etc to arrive at a VAM that could provide more accurate estimates of teacher value-added.

2) Observations are, however, qualitative and subjective. Do you disagree?

3) Raw testing data are, however, quantitative and objective. Do you disagree?

4) VAM would give different results depending on the chosen design, however this method would impact all data analyses equally. We should be willing to accept a bit of systematic error if it meant we actually had an idea of the precision of part of our evaluative methods.

5) One thing 90% of the commenters in this thread fail to admit is that they know not what “accuracy” means.

1) When I dont have better things to do–but its out there. Big name ppl in VAM research on both accounts.

2) Qual yes, subj yes. Degree of subjectivity? It depends. But you equate subjectivity with being wrong. Subjective evals can accurately identify good and bad teaching.

3) quant and objective, yes. But do they provide the researcher the ability to make valid and reliable inferences? Maybe, maybe not, It depends on the psychometric properties of the test. It would appear that the NY state tests do not meet the psychometric properties necessary. Further, you need to know whether teachers are teaching to the test. Excessive teaching to the test will make the scores quant and objective, but mean nothing.

4) It would not impact all teachers equally. And thats the point. Subjective decisions about the data, modeling, etc will differentially impact different teachers. Its subjective. Are you willing to risk your career on an imprecise measure? On a measure that only assesses one component of your overall job?

5) Well, one thing we all know–your reading comprehension is pretty poor and your logical reasoning skills are in the bottom 10%.

1) I estimate the odds are 10:1 that you won’t do so, or that you’ll link to a “citation” like you did last time: another unsourced blog post.

2) “Qual yes, subj yes.”

Good.

“But you equate subjectivity with being wrong.”

No, I equate subjectivity with a lack of an objective basis to determine which teachers are good and which are bad. It may be right and it may be wrong, but one thing is for certain: nothing is for certain.

3) “quant and objective, yes.”

Excellent. You agree there are important differences between the data sets. It’s due to these differences alone that we should see the need for utilizing BOTH in teacher evaluations.

4) “It would not impact all teachers equally.”

Does VA use different calculations for different teachers?

If you meant instead that the specific calculations used for all teachers will be biased toward a particular type of teacher, this systematic error could be accounted for with some oversight. Do all teachers with a given type of student average lower than other teachers with another type of student? I don’t imagine this should be all that difficult.

“Are you willing to risk your career on an imprecise measure? On a measure that only assesses one component of your overall job?”

I’d rather my evaluations be based on multiple lines of evidence, some of which are objective and thus have a known error rate, than solely on the whims of my employers.

5) You’re the one trotting out an argument related to the presumed accuracy of VA. You’re faulting it for not giving an accurate portrayal of teacher effectiveness, even though you won’t *EVER* know what the specific, accepted effectiveness of each individual teacher is, and thus you cannot calculate accuracy.

Yes, VA analyses (as well as all other options on the table) can likely be improved to give more precise accounts of effectiveness, but to say it can be made more accurate is pointless. For all we know its accuracy is spot-on.

To say it could be improved is also not a reason to abandon it, but rather a good reason to start investing NOW in further refinement. In the meantime, it can and ought to be used to supplement other available data.

Rhetoric:
Outgoing DC Mayor Fenty
“Much of the discussion revolved around education, and Fenty made the case that he was a martyr to his own education reforms — that he paid the price for being on “the leading edge of a movement.”

“If it’s a war,” he said, “someone’s got to be at the front of the line, and they’ve got to get killed first. That’s how you win a war, is by going forward.”"

"full of very lively short items and is always on top of the news...He gets extra points for skewering my
high school rating system"
-- Jay Mathews, The Washington Post

"a daily dose of information from the education policy world, blended with a shot of attitude and a dash of humor"
-- Education Week

"designed to cut through the fog and direct specialists and non-specialists alike to the center of the liveliest and most politically relevant debates on the future of our schools"
-- The New Dem Daily

"peppered with smart and witty comments on the education news of the day"
-- Education Gadfly

"I have just three 'go to' websites: The Texas Legislature, Texas Longhorn sports, and Eduwonk"
-- Sandy Kress, former education advisor to President Bush and former chairman, Dallas Board of
Education