July 21, 2006

The Era of Passion

This will be a continuation from my previous post.Daran’s response in particular deserves to be looked at here. In it, he argued that I was simply setting the standards so high that no amount of information could realistically meet them; that what I was doing was tantamount to what tobacco companies have done every time they argue that there is no “real proof” that smoking increases the risk of cancer.

I obviously have not presented my argument very accessibly. I will attempt to remedy that immediately.

I’m not arguing that we have to reach Godlike levels of certainty–what I’m saying is that we need to start with methodological questions before we can really make a case for anything.

Starting, of course, with what information is available. If there isn’t very much of it, then there’s not much you can do, of course–but if there isn’t all that much, then drawing strong conclusions from it and throwing out accusations with any certainty is simply irresponsible.

Now, if you do have some information, you have to ask yourself what to look at in particular–what one thing you can follow, and how trustworthy an indicator it is for the subject you’re interested in.

Once you’re reasonably certain of what the margins of error associated with what you’re observing are, that is when you can ask “what could this mean?” and explain why you believe it best to be interpreted in a particular way.

No, I’m mourning because eight-year-old Iraqi girls are getting kidnapped and raped, and it’s happening more often now than it used to.

Both Iraq pre-invasion and Iraq post-invasion are sick societies in which huge portions of the population have their substantive freedoms curtailed. However, there’s strong evidence that for girls and women in particular (but not exclusively), things have gotten much worse since we invaded – which is an incredible achievement, considering how horrible things were pre-invasion.

In this debate, Ampersand makes a powerful emotional appeal–young children being raped, women in general finding their situation more and more helpless. And to defend this appeal, he speaks of “strong evidence”.

The debate gets increasingly pointless:

If it was your own face with acid splashed in it, I doubt you’d say that what matters is that it happened to you in a liberal democracy.

If I am free to vote but not to walk the streets, I am not substantively free. If I am free to vote but the political system lacks the will to protect me from being kidnapped, raped and sold into sexual slavery, I am not substantively free. If my odds of being violently killed are high enough so I live in perpetual fear, I am not free.

again you show your utter refusal to hold your policy up to any standards at all

given the truly astonishing record of incompetence and stupidity your leaders have displayed in Iraq so far, why on Earth should I take your empty belief that improvement is just around the corner seriously?

Those theories may be empty to you, but if so, it’s because you’ve had that governance and those rights so long that they seem like air and water – natural entitlements that are part of the universe, not something precious, painfully scraped out of nothing by the blood and sacrifice of past generations.

You and yours don’t have the stomach to see it done; noted, and not a big surprise.

There are obviously several dimensions to this debate; it’s quite good on a number of them.

Yet while Ampersand bemoans Roberts inability to “hold up to any policy standards”, he continues to lack any kind of standard to judge his own information.

He argues that “it appears that Saddam’s regime wasn’t as awful as the post-invasion regime.” And what is this based upon? What sort of comparison are we even capable of making?

I suppose you could argue that reported cases of abuse are up–though I don’t know that for a fact. But if that’s so, then what exactly does that mean–how many cases were ever made any kind of public knowledge under Hussein’s dictatorship?

It’s akin to arguments of “epidemics” in different diseases, because their reported occurances increase. Yet any serious statistical observation reveals that the more doctors learned about the disease, the more often they diagnosed it correctly–meaning that while the reported occurances might increase, that may be more a matter of improving measurement than anything else.

Moreover, I think it would be difficult to make the case that in a system which utilized various elements of kidnapping and torture in the night, including rooms where women were raped in order to extract information from their kin, and left many mass-graves in its wake–things were safer, and people more free, than they are today.

Now, in claiming that such atrocities exist, I’d like to point out that these things are well documented. Mass graves have been uncovered, and are fairly common knowledge.

But that does not mean that Ampersand’s conclusion is wrong. It could very well be that women are being abused on a greater scale than they were in Hussein’s dictatorship.

If that is true, it is too important a matter to present so very haphazardly.

Perhaps I may seem obsessive on matters concerning method, but I find it very difficult to ignore all the people in my day-to-day life as well as in the online and written world who express outrage over the people who disagree with them, as if those who disagree are impeding progress. Yet the only ones impeding anything are the people who harp on issues of great importance without taking them seriously enough to get information beyond the anecdotal.

One should be moderated by one’s information. If all you have are anecdotal accounts, with no sense of actual proportions, then claiming there is a “Hidden War on Women” and then proceeding to harp on other people’s failures to “hold up” to “any standard at all” is simply self-indulgent.

What’s more, when the actual methodology of judging information becomes secondary, things deteriorate rather rapidly. Robert wouldn’t believe what he believed if he lived there, Ampersand and those who agree with him haven’t the stomach to see things through….and so on.

The first remark is pointless because it assumes that a certain factual conclusion has been reached–that things are definately worse, and so Robert’s opinion is heartless. It would be far more productive to discuss whether or not things actually are worse, than to condemn someone’s beliefs over a fact that has not been established.

Likewise, rather than talking about who has the stomach to do what, debating what the cost of Iraq has been, a subject of some controversy by itself, would probably be a better starting place. After addressing the question of what the cost has been, it would then make a great deal of sense to ask whether Ampersand or anyone else for that matter feels that it is worth it.

(EDIT: I originally oversimplified this, but for the purpose of this post, I’d just like to say there there’s somecontroversy over a Lancet Article dealing with casualties in Iraq–and that’s one example of why reasonable disagreement can occur on the subject)

Passion is an essential feature in a human life. But in debate, if one indulges in a righteous passion when they have not even bothered to seriously examine the information from a methodological standpoint, you are far more likely to devolve into ad-hominem attacks and talking past one another.
To return to Daran’s charge: the Tobacco company argues that you can’t prove smoking is risky, because you can’t prove anything 100%, so if that’s your standard, then everything will fall short.

On the flipside, if you’re using a handful of quotes, the sea of anecdotes that exist out there make it easy for anyone to support what they believe.

The only way to guard against either outcome is to make a serious examination of your own standards; to provide an open explanation of what your methodology is when you engage in debate. That allows the person you’re debating with to either try and meet your standard with counter-information, or to criticize your standard as being to either of those two extremes I mentioned in the previous paragraphs.

But without that, no amount of information has any meaning. Because unless you have a standard by which to judge information, it can mean just about whatever you want it to.

The problem is that Adam’s linking to a non-professional article by a writer who doesn’t understand the methodological issues – and that’s the sum total of his evidence that the Lancet methodology is horribly bad. (Edited to add: So far.)

In any case, to describe the Lancet’s methodology as the worse possible methodology is either hyperbole or ignorance.

Adam, have you actually read the Lancet article?

In any case, I do intend a longer response to the points you’ve brought up, but it’s not a post I can hack out quickly. I’m hoping to write it while I’m in NY next week, since then I can work on it without taking time away from my drawing board.

Charles, if you wish to discuss the methodology of the Lancet study, I would be happy to.

I didn’t want to get sidetracked in this post; I merely pointed out why there could be reasonable disagreement as to the number of casualties there have been. I linked to an article which discussed why the methodology was flawed–the Lancet Study simply concluded that they could be 95% certain that between 8,000 and 194,000 had died, and the 98,000 figure (often rounded up in headlines to 100,000) simply came from averaging the two possible extremes.

In the words of the Slate Article, “This isn’t an estimate. It’s a dart board.”

But if I’ve missed something about how the Lancet study was conducted, or if there’s some other study you’d like to bring to the table, I’d be happy to go into that with you.

The problem is that Adam’s linking to a non-professional article by a writer who doesn’t understand the methodological issues – and that’s the sum total of his evidence that the Lancet methodology is horribly bad.

This is my fault–I shouldn’t have brought up the Lancet article, it only fogged up the point I was trying to make.

In any case, to describe the Lancet’s methodology as the worse possible methodology is either hyperbole or ignorance.

Probably a bit of both, I’m afraid–but from what I do understand, they stated that there was a 95% confidence interval that no fewer than 8,000 died, and no more than 194,000. But with a range so wide, the findings are essentially meaningless–unless I am missing something, which is not exactly unlikely 😀

And Ampersand, I will be looking forward to that post. Hope you have fun in New York beyond just finding holes to poke in my arguments (probably won’t take all that much time or energy anyway).

I have edited the section of this post dealing with the Lancet Article so that it better suits the point that I was trying to make. I still hold the opinion that I do, but recognize that it had no place in the context of what I was talking about.

And of course, I’m still up for taking the subject by itself and discussing it 🙂

Omigod my eyes (and brain) glazed over trying to read this post. (I didn’t get all the way through it. Your ideas just don’t track together.) Again with the methodology and call for objective, verifyable, even reproduceable evidence? I don’t believe that a rigorous statistical methodology can be expected in a war zone, if indeed that’s what you’re seeking. And besides, as with counting crowds of assembled people, once the numbers mount beyond a certain level, they become indistinguishable. Twenty thousand attendees is greater by a factor of two than ten thousand, but both numbers are already high enough to lose some meaning.

As to the title of your post, yes, we live in an era when passions rule reason rather than the other way around. It’s convenient when they’re in agreement, but our current leadership trusts the former more than the latter. The general public is not much different. However, I’m not sure we can simply wish a sober, academic rationalism into existence when the habits of mind that inform that way of thinking are being eroded around us. Take your typical college faculty debate and see who actually appeals to logic rather than emotion or political posturing.

Maybe we can tighten things up considerably, but I caution: be careful what you wish for.

I don’t believe that a rigorous statistical methodology can be expected in a war zone, if indeed that’s what you’re seeking.

I disagree. It’s difficult to get coherent information out of a war zone, but we do get information. The rigorous statistical methodology comes in when we are attempting to interpret that information, and as I said, I’m no talking about achieving absolute knowledge here, Brutus.

To be honest, I don’t quite understand your reaction to this sort of thing. You make several claims here:

we live in an era when passions rule reason rather than the other way around. It’s convenient when they’re in agreement, but our current leadership trusts the former more than the latter. The general public is not much different

Yet since you bemoan methodology, you are clearly free to claim just about anything, since there is no way to present counterevidence.

What I seem to have a difficult time getting across here is that discussing methodology doesn’t have to come down to massive, objective, scientific studies–it can be something small, like: I am going to look at GDP in order to evaluate something. Now, the problem with this is that GDP tends to leave out X and understate Y, but for the point I am making, I think it works reasonably well enough.

What you appear to be arguing, Brutus, is “that’s too hard. I’d rather just talk about what I believe to be the case. Talking about why I believe that is tedious.”

Sorry to harp on the Lancet thing, but it’s important; if I can’t convince you that you’re wrong on this, then I don’t think there’s any point in continuing the discussion. Plus, I can argue about Lancet without doing any new research. 🙂

Probably a bit of both, I’m afraid–but from what I do understand, they stated that there was a 95% confidence interval that no fewer than 8,000 died, and no more than 194,000. But with a range so wide, the findings are essentially meaningless–unless I am missing something, which is not exactly unlikely.

As Daniel Davies pointed out, as wide as the confidence interval is, the number “zero” does not fall within it. We can be extremely confident that (measured by deaths) things are worse post-invasion than they were pre-invasion, and the only question is how much worse. I don’t see how you could call that an “essentially meaningless” finding.

Secondly, your dartboard analogy (which you quoted from Slate, but you seem to endorse it) implies that because the confidence interval includes numbers between 8,000 and 194,000, therefore any number in that range is equally likely to be true (like throwing a dart at the numbers). But that’s wrong. Just because all those numbers are included in the range doesn’t mean that they’re equally likely to be true; with the kind of statistical analysis the Lancet authors used, the numbers at the center of the range are much more likely to be true than the numbers at the extreme edges.

It should also be noted that the Lancet survey is much more likely to be an underestimate than an overestimate, both because the cluster sampling method is more likely to underestimate than it is to overestimate deaths, and because the researchers excluded the cluster with the largest number of violent deaths from their results.

Second-to-lastly, you seem to be saying that a wide confidence interval equals bad methodology. That’s such a bewildering and odd claim that I don’t know what to say to it. This type of cluster sampling in war zones, which always leads to a wide confidence interval, is a well-established methodology that has been used again and again, in numerous studies, and published in well-respected peer-reviewed journals. What is your basis for claiming that this is bad methodology?

Finally, although you’re right to say that there is a “controversy” over the Lancet article, it’s similar to the “controversy” over evolution vs. creationism, or whether or not global warming exists. Yes, reasonable people disagree about these matters; but that doesn’t mean that the opinions on each side are equally reasonable and well-informed.

Second-to-lastly, you seem to be saying that a wide confidence interval equals bad methodology. That’s such a bewildering and odd claim that I don’t know what to say to it. This type of cluster sampling in war zones, which always leads to a wide confidence interval, is a well-established methodology that has been used again and again, in numerous studies, and published in well-respected peer-reviewed journals. What is your basis for claiming that this is bad methodology?

You have convinced me to take another look at the matter.

The nice thing about going to a university is that I can find a statistician to ask about confidence intervals 🙂