Anti-Antiscience and Statistical Parlor Tricks

As a result of the Weblog Awards contest, I’ve been introduced to a number of blogs, such as Bad Astronomy and Pharyngula, of which I’d been previously unaware, although they are both popular and have a loyal following. Bad Astronomy has a set of links to what it calls Anti-Antiscience sites, which include their rival Pharyngula and similar sites. “Anti-antiscience” seems to include astrology, mentalism, things like that; Phil Plait on Bad Astronomy mentions James Randi a number of times. I followed one of the links to a site called Memoirs of a SkepChick, one of whose recent posts was a review of Criss Angel’s (pseudo ?-) altercation with a contestant on Phenomenon:

After his pathetic nightclub act, Callahan was praised by the Cosby kid, the host of the show, and Uri Gellar (of course). Criss Angel then whipped out an envelope and offered Callahan and Gellar a million dollars of his own money if they could tell him what was written inside the envelope. Instead of immediately going back into his fake trance, Callahan called Angel an ideological bigot and lunged at him in what I suspect was intended to be a threatening manner.

Which gave me the following thought: the focus of this site is also “anti-antiscience”. The SkepChick is against lame parlor tricks and pathetic nightclub acts; so am I. MBH98 statistical methodology is essentially a parlor trick – a point that becomes obvious to any statistician that has spent the time to investigate it (a point clear to Jean S and UC as well as myself). It’s not just the principal components methodology – although the principal components methodology was highly relevant to the show in the initial presentation of the parlor trick. When the principal components aspect of the trick was exposed, they re-tooled the trick (in Ammann and Wahl who use prior results from realclimate without attribution) so that the woman still appears to be sawed in half even without using principal components. And all the innocent climate scientists say WOW!

One of the purposes of statistical analysis is to expose parlor tricks, especially where there is self-deception on the part of the scientists. In our first articles, we criticized several aspects of the parlor trick in MBH98. We did not say that this was the only parlor trick in the world or that this was the only way that the magician’s assistant could be sawed in half. So when the magician’s assistants and serial coauthors, Ammann and Wahl, write that you can still saw the woman in half without using principal components as part of the trick, it doesn’t mean that our criticism of the original parlor trick was wrong – only that they’ve re-tooled the trick a little and you have to watch for the pea under the thimble. I’ve discussed their particular pea under the thimble before – and, in its own way, the overfitting in Ammann and Wahl should be even more embarrassing to the climate science trade than MBH98 ( see posts here , especially here herehere .

I’ve tried to tread a careful line in which I’ve been critical of these lame parlor tricks without presuming that the entire AGW theory depended on lame parlor tricks. It seems obvious to me that policy should not depend in any measure on parlor tricks – even if the policy ultimately proves to be the right policy. In addition, I think that it doesn’t even do people concerned about AGW any good to use things like MBH and its cousins. They should swear off such parlor tricks and focus on their best arguments. They might find that the effort sharpens their own presentations and makes them more convincing. (And, as readers of this blog know, we have thus far been unsuccessful in locating a straightforward engineering-quality exposition of how doubled CO2 leads to 2.5 deg C warming. I’m not saying that such an exposition is impossible; merely that reliance on things like MBH has perhaps diverted scientists from the expositions that they should have been working on.)

After a while, it becomes a bit discouraging that the MBH parlor tricks continue to re-surface under various guises, that climate scientists seem to believe in these tricks and that so many climate scientists apparently don’t understand how the tricks work. It really doesn’t give third party statisticians a very good impression of the acumen of the trade.

In passing, it’s interesting that Phil Plait of Bad Astronomy has also made an issue of data suppression. In a recent post entitled NASA suppressing aeronautic data: Part II, he said:

The bottom line is that Luedtke was 100% in the wrong about not releasing the data, and his reasoning for doing so is garbage. Griffins response was fine until he tried to spin this, which was also garbage.

You’d think that someone who thinks like this would support the campaign to make Lonnie Thompson and other climate scientists release their data, especially when, as in Lonnie Thompson’s case, the data is used in Al Gore’s hockey stick, some of it is over 20 years old and inconsistent grey versions have been circulated.

In reading the anti-antiscience comments, I was reminded of a famous comment by Keynes in 1940 in his review of early econometric models – and one that should always be kept in mind when considering complicated models:

But my mind goes back to the days when Mr Yule sprang a mine under the contraptions of optimistic statisticians by his discovery of spurious correlation. In plain terms, it is evidence that if what is really the same factor is appearing in several places under various disguises, a free choice of regression coefficients can lead to strange results. It becomes like those puzzles for children where you write down your age, multiply, add this and that, subtract something else and eventually end up the number of the Beast in Revelation.

Bad Astronomy is against people taking parlor tricks seriously. So are we.

Well stated, Steve. The thing I find so frustrating about the Mannian parlor tricks is that there haven’t been more eminent academic statisticians stepping forward and calling this stuff what it really is. We had Wegman, but where was the groundswell of support from his peers? I guess they just have their heads in the sand and are afraid speak up or figure the political/cultural ends justify the means.

I think it would be fascinating to de-contextualize the data, and then present the Mannian corpus to a variety of academics for comment. I’d pony up some big bucks to hear Brad Efron, George Box, or Norm Draper rip it to shreds.

Other blogs share the opinion when it comes to most topics, and then become mute (or cantankerous) on this one. There’s a strange politically-driven schizophrenia wrt skepticism, which applies it to only certain subjects.

In the SkepChick’s account of Phenomenon, she discusses in passing what statisticians might call an “out of sample” test.

Criss Angel then whipped out an envelope and offered Callahan and Gellar a million dollars of his own money if they could tell him what was written inside the envelope.

In the proxy field, this arguably corresponds to the “Divergence Problem”. Out-of-sample tests on a linear relationship between tree ring widths and increased temperatures have typically shown a divergence in the warm 1990s and 2000s. In most walks of life, this would be regarded as disproving any belief that these proxies could have identified warm past periods, just as a failure by Callahan or Gellar would disprove their supposed mentalism. Michael Mann and others have presented lame reasons why they should not have to guess what’s in Criss Angel’s envelope: updating the proxies is too “hard” and too “expensive” – something that Pete H and I disproved with our recent field trip to the Mt Almagre bristlecones.

In other cases, conflicting new data seems to disappear into a black hole. What’s happened to Lonnie Thompson’s Bona Churchill results. My long-standing prediction is that the 20th century dO18 values will be lower than 19th century dO18 values. This is sort of a Monte Hall situation in reverse: if the dO18 values had increased, they’d have found a way to get them in print in time for AR4. So what should a Monte Hall contestant, asked to predict Bona Churchill values, do? The only rational thing is to predict that 20th century dO18 values at Bona Churchill went down.

In the past I have been a fan and supporter of The Committee for Scientific Investigation of Claims of the Paranormal or CSICOP. Its now the Committee for Skeptical Inquiry or CSI. Recently I have been sorely disappointed by their acceptance of AGW without their previous intellectual rigor. Its really quite sad.

#9. Jean S, that’s a pretty funny video. You know, the Jim Callahan act is, in its way, a “detection and attribution study”, sort of like a Hegerl-Myles Allen study also purporting to detect something in a black box and saying – oooooooh, this is impossible. Maybe Myles Allen should go up on a stage, like Callahan, and writhe and contort with spooky music in the background.

Just as there is a “Magician’s Code”, in which the secrets of the trick is never divulged, there appears to be a “Climate Scientist’s Code”, in which the secrets of the data and processes are never divulged.

Everything will be fine in Oz, as long as we ignore the Mann behind the curtain…

#12, Steve, you leave me completely wondering what you snipped. It’s making me think that I should develop my blog software to have a group edit functionality, whereas for a period of time after a comment has been made, people can suggest draft revisions. Perhaps, people could also vote on these revisions. And there would some sort of mechanism to resolve conflicts, etc.

Steve: I didn;t snip anything in that post. The post as written contained several points demarked by “snip”, but I didn’t snip them. The “snip”
was inserted by the author himself. I’ve deleted the post entirely as it’s unreasonable for the poster to impute that I’d snipped things that I hadn’t. The post looked like it was a botched draft anyway.

I think that it doesnt even do people concerned about AGW any good to use things like MBH… They should… [instead] focus on their best arguments. They might find that the effort sharpens their own presentations and makes them more convincing.

I am not qualified to comment on the statistical aspects of this, I leave that for the experts in statistics. And so should those who are making the case for action on this problem. If it’s really a problem or not is immaterial to the crux of the matter.

I find it perplexing that those who are calling fo action on the policy level don’t seem to have realized that “nothing” (or “not enough”) gets done because instead of focusing on making convincing and compelling arguments to implement x, the time and energy is spent arguing about subjects they themselves are not experts in, acting as if they’re more concerned about arguing minutia rather than making something happen.

You don’t say “Why should I give you my work if you’re just going to tear it apart.” (Which seems o me to be an admission it’s bad work, but hey….) What you say is “You’re an expert at y can you check my work for me in that area please?”

I think that it doesnt even do people concerned about AGW any good to use things like MBH They should [instead] focus on their best arguments. They might find that the effort sharpens their own presentations and makes them more convincing.

“After a while, it becomes a bit discouraging that the MBH parlor tricks continue to re-surface under various guises, that climate scientists seem to believe in these tricks and that so many climate scientists apparently don’t understand how the tricks work. It really doesn’t give third party statisticians a very good impression of the acumen of the trade.”

Readily explained by the R-word you don’t like showing up in discussions here, Steve. It’s the piety part of AGW that prevents so many from seeing, or believing, what is right under their noses.

It’s interesting how the warmers jump on any divergence of temperature as a function of solar cycles to refute a strong solar/climate connection, but remain silent on their own proxy divergence problem. Yet another proof of the principle that irony, like entropy, always increases.

All of this about out-of-sample testing and the paranormal makes me wonder if we could set a Climate Audit challenge:

1. Ten rural sites with long stable well-established temperature records
2. From close to these sites, 30-50 high quality treering cores have been taken from living and dead trees.
3. The tree ring core groups have been coded and randomized
4. The individual measurements are sent to a lab where the treerings are measured to the usual high standard. With the analysis comes back a set of data from each site including ring widths and maximum density. The treering outer ring is dated.

The challenge:

Reproduce the temperature change of the temperature station to within 0.5C (most dendroclimatologists) or 0.2C (Michael Mann, who said so to the NAS Panel)

In regard to Mann vs. Ammann and Wahl, an excellent strategy in stage magic is to do the same basic effect twice, but slightly differently, using different methods. The audience assumes the same trick was used twice, and dismisses any explanation that doesn’t explain both versions.

Speaking of magic, many years ago on the Tonight Show, Orson Welles performed some shaggy-dog card trick that was so long and circuitous that in the end the reaction wasn’t “wow!” but more like “okay…” Sort of similar to the Ammann and Wahl paper.

Well yes and no. My comment was prompted by the mention of The Amazing Randi and Youd think that someone who thinks like this would support the campaign to make Lonnie Thompson and other climate scientists release their data, especially when, as in Lonnie Thompsons case, the data is used in Al Gores hockey stick, some of it is over 20 years old and inconsistent grey versions have been circulated. Along with comment #6 Skepticism seems to only apply to matters that are not already part of your belief system. This is apparent in many issues with political undertones.

I remember CSICOP when it was flying high with Carl Sagan and Randi. Sagan popularized the axiom Extraordinary Claims require Extraordinary Proof. Thats the first thing I thought of when I saw the Hockey Stick eliminate the Little Ice Age and the Medieval Warm Period. Randi was especially good at exposing hucksters and the self deluded . His work with Peter Popoff and Jacques Benvenisti was just as advertised Amazing. Randi later had a falling out with CSICOP over a slander suit, and things havent been the same over there. CSICOP was especially good at using talented outsiders such as Steve to help with their debunking so I never been too impressed with the attacks on Steve as a non-climatologist. I used to rely on them for good solid scientific skepticism. I wrote them to suggest that the Hockey Stick controversy would be a good match for their skills and pointed them here. I wrote in protest after their publication the report mentioned above, all to no avail. That skeptical attitude seems lost over there, but lives on here, so I keep coming back here.

Out-of-sample tests on a linear relationship between tree ring widths and increased temperatures have typically shown a divergence in the warm 1990s and 2000s.

Typically? Didn’t Rob Wilson say that 6 of 18 series showed the DP, most of which were N American?

Steve: In large population samples e.g. Briffa et al 1998, there’s a divergence problem. Obviously if enough care is taken with the subset chosen, the mentalist can be successful. Wilson’s 18 are pre-selected. That doesn’t meet the Criss Angel challenge.

Re 14 “snip” (as in cut) is a delimiter for a passage cut and pasted from a larger work and has no other significance. I cut a passage from Steve’s initial work on this thread and pasted it as the topic to be addressed by the rest of my post. If that is not obvious, please advise what indicator you want to use. It wasn’t a draft. It was a on topic item that could be of interest to many. Murray

I hope Steve McIntyre is invited to speak at TAM6. (TAM refers to The Amazing Meeting from the Amazing Randy and is dedicated to psudoscience bashing.)

I have also listened in horror to The Skeptics Guide to the Universe which is mostly very good objective science. Sadly, this is not the case when they talk about AGW. I was amazed at how the idea that just because something is studied over a fair amount of time that the truth must be knowable.

Steve: It’s not my position that truth is unknowable in respect to climate science topics. Just because the hockey stick studies are flawed doesn’t make everything unknowable.

#35. If Wilson (or someone else) is prepared to say that white spruce chronologies at treeline (or whatever) are temperature proxies, then you have to use the population. Or better, if you’ve identified that as a valid proxy, go out and sample some new white spruce sites and see if the hypothesis holds up that they are temperature proxies out of sample. You can’t ex post just use the ones that go up in the 20th century.

It’s the same with G Bulloides percentage. You can’t say that G Bulloides percentage in the upwelling Arabian Sea connects with world temperature but G Bulloides in upwelling Cariaco doesn’t.

You can’t choose after the fact. That’s one of the things that makes these sort of studies no better than a parlor trick.

The problem, Boris, is that divergence may mean that the “proxy” is NOT in fact a proxy. Until you understand what the source of the divergence is, you simply can not rule out that possibility.

It seems you are presuming that the proxy IS a proxy, and then suggesting that maybe its ability to reconstruct the past is limited. The divergence problem is deeper than that.

If the 18 are carefully selected, then 12 of them conforming to the expectation (correlation with temperature) is not surprising. One has to remember that this result is expected, based on a long history of prior experience and cherry-picking. That’s why the out-of-sample (modern-day) test is so necessary and so meaningful. The envelope’s contents are diverging from that which was predicted.

It’s interesting that so many people, scientists included, consider certain topics off-limits to Skepticism. It’s like how theatre requires a “willing suspension of disbelief.”

I wasn’t aware of the devolution of CSICOP, as I havn’t read SI is several years, but I was aware that the AGW proselytizer & Seed correspondent, Chris Mooney, does some research & writing for CSICOP, so it’s not surprising. Sad, though, to see a former standard-setting organization sunk into willingly suspended disbelief.

I’m a reader and occasional writer on both. The inquiring minds of both demographs have a lot in common, but they deal with vastly different deserving subjects most of the time. There’s no shortage of social fabric to be ripped by the skeptical inquirer.

The Skeptics tended at first to concentrate on debunking the paranormal, like showing how U Geller bent spoons. Some people criticised Sagan because of his desire to detect extraterrestial broadcasts, but that was a while back and vogues change as technology and it best uses change. Randi is a magician and a beaut thinker too. The Skeptics offer a big money prize to anyone who can telecommunicate, even do things like water divining, under controlled conditions. (The Australian Skeptics were left a large endowment by a dying admirer, which has largely gone into scholarships for youngsters and better publications). Of late, past couple of years, the emphasis has shifted to more serious things, like preventing hoax medical apparatus and operators from being licenced, like promoting vaccination in the face of opposition, like courses for teachers wishing to teach inquiry and the scientific method and so on. Quite recently, the rationality of extreme religious belief, formerly politely discussed only in passing, is starting to be spotlighted, maybe as a delayed response to the WTC attack.

CA deals with the more pure (as opposed to applied) scientific end of the spectrum of skepticism and is much more mathematical. People from either group would probably meld very well, but there is no good case to start mixing purposes and projects because that would dilute the effort of each. Some mixing of topics is already spontaneous, because climate science attracts skepticism from those used to dealing with numbers, interpretation, methodology and policy influence.

It’s quite instructive to participate in both. Many skeptics have a wicked sense of humour, so it’s not all heavy going. Why not go join up your local chapter od Skeptics?

to be honest, if it weren’t for Steve’s work here at CA, I would have completely tossed out the whole AGW hyptothesis out as falsified some time ago/ Per their models, the troposphere warming should be 1.3 times the surface warming. The surface temps are way out of whack. Also, it is possible that there is an AGW signature riding on top of other forcings. Denial that these forcings and natural variability exist, through the work of Mann et al just denies the obvious and creates skepticism of the whole theory. Since it is clear that the data is so shoddy, it leaves room that the theory might be true.

No. If you take Hansen’s surface temps at face value, and the troposphere temps at face value, the models are clearly wrong. They are simply falsified. The only possibilities that the models are correct is if someone made a huge error in the sat data, or the surface temp data is as bad as it looks on this site.

Mann’s historical data is simply out of whack with everyone else’s, as well as written history. For instance, they were eating strawberries in January in Germany during the MWP. Greenland and Antarctic ice cores concur and support the historical record, but not Mann’s hockey stick.

The only possible argument for AGW that has not been eliminated to my satisfaction is that it could be riding on top of natural variability, and thus a threat if it amplifies a natural warming trend. This is not satire, it is simple logic, think about it before you dismiss it.

#43 — Yorick, how would your logic change its conclusion if the case included that the physical uncertainty of climate models was more than (+/-)100 C per century (not to mention (+/-)2-5 C per annum)?

It seems obvious to me that policy should not depend in any measure on parlor tricks – even if the policy ultimately proves to be the right policy

Yes, this should be obvious and should be part of the swearing in of polical figures. Your work is contentious for the same reason it is of high quality – you are challenging deeply held feelings in the AGW community that are not a product of the science, rather of the politics. I’m not qualified to address the statistical issues in Mann though I am totally confused about how little interest there appears to be in the climate community to the points made in the Wegman report to congress.

They have a proven physical phenomenon regarding CO2 and long wave radiation that works in a glass tube.
They have a theory that generalizes this to the climate as a whole.
They have a computer model which they claim is based on physics and their theory.
The data collected to date fits so poorly as to falsify their model.

So, the ony way the model could be right is that the data that falsifies it(surface temps) is incorrect. Then they get a second bite at the apple. I would think that if one removed the UHI and other effects from the ground temp data, it is possible that one would get the better fit with the troposphere data they need to prove their theory.

So if I understand correctly, a model constructed from a proxy like tree rings that presumes to be skillful to calculate past temperatures based on a correlation to measured recent temperatures must, by definition, also be a model that predicts how tree rings will behave in the future as a function of temperature. That’s why out of sample verification is important. It can falsify the conclusion that the proxy model correctly correlates temperature with tree ring behavior. The best test would be to sample the same trees again some years later, as was done with the bristlecones. The fact that this apparently hasn’t been done on a routine basis makes the validity of the models suspect.

Air temperature, whether surface or at higher altitudes, is a poor metric for determining the sign and magnitude of global radiation imbalance. Ocean heat content is far better, but harder to measure. So the failure of computer models to predict the correct relationship between near surface, lower troposphere and middle troposphere temperatures is unsurprising. It also says little or nothing about greenhouse theory in general. It does, I think, imply an upper bound on the climate sensitivity that is much lower than 4.5 to 6 C, though.

The only way a lack of conformity does not defeat the proxy is if the model can be revised slightly to fit the new observations. This is post hoc revisionism, which means you start again with your hypothesis. i.e. Strictly speaking, the model is “refuted”, but you do earn “another bite at the apple”.

One way the linear proxy model could be revised would be to suppose tree responses are nonlinear (i.e. upside-down quadratic response). The linear part of the hypothesis would then apply when temperatures were lower than today (ring width proportional to temperature), but for temperatures warmer you would be operating in the nonlinear portion of the response curve (rng width not related to, and then inversely related to temperature). This might explain, for example, the occurrence of present-day “positive and negative responders“, and you would furthermore predict these would gradually all become negative responders in the future. In terms of paleoclimatology, clearly, a nonlinear model has important implications for trying to reconstruct a MWP event or a “millenial megadrought”. Above a certain temperature the event is unrecoverable. Doh.

Ultimately, the model is nothing more than hypothesis until the physiological basis for the alleged response is understood. So, why do some trees “diverge“? Why are there “positive and negative responders”. Dunno. It’s an unsolved mystery.

Ultimately, the model is nothing more than hypothesis until the physiological basis for the alleged response is understood.

Re:#48

The best test would be to sample the same trees again some years later, as was done with the bristlecones. The fact that this apparently hasnt been done on a routine basis makes the validity of the models suspect.

The most disconcerting question then becomes: Why do we not see efforts in the climate science field to seriously pursue a better understanding of these obvious holes in the models.

#52 I can’t remember the paper, perhaps I can look it up at home later. But, in biology and ecology a living organism or system will grow up to the limiting factor and beyond. This sets up the typical sinsoidal curve of populations.
It is strange to see all these negative feedbacks kick in everywhere except AGW. Imagine our sun without a negative feedback, it would get hotter and hotter until it consumed the earth or went nova. Thankfully even a suspended on-going nuclear bomb has some negative feedback.

It would be good practice to replace deleted posts with a placeholder, such as “post deleted by moderator”, to keep the post number-scheme intact. It’s annoying to refer back to a post & find its number has changed.

Thanks & cheers — Pete Tillman

Steve: It takes much longer to snip than to delete or move. I occasionally try to economize on my time. Readers can ask people to stay on topic as well. bender’s helpful that way.

It is also very helpful to reference prior posts by linking to the URL for the comment, which is contained in the comment number at the right side of the comment header. Thus if the comment number gets changed, you are still linking directly to the comment.

But Earle, that linking doesn’t work if it gets moved to another topic, I don’t think…

What I like to do is (depending on the subjet) is either quote something, blockquote something bigger, or at least refer to both the person that said it, by name, and a little mention on what the topic is. For example, I think it’s pretty clear that I’m speaking on what you said about linking posts that may get deleted or moved or snipped. (Or at least clear to anyone that’s been reading this blog for a while!)

I’m not a scientist, but I am a magician. I’ve attempted to follow this whole climate change thing for years and I’ve always felt there was something wrong with the science. Much like someone watching a skilled magician work, you know that something unseen is happening, but you don’t know what. If someone should shout out “I see the coin in his other hand”, I have two choices — open that hand and look a bit foolish, or create some misdirection and get rid of the coin in some way.

I would suggest that you have seen the coin in the other hand and I highly doubt that a scientist will willingly open up and admit to ‘trick’. That means there is some other misdirection going on to divert attention. What is that, I wonder?

YES I second almost all the comments ..good to see real skeptics
– I have been sad to see that when it comes to Global warming many top US Skeptics throw all their principles away and suddenly come over all “True Believer” just like the UFO hunter’s, anti-anti-vaccine people with “confirmation bias”, “absolute certainty beyond evidence”, emotion, SHOUTING & namecalling etc.
with there own version of Dowsing which works, works, but cannot predict the climate any better than chance.
– I would love to see Steve McIntyre speak at TAM
– Such a strange story : Mad Mann at TAM 2013
– I hope that TAM will be the place that Mann will be questioned properly
– it is essential that Mann’s appearance is not completely controlled & scripted and that questions can be asked freely without having to be submitted for clearance before hand. That people should be able to ask him about his past predictions and should be able to ask to make firm predictions for the near future that can be tested by time.