Lambert, Lott and Mann

Readers of this blog may have noticed some chaffing back and forth between me and Tim Lambert. Anyone that’s followed the chaffing may have noticed that Lambert has spent a lot of time criticizing studies by John Lott on guns on statistical grounds. On a personal basis, I dislike/hate guns and cannot imagine why anyone would want one in their house. I have pretty typical urban Canadian views. Intuitively, I’d be inclined to think that anyone purporting to prove that more guns leads to less crime is likely engaging in pretty suspect statistics and would be inclined to think that Lambert’s criticisms of Lott are probably meritorious. But it’s not a topic that so far has interested me enough even to read Lambert’s criticisms of Lott.

By chance, while I was googling a complete different statistical topic, I stumbled across another statistical take on Lott, also severely critical. What intrigued me is that these criticisms of Lott’s methodology parallel my criticisms of Mann’s methodology. Maybe I’ll have to look at the Lott criticisms some more.

Lott’s work is an example of statistical one-upmanship. He has more data and a more complex analysis than anyone else studying the topic. He demands that anyone who wants to challenge his arguments become immersed in a very complex statistical debate, based on computations so difficult that they cannot be done with ordinary desktop computers. He challenges anyone who disagrees with him to download his data set and redo his calculations, but most social scientists do not think it worth their while to replicate studies using methods that have repeatedly failed. Most gun control researchers simply brushed off Lott and Mustard’s claims and went on with their work. Two highly respected criminal justice researchers, Frank Zimring and Gordon Hawkins (1997) wrote an article explaining that:" just as Messrs. Lott and Mustard can, with one model of the determinants of homicide, produce statistical residuals suggesting that ‘shall issue’ laws reduce homicide, we expect that a determined econometrician can produce a treatment of the same historical periods with different models and opposite effects. Econometric modeling is a double-edged sword in its capacity to facilitate statistical findings to warm the hearts of true believers of any stripe."

Zimring and Hawkins were right. Within a year, two determined econometricians, Dan Black and Daniel Nagin (1998) published a study showing that if they changed the statistical model a little bit, or applied it to different segments of the data, Lott and Mustard’s findings disappeared. Black and Nagin found that when Florida was removed from the sample there was "no detectable impact of the right-to-carry laws on the rate of murder and rape." They concluded that "inference based on the Lott and Mustard model is inappropriate, and their results cannot be used responsibly to formulate public policy."

The analogy would be even more complete if Lott, in addition to a complicated methodology, did not disclose his data (until his feet were gradually and increasingly put into the fire), misrepresented his methodology in important particulars, failed to disclose adverse cross-validation statistics, etc. etc. When I see a comment about the impact of slight changes in Lott’s statistical model, the line of analysis seems pretty similar to our analysis of the presence/absence of bristlecones.

It may be that Lambert on Lott and M&M on Mann have more in common than we may have appreciated. Also, it’s pretty easy to understand that if you had some other studies by Lott’s co-authors finding similar conclusions, anti-Lottians would not necessarily be very impressed and would be quick to examine each one of those other studies to see exactly where the flaw was. I’ll try to spend some time working through the analogy.

I don’t want to get involved in nuances of Lott i.e. for the purposes of arguing the impact of bristlecones on MBH98, I don’t RELY on the impact of Florida on Lott or any other aspect of the criticism of Lott – obviously our critique of Mann is completely independent. In the Mann context, it’s obviously inappropriate that his results depend on the presence/absence of bristlecones and equally inappropriate that these results should not have been reported and equally inappropriate that he should have said that his results were robust to the presence/absence of dendroclimatic indicators, when he knew that they were not robust to the presence/absence of bristlecones. Imagine comparable circumstances in Lott (or Lott withholding cross-validation R2 statistics or the equivalent.)

Actaully Steve, I think you should think twice about slagging Lott in such simplistic terms. That…guy from Rutgers is basically taking a crap all over the notion you have been pushing: openness and sharing data. My understanding is that in at least one area Lott has been pretty good–sharing his data and methods. Isn’t that what you want from Mann, et. al.? One could also argue that Lott’s issues with Lambert and others stem in large part from this openness. Lambert though tends to take it to a level that is…well bordering on the disturbed. You can see it with his stuff on your co-author McKitrick.

I also caution you to rethink the possible thought that this will get Lambert to let up on you. It wont.

My final advice is simple: Stay on target. Getting dragged off into guns and crime is going to divert your efforts from where they should be. Save the stuff like guns and crime for later.

Steve V., you’re undoubtedly right. I’ve tried to keep comments narrowly-based. I’ve been surprised at the market for the narrow focus of the blog – it was over 125,000 hits in July – so I’m obviously hitting some kind of audience without straying off point. The trouble with feeding blogs is that it’s easy to post up stuff off the top of your head and I guess I’ve done that here. The thought that I had in mind was that anyone slagging Lott for his methodology should be able to see worse methodology in Mann. To the extent that Lott was open, then Mann was that much worse than Lott. I certainly didn’t mean to vary any of my positions on openness or get into a debate on Lott’s methodology. So people: I’ve got a couple of choices here. If it looks like we’re going to debate guns, I’ll either have to freeze the comments on this post or delete the post entirely as an off-topic. I don’t want to support an after-market arguing back and forth on guns, so I may do the latter and post up a notice on this on another post.

Perhaps a better example than “Lambert, Lott, and Mann” would be Legates, Labat, and Mann. In this case, Legates et al. show that Labat et al.’s claim that global runoff increases with global temperature is a statistical artifact based upon a single data point, which, if removed, leaves no statistical relationship.

The title of paper and its abstract:

From Advances in Water Resources, Article in Press. “Comments on “Evidence for global runoff increase related to climate warming”” by Labat et al.

We have examined the evidence presented by Labat et al. and found that (1) their claims for a 4% increase in global runoff arising from a 1 C increase in air temperature and (2) that their article provides the “first experimental data-based evidence demonstrating the link between the global warming and the intensification of the global hydrological cycle” are not supported by the data presented. Our conclusions are based on the facts that (1) their discharge records exhibit non-climatic influences and trends, (2) their work cannot refute previous studies finding no relation between air temperature and runoff, (3) their conclusions cannot explain relations before 1925, and (4) the statistical significance of their results hinges on a single data point that exerts undue influence on the slope of the regression line. We argue that Labat et al. have not provided sufficient evidence to support their claim for having detected increases in global runoff resulting from climate warming.

Well don’t get me wrong Steve M. I think Lott has done some rather questionable things at best. On the other hand you have much bigger fish on the hook that Lott. I say cut bait on Lott and stick with the big one you’ve got. Freezing the thread is the best, IMO. Deleting is not usually a good practice since it makes people what you might be trying to hide. So keep it and close the comments.

And beware Lambert; he is like a tarbaby in that once you get sucked in it is hard to extricate yourself. He will write dozens of posts often times on the most obscure things. He’ll pick apart every little mistake (Ah-ha! A dangling particple–I’ve got that McIntyre NOW!) and then extrapolate from there so that everything you do is tainted. Just ignore him, or at best don’t respond directly to him.

I don’t know. I think there are plenty of supporters of Steve who can keep HLH (Hunter Lambert Hearndon) occupied if need be. It’s not like any of the three were making substantive points which would require advanced math to counter. Would that they were!

BTW on the putative subject of this thread, I can’t speak to Lott’s statistics, but at least he didn’t make things up out of whole cloth like Bell-whatshisname did.

While I have had some inclination to mention some analogies between the Bell-whatshisname case and some MBH98 developments, I think of the
1800+ posts at HNN on the former, and would not want to encourage such a distraction here.

Lott did make things up like Bellesiles did. The parallels are kind of eerie. (Lott/Bellesiles) claimed that (brandishing a gun stops crime 98% of the time/gun ownership rates in the early republic were very low) based on (a survey/a study of probate records) he conducted. Unfortunately no-one can check (Lott’s/Bellesiles’) work because he claims that the data was lost in a (disk crash/flood) and indeed there isn’t any good evidence that the (survey/study) was ever conducted at all. You can find out more details from James Lindgren’s (report/review).

A Steve Sailer reader made an interesting comment a while back about the Lott/Bellesiles affair that may also bear on MBH98 vs MM05 controversy:

Bad Social Science? A reader asks:

On your reader’s question about John Lott and gun studies: I have eavesdropped on this argument a bit, far too little to be a useful expert, but enough to be struck by a nagging suspicion. Both Lott and one of his pro-gun-control adversaries, Michael Bellesàƒ⭬es, have apparently been caught using some fraudulent data.

Question: does this mean that:
1) academics who study gun control are unusually dishonest? or
2) gun control is one of the very few social issues so controversial that a competent adversary will actually check the footnotes of an academic’s book or article and try to replicate the results, as is routine in the physical sciences?
If 2, there is likely a huge pile of worthless or fraudulent junk out there on a host of social and political questions that gets cited, quoted and accepted, becoming the basis for the conventional media wisdom and thus much policy. Could such things be?

I’m getting the idea that replicating results may not be so routine in the physical sciences, either.

With all due respect, John, I don’t see anything untoward happening in this thread. If anything the discussion has been trending toward a discussion of due diligence and fact checking, things which are precisely what this site is about. The only one who’s even brought up gun use per se is Steve himself.

1) Dr. Lambert has his own problems, from when he ventured out of his expertise into thermodynamics and the cos vs. cos^2 computational error of MBH. On the latter, he has both reversed (corrected) and then revised his original position on this subject — both a clear lesson in checking carefully before jumping stridently out of one’s expertise.

2) Dr. Lott is now commenting on GW — IMO, this can only lead to an defocusing of his (Gun Policy, Cause and Effect) expertise, already under serious and arguably credible attack.

While I can see the observed double standard of Dr. Lambert’s methodological scrutiny between MBH and Lott (Mr. McIntyre’s point), the important point is flawed AGW paper(s) and how subsequent climate science and national/world policy is still under its grip. IMO, it is still a revisionist’s view to state that MBH ’98 and ’99 played a minor part in the IPCC’01 TAR and the present decade policy discussions… especially since current governments (see thread “Spot the Hockey Stick”) still use this icon (and the IPCC’01 TAR) to prescribe cost-ineffective policy, from its dramatic and insubstantiated (if not incorrect) form!

Hye
I see that you are talking about one of my article in
advances in water resources but only of the comment of Legates.
I must say that i reply to all the statistical issue raised
by legates et al. also in ADWR and their all their issues
HAVE BEEN CLEARLY SHOWN AS WRONG WITHOUT DOUBT

However, you can check by downloading the global runoff data set describes in Labat et al. 2004 at the following web pagehttp://www.lmtg.obs-mip.fr/user/labat/ (see on the left global runoff data set)

Hye
I see that you are talking about one of my article in
advances in water resources but only of the comment of Legates.
I must say that i reply to all the statistical issue raised
by legates et al. also in ADWR and their all their issues
HAVE BEEN CLEARLY SHOWN AS WRONG WITHOUT DOUBT

However, you can check by downloading the global runoff data set describes in Labat et al. 2004 at the following web pagehttp://www.lmtg.obs-mip.fr/user/labat/ (see on the left global runoff data set)

I have also emailed Dr. Labat directly about the situation, we’ll see what transpires. The odd part to me is that not only is the file not there, the “labat” folder in the “foo” directory doesn’t exist.

Thank you for posting your data. Would that all climate scientists were as prompt and gracious as you.

(A note to folks who don’t use commas as thousands separators, don’t try to paste it into Excel. Paste it into a word processor and change the commas to decimal points).

A few initial questions from a first look at the data:

1) Your “Global” figure is not the sum of the continental figures. I can only assume that this difference is the runoff from Greenland, Australia, and Antarctica. Is this assumption correct, or is it an error in the transcription of the data?

2) If this is correct, how did you obtain or estimate this figure?

3) Why does it vary so widely? The standard deviation of the continental annual runoff figures (Asia, Africa, Europe, North and South America) are all between 8-10% of their mean. The standard deviation of their sum is 4% of its mean, in the range of what we’d expect. The standard deviation of the “Global” figure is 5% of the mean. But what I am calling the “Other” category (the difference between the continental totals and the “Global” figure) has a standard deviation of 29% of its mean. Why is that?

4) Finally, each of the continents is positively correlated with at least one (Africa and Europe) or two (Asia, North and South America) other contents. The “Other” category is negatively correlated with all of them.

The data set is posted up in an annoying format. Here’s a R script to produce a cleaned-up version of the table – lots of little ad hoc devices to wade through the html tags. The gsub command replaces commas with a period for use in an as.numeric command.

If I understand well, Dr Labbat’s results, assuming they have some significance thanks to magical wavelets and miracle “reference” correlations capable of recovering missing data, killing spurious trends due to water usage change or river geo-engineering, show that there is MORE water on continents as a direct consequence of global warming ?

If that’s it, so his introductory rhetorics remark “Climate change is one of the major challenge that humanity will have to face over the next decades” contradicts the results of his research.

Hye
i will not reply to “demesure” who is not able
to distinguish my name from a famous bier ….
concerning Willis Eschenbach remarks, all answers are included
in the reference labat et al. (2004). The main answer
could be that all results have been normalised to
baumgartner and Reichel (1975) estimates and that a 1/0.89
coefficient have been applied to global runoff data in order
to take into account Australia runoff.
Best regards

PS : Sorry for the format of the data but i try to answer
as quickly as possible …

May I suggest that you simply archive an ASCII file. Given that there’s a little interest in the issue, why don’t you post up a pdf of your reply to Legates?

#31. Your refusal to reply to demesure because he supposedly confused your name with a famous beer is both petty and wrong. As a Canadian, I can authoritatively say that the beer is spelled “Labatt” and demesure’s “Labbat” is neither. Given that you criticize Demesure for spelling, I presume that you mean the famous “beer”; I am unaware of any famous “biers”, although funeral directors may know of some.

… all answers are included in the reference labat et al. (2004). The main answer could be that all results have been normalised to baumgartner and Reichel (1975) estimates and that a 1/0.89 coefficient have been applied to global runoff data in order to take into account Australia runoff.

I fear that the scope of my research is too broad (and the scope of my wallet is too narrow) to pay $30 for every study of interest, much as I would like to read yours, so instead I have taken another look at your data.

There is a very clear offset of one year between the sum of your continental data (which I will call “Sum”) and the global data of Baumgartner and Richel (“Global”). This is most evident in 1877, when the difference between the Sum and Global data is only 1% of the larger … extremely unlikely. Shifting the Global data back by one year (or your data forward by one year) improves the residual (reduces the standard deviation of the residual by 8%), significantly increases the r^2 between the two datasets from 0.54 to 0.60, and brings the 1877 difference back to 10%, about where we’d expect it to be. I’d say the error is real, and needs to be fixed.

I have to confess, when I see this kind of error in a dataset, I get nervous … it makes me wonder about the problems I haven’t seen …

I was also surprised to see that there is only a tiny and statistically insignificant trend in both the Sum (+0.2% ± 2.4% per century) and Global runoff data (+0.4% ± 2.1% per century). Since there is a bit of a trend in temperature over that time (~ 0.4°C/century), I found this odd. Nor is there any clear relationship with HadCRUT3 temperature data, the r^2 in both cases is less than 0.00.

Finally, since the 95% confidence interval for the trend of both the Sum and Global data is greater than ±4% per century, it would seem difficult to me to tease out some statistical relationship between either dataset and any other variable. I suspect that your conclusions are correct, that precipitation in fact goes up as the world warms. It seems inevitable because there must be more evaporation in a warmer world, which has to go somewhere. However, proving that from the data you have shown seems difficult at best. Not impossible … but certainly difficult.

Hye
I am very pleased about your careful analysis.
Please read my artcile carefully and you will
see that my trend only ocurs above the 1925-1994 period.
Then, Baumgartner and Riechel study has benn published in 1975
so i do not understand your graphic.
Please make the same correlations as me and then we can talk.

Finally, I encourage of course all hydrologists
around the world to collect more and more data in order to make
my estimate better and better …

Hye
I am very pleased about your careful analysis.
Please read my article carefully and you will
see that my trend only deals with the 1925-1994 period.
Then, Baumgartner and Riechel study has been published in 1975
so i do not understand your graphic since this pulbication only gives
estimation of mean annual continental runoff.
Please make the same correlations as me and then we can discuss.
Finally, I encourage of course all hydrologists
around the world to collect more and more data in order to make
my estimate better and better …

#40 Margo, speaking of American cheese (than which not just French cheese but any food is superior), an old joke about Canada ran along the lines that the Founding Fathers wanted to achieve a country with French culture, British know-how and American efficiency. Instead we got French know-how, American culture and British efficiency.

One of my early criticisms of MBH98 – and they’ve still not confessed – had to do with hydrology and France.

Mann allocated a precipitation series from Paris to a New England gridcell; one from Toulouse to South Carolina and one from Marseilles to Spain. I never quite understood his fascination with French precipitation series as proxies for global temperature with remarkable teleconnections. I used the phrase: The rain in Maine falls mainly in the Seine. In the Nature Corrigendum, he admitted that the series did not come from the cited source; instead said that they came from “NOAA” with no further identification.

#41 You must be thinking of “Kraft American Cheese Product Singles” when you write so disparagingly of American cheese, Steve. Kraft “American Cheese” isn’t cheese. It’s not even allowed to be called cheese by law, hence the “cheese product.” That said, there are plenty of excellent cheeses made below the 48th parallel. As a vegetarian, I’ve sampled a number of them. I agree, though, that the very best heart-stopping cheese is made in France, usually containing about 80% butter fat.

Some time back, I ran across a cheese imported from Israel, made in Nazareth. ‘Cheeses of Nazareth’ has had quite a run in our family humor since then. 🙂

MBH98. I really don’t know where the Bombay series came from. A couple of years ago, I calculated correlations of MBH Bombay to historical precipitation series and there was virtually no correlation to actual Bombay and the closest as I recall was Philadelphia, but it wasn’t high enough to prove a match (whereas Paris and Toulouse could be conclusively related to mis-located North American series.)