February 16, 2005

Poor Scientific Review

Vol. 81, No. 6 (June 2000) of the Bulletin of the American Meteorological Society contains an extremely eye opening article on poorly conducted, peer reviewed science. According to the article, meteorology has been on a 20 year slide in scientific quality with more and more papers being met by fewer and fewer comments (a drop of more than 50% in commented papers in that period). The lack of comments seems to have had some serious effect as to scientific validity.

I have used the terms “sometimes,” “may,”
“some,” etc., in my comments here. If these vague
words referred to less than 20% of papers, perhaps that
would be acceptable. Some poor papers would get
through any review process. In research topics that I
read, however, 50% may be a closer estimate of misleading
or fundamentally wrong papers. Such a number
is not easy to evaluate. If it is this high, however,
we have a problem that should not be ignored.

I was led to this paper by a comment on a general item on the poor state of peer review, especially in climate science. Steve McIntyre undertook to verify a central study in the IPCC report, the MBH 98,99 "hockey stick" graph and uncovered enough error to pretty conclusively demonstrate that (irrespective of what theory is ultimately right or wrong) the entire world has embarked on a global warming crusade without checking for scientific validity.

Huge diplomatic rows between the US and so many countries in Europe may be based on bad science. Global growth may be cut needlessly and millions in the third world prolonged needlessly in their poverty in part due to papers that can't survive outsider fact checking. The two sides in that controversy have set up dueling blogs with the pro-warming side being answered by McIntyre's show your work site.