Google Scholar illiteracy in the PNAS

A recent paper published in the PNAS, “Expert credibility in climate change” is being used as propaganda to claim that 97% of all climate scientists agree with the IPCC and the need for government action on climate change. An analysis of this paper does not support these conclusions.

PNAS reviewers and author’s William R. L. Anderegg, James W. Prall, Jacob Harold and Stephen H. Schneider are apparently Google Scholar illiterate since searching for just the word “climate” with an author’s name will bring results from non-peer-reviewed sources such as books, magazines, newspapers, patents, citations, duplicate listings and all sorts of other erroneous results. Such as 16,000 from the Guardian, 52,000 from Newsweek and 115,000 from the New York Times. There is no “peer-reviewed journal only” search option in Google Scholar.

Cherry Picking:

It is clear the authors cherry picked away skeptics using subjective criteria,

“we imposed a 20 climate-publications minimum to be considered a climate researcher.”

So if a scientist published only 19 or less papers on the climate he is not considered an “expert”. They did this intentionally as they noted,

Volume of publications does not indicate scientific truth nor does it denote expertise. It cannot be ignored that skeptics extensively publish peer-reviewed papers so they have to use this propaganda to subjectively define “experts”. An objective criteria for determining if an author has done climate research would be if an author has or has not published a paper on the climate. Expertise is simply an opinion and who is considered an expert will change based on who you talk to.

Climate Patents:

By default Google Scholar is set to search both “articles and patents” yet no mention of searching only for articles is in the paper.

So why were they searching for patents and how is a patent that contains the search word “climate” a relevant “climate publication”?

Verification:

An attempt to reproduce the results using their methods was unsuccessful,

“we collected the number of climate-relevant publications for all 1,372 researchers from Google Scholar (search terms: “author:fi-lastname climate”)”

Clearly these were not papers by Phil Jones of the CRU on climate change.

Looking on Prall’s list of IPCC AR4 Working Group 1 Authors referenced from their Supplemental Information you see Phil Jones listed with 724 climate publications not the 6,580 that I found using their search method. A link is provided under “GS queries” for Phil Jones labeled “CLIM”, clicking on this link brings a surprising revelation, the search term is changed to “author:PD-Jones climate”. When their paper explicitly said “author:fi-lastname climate” and no mention is made of including the middle initial. It appears Prall added the middle initial arbitrarily to the authors on the list further undermining the consistency of their results. Using this search term I again searched Google Scholar,

Citations for this book are counted over 20 times in Google Scholar, author:PD-Jones “Climate since AD 1500” further inflating the erroneous results. No mention of turning off citations is in their paper as this feature is on by default in Google Scholar and in the “CLIM” link from Prall’s page.

The climate total number of 724 for Phil Jones on Prall’s list is unverifiable using the methods outlined in their paper and appears to be made up.

It is clear they used the total number of climate publications because this is explicitly stated in their paper,

“We ranked researchers based on the total number of climate publications authored.”

However no verification of these results was done by the authors as they only mention,

“We verified, however, author identity for the four top-cited papers by each author.”

It appears they only verified the top four results for their “citation analysis” not for the total amount of results using the search word “climate” for each of the 1372 authors. As demonstrated here, without complete verification it is not possible to draw any meaningful conclusions that include erroneous results.

Conclusion: the study is worthless due to Google Scholar illiteracy and Cherry Picking.

This entry was posted
on Friday, August 6th, 2010 at 4:07 am and is filed under Uncategorized.
You can follow any responses to this entry through the RSS 2.0 feed.
Responses are currently closed, but you can trackback from your own site.