Paul Wouters and Sarah de Rijcke @ CWTS

Main menu

Monthly Archives: September 2011

Post navigation

“My research: > 185 papers, h-index 40.” A
random quote from a curriculum vitae in the World Wide Web. Sometimes,
researchers love their Hirsch index, better known as the h-index. But what does
the measure actually mean? Is it a reliable indicator of scientific impact?

Our colleagues Ludo Waltman and Nees Jan van
Eck have studied the mathematical and statistical properties of the h-index.
Their conclusion: the h-index can produce inconsistent results. For this
reason, it is actually not the reliable measure of scientific impact that most
users think it is. As a leading scientometric institute, we have therefore
published the advice to all universities, funders, and academies of science to
abandon the use of the h-index as a measure of the overall scientific impact of
researchers or research groups. There are better alternatives. The paper by
Waltman and Van Eck is now available as a preprint and
will soon be published by the Journal of the American Society for Information
Science and Technology JASIST.

The h-index is a measure of a combination of productivity and citation
impact. It is calculated by ordering the number of publications by a particular
researcher on the basis of the total number of citations they have received.
For example, someone who has an h-index of 40 has published at least 40
articles that have each been cited at least 40 times. Moreover, the remaining
articles have not been cited more than 40 times each. The higher the h-index the
better.

The h-index was proposed by physicist Jorge Hirsch in 2005. It was an
immediate hit. Nowadays, there are about 40 variants of the h-index. About one
quarter of all articles published in the main scientometric journals have cited
Hirsch’ article in which he describes the h-index. Even more important has been
the response by scientific researchers using the h-index. The h-index has many
fans, especially in the fields that exchange many citations, such as the
biomedical sciences. The h-index is almost irrresistable because it seems to
enable a simple comparison of the scientific impact of different researchers. Many
institutions have been seduced by the siren call of the h-index. For example,
the Royal Netherlands Academy of Arts and Sciences (KNAW) in the Netherlands inquires
about the value of the h-index in its recent forms for new members. Individual
researchers can look up their h-index based on Google Scholar documents via
Harzing’s website publish or perish. Both economists and computer scientists
have produced a ranking of their field based on the h-index.

Our colleagues Waltman and Van Eck have now shown that the h-index has some
fatal shortcomings. For example, if two researchers with a different h-index
co-author a paper together, it may lead to a reversal of their position in an
h-index based ranking. The same may happen when we compare research groups.
Suppose we have two groups and each member of group A has a higher h-index than
a paired researcher in group B. We would now expect that the h-index of group A
as group is also higher than that of group B. Well, that does not have to be
the case. Please note that we are now speaking of a calculation of the h-index
based on a complete and reliable record of documents and citations. The
problematic nature of the data if one uses Google Scholar as data source is a
different matter. So, even when we have complete and accurate data, the h-index
may produce inconsistent results. Surely, this is not what one wants using the
index for evaluation purposes!

At CWTS, we have therefore drawn the conclusion that the h-index should not be used as measure of scientific
impact in the context of research evaluation.