Monday, July 27, 2009

Journal prestige metrics

Occasionally when I discuss an article with my adviser, she asks me a question that completely mystifies me. "What journal was this article published in?" she'll ask. "Journal of Experimental Social Psychology," I'll reply. Then she nods knowingly as if incomparably important information has just been conveyed and continues talking.

I am left wondering throughout this entire experience what the heck just happened.

For non-graduate students, my problem might not be exactly clear. You see, no one ever sits us poor graduate students down to tell us "Psychological Science publishes short reports that are likely to start new lines of research" or "Journal of Personality and Social Psychology is the most prestigious journal in social psychology". We often need to figure out all of this information by blind groping.

But wait a second. Journals are published by scientists whose careers depend on knowing this kind of information. Surely some enterprising scientist has assembled some systematic information about which journals are important in which fields . . .

Indeed, many people have aggregated this information. In fact, they have aggregated competing sources of information, which has spawned a whole contentious field of journal influence measurement. Many measures of journal influence exist, the most well-known of which is probably the impact factor. This statistic is calculated by Thomson Corporation, which has now merged with Reuters. Thomson Reuters maintains an index of scientific journals. The impact factor of a given journal is roughly calculated using the index according to the following algorithm:

A = number of times articles published in a given journal were cited by indexed journals over the past two years
B = number of total citable items in index over the past two years
Impact factor = A / B

So, a journal that publishes articles that are cited roughly once per year will achieve an impact factor of 1. The above explanation is an oversimplification of the actual calculations, of course, but it conveys the basic idea behind the impact factor.

However, there are several problems with the impact factor. First of all, the two year window may be entirely too small - classic articles are often not recognized as such upon initial publication, and when they are, they can generate citations for years (especially when the author of said articles is one's adviser). Additionally, journals can game the system by publishing a large proportion of review articles, which do not report on new data but are useful to cite because they aggregate large portions of a given topic into one place.

The worst problem with the impact factor, however, is that Thomson Reuters treats the statistic as proprietary information. Bastards.

Enter Eigenfactor. Eigenfactor is a PageRank algorithm that operates somewhat like that of Google fame. Journals are weighted according to not only the sheer number of citations received, but also by who does the citing. Thus, journals cited by Nature can feel smug about joining the lofty ranks of the scientific elite, whereas journals cited by the Journal of Parapsychology are not quite so lucky.

There are two additional cool things about Eigenfactor:

1. The information is open source
2. The visualizations on the Eigenfactor website are pretty dang cool

To give you a sense of which journals Eigenfactor picks out as "important", here are the top 10 general science and social science journals by Eigenfactor rank (as of 2006):

Top 10 non-review science journals byEigenfactor™ score (2006)

Top 10 social science journals byEigenfactor™ score (2006)

1. Nature

1. American Economic Review

2. Science

2. Journal of Finance

3. Proceedings of the National Academy of Sciences USA

3. Journal of Personality and Social Psychology

4. Journal of Biological Chemistry

4. Quarterly Journal of Economics

5. Physical Review Letters

5. Journal of Financial Economics

6. Journal of the American Chemical Society

6. Social Science & Medicine

7. Physical Review B

7. Journal of Political Economy

8. Applied Physics Letters

8. Psychological Science

9. New England Journal of Medicine

9. Journal of Economic Theory

10. Astrophysical Journal

10. Journal of Consulting and Clinical Psychology

Nice to see my social psychology homeboys at JPSP cracking the ranks of the top 10 social science journals. Note that at the Eigenfactor website you can see the relative Eigenfactor scores of just about any journal ever (for reference, Nature has an Eigenfactor score of 1.9917, while the Journal of Personality and Social Psychology has an Eigenfactor score of .07281. In contrast to the JPSP, the Journal of Urology has an Eigenfactor score of .1169. Burn.)

Now for some visualizations. I give you A MAP OF SCIENCE:

Click to embiggen

This visualization represents citation flow between journals categorized by discipline. If you click through to the page giving the full visualization, you will find an interactive version. Hovering over over segments from the outer ring will yield each discipline's Eigenfactor score, whereas hovering over the inner right will yield a given journal's Eigenfactor score. Line size and opacity represent mutual citations between journals.

What's interesting from my perspective is how tiny psychology is in the giant universe of science -- psychology is dwarfed by physics, and even more so by molecular biology. It's humbling to think that a field I care about very much is really quite tiny in the grand scheme of things.