IF versus EF

I have only recently become aware of the existence of the Eigenfactor (EF). It is a proposed measure of the overall influence, impact, prestige of a scholarly journal in its own discipline, or field. The one and only measure with which I was familiar is the well-known Impact Factor (IF), which is actually fairly straightforward to understand. By contrast, the eigenfactor is determined through a rather complex procedure (I am not going to discuss its computation in this post — for details, see here).

IF and EF aim at measuring the same thing. Quoting from the relevant Wikipedia entries:The impact factor … is frequently used as a proxy for the relative importance of a journal within its field… The Eigenfactor … is a rating of the total importance of a scientific journal…
So, it seems fair to say that if there is a difference in scope, we are mostly talking nuances.
Since they both purport to measure generically “importance”, the obvious question is whether these two indices agree with one another. In other words, given any two science publications in the same field, are the EF and IF generally consistent, when it comes to assessing the relative importance of one versus the other ? Exploring this aspect is purpose of this post — not that of offering an opinion on which one of the two is better (of course there may be many more such measures around of which I am unaware) [0].
Identifying quantitative, as objective as possible measures of performance in any field of human activity, is a problem of general interest to me. As a reasonably active scientist, and as a member myself of editorial boards of two journals, the issue of which yardstick best gauges the impact of a journal (something ultimately so relevant to the careers of many of us [1]) acquires substantial, more immediate practical significance. Should we go by IF or EF when evaluating a candidate’s CV for purposes such as a faculty appointment, promotion, prize ?

In Table I, IF and EF are compared for a set of 40 physics journals. Values of the EF can be obtained online directly from Eigenfactor.org, those of the IF (2010) from the home pages of the journals and/or from Wikipedia. Also included for comparison are the corresponding values for a few prestigious, interdisciplinary science publications. The sample seems sufficiently large to justify some statistical analysis.
The Pearson’s correlation coefficient for EF and IF based on this set of data is 0.1, which in practice means that IFs and EFs do not go together — for instance, a high IF does not correspond to a high EF any more often than it does to a low EF. So, at least one of the two does not really measure what it claims.Table II shows the top six physics journals, as ranked by either EF or IF. Only two journals, Physical Review Letters and Astrophysical Journal, appear at the top of both rankings; all other journals are only regarded as “influential” on going by one of the two scores, not the other.

Which is the right one ? Are they both wrong ?
As I wrote above, I am not going to express an opinion because I do not know enough about the EF.
Aside from the fact that some intelligence has gone into it, one of its alleged properties is that, unlike the IF, which merely counts citations, the EF weighs them differently, depending on where they are made — citations that appear on more influential publications are given a greater weight. Some of the most striking aspects of the EF are the following:

The EF does not seem to like review articles, hence the relatively low score of publications such as the Review of Modern Physics.

The EF does not like for some reason Nature Physics, which is the physics journal of highest IF (except for Reviews of Moden Physics and Advances in Physics, which publish review articles). If the community went by EF, Nature Physics would be regarded as a fairly low key publication. I would be very interested in knowing what it is about Nature Physics that the EF penalizes, or does not reward. Too few articles published, maybe ?

While the IF tends to amplify differences at the very top, the EF does the opposite. For instance, Nature is regarded as roughly five times more influential than Physical Review Letters by IF, but it is only so by roughy 40 percent, based on EF. Here too, if EF were to become the law of the land, I can imagine a huge drop in the number of physics submissions to journals such as Nature, Science, PNAS, given that the payoff in terms of prestige, compared to Physical Review Letters, would no longer be so impressive as it is now, with people mostly looking at IF.

If we call “influential” the journal that we read often, in which we are most likely to find articles that have direct relevance to our own research activity, then the ranking yielded by EF makes much more sense to me than that based on IF. EF seems to reflect reality as perceived by the vast majority of us who do physics as a profession.
It may well be that the difference between EF and IF is that one of the two does not really measure “influence” as much as prestige. A physics article published in Nature [2], for example, may bring its authors recognition and accolade from their university administration and a restricted circle of scientists working in the same area, but perhaps will not set a new trend, spark a collective investigative effort within the community, significantly more than one published in, say, Physical Review Letters.

One thing that seems indisputable, though, is that only one of the two measures will stand the test of time, and emerge as the standard one.

[1] As opposed to leaving it to the idiosyncratic judgment of a small committee, or even of a single individual.

[2] The comparison between a journal like Nature, which is interdisciplinary, and one like Physical Review Letters, remains ultimately meaningless, at least insofar as one does not know what the contribution of physics articles is to the high IF of Nature.

Comparison of Eigenfactors and Impact factors of 40 physics journals. Impact factor values are from 2010. Also shown for comparison are the corresponding data for three prestigious, interdisciplinary science journals (entries in red). Back

Share this:

Like this:

Related

This entry was posted on June 15, 2012 at 10:07 pm and is filed under Academia, Physics, Research, Science. You can follow any responses to this entry through the RSS 2.0 feed.
You can skip to the end and leave a response. Pinging is currently not allowed.

22 Responses to “IF versus EF”

Thank you for the blog post. The reason you observe such dramatic differences between Eigenfactor scores and Impact factor scores is that Eigenfactor and Impact factor DO NOT measure the same thing. Once one recognizes this, your observations become less puzzling.

We developed the Eigenfactor score as a metric of the total importance of a journal, analogous to total count of citations to that journal over a given period. This metric scales with the size of a journal. All else equal, if a journal doubles the number of articles published annually, its Eigenfactor score should double. Such a metric can be useful for a number of purposes, such as making collections decisions about which journals to purchase with a limited library budget.

However, the Eigenfactor score is not the proper metric for deciding which journal is the most prestigious venue for publication. Individual authors or researchers interested in deciding where to send their work will be interested in a journal’s influence per published article. This is what the Impact Factor score measures. All else equal, if a journal doubles the number of articles published annually, its impact factor should remain unchanged.

Another of the Eigenfactor metrics, the Article Influence score, measures per-article influence and thus is more appropriate for comparison with the Impact Factor score. The correlation coefficients between Article Influence and Impact Factor scores are typically in the 0.5-0.99 range, as we describe in Figure 1 of this paper: http://octavia.zoology.washington.edu/people/jevin/Documents/West2010JASIST.pdf

Given that Eigenfactor scores and Impact Factors scores are different tools that do different jobs, the major differences you observe are not as problematic as it might seem at first glance.

thank you for the clarification. The reason I became interested in this subject is that it is felt by many of us that the widespread adoption of the impact factor as a measure of the importance of the publication venue, may have a number of unfortunate consequences, perhaps the most serious being that of putting a number of respectable specialized publications at a disadvantage — in some cases to the point where their own very existence is jeopardized, given the pressure on researchers to go for high IF journals. Granted, much of that stems from the fact that we all have to deal with university administrations all too keen on reducing evaluation to a single number, or restricted set thereof.
So, the notion of possibly replacing the IF with another measure, one giving a more realistic, balanced and fair assessment of the true impact of a publication in a specific area of inquiry, is one that many of us would happily entertain.

Italy were definitely the better team, no doubt about that. I’ve really come to appreciate Andrea Pirlo who really does anchor that team. But Balotelli missed way too many opportunities and Cassano was invisible. Better wake up for the Germans…

OK, listen: true, Mario did miss one great opportunity, though you have to give some credit to both Hart and Terry on that occasion. However, the man constantly had two-three English players marking him, he kept creating favorable situations for us, in which the English defence would be in numerical inferiority, of which Italy did not take advantage, mostly because Cassano was non-existent. I think he played a good game, even with the missing opportunities.

My prediction for the final – De Rossi scores early in the first half from outside the box, ball taking a deflection. Spaniards equalize late in the second on a Pedro header at the near post. Balotelli scores in overtime on a breakaway.

After the game, Spain defaults, there is a bank run in Italy, the European Central Bank and Angela Merkel says, “Screw you all.” The Euro disintegrates, the EU falls apart, chaos ensues, Balotelli decides to play for the Los Angeles Galaxy.

Well, the score seems harsh, I thought a two goal difference was an accurate and fair reflection of a difference which is there and is undeniable. Luck wasn’t on our side either as we suffered two major injuries, but the substance is that Spain is the number one now…

I thought the first goal was fabulous, but you wonder if Chiellini would’ve cut off the run that led to the second goal. I would’ve liked to have seen more of Thiago Motta who is an exciting player.

I’m sure this will get examined to death by the Italian press, but should you be making three substitutions that early in the game? Shouldn’t you at least hold one back for a little while just to see how the second half is developing? Cassano’s injury certainly had important consequences in that respect, because it ties up a substitution right away. But better one half of Cassano than none, I guess…

Look, the Italian squad was outplayed in just about every respect. Yes, we were also unlucky and yes, maybe Prandelli could have made different choices, so that the score could have been kept down to a more reasonable level. Does it make all that much difference ? To me, no. 4-0 is bad, but it isn’t Italy’s fault if they were the team that made it to the final…

My final thoughts…
1) It’s amazing that Spain could be that good without two of their best players (Puyol and David Villa). That’s a deep team…
2) Trust Torres to start playing once the game is over – typical.
3) Italy have to feel pretty good about their chances in the World Cup – with Balotelli, a healthy Cassano, Motta, and a final campaign for Pirlo and Buffon. That’s a pretty damn good team right there.
4) Balotelli needs to get over himself. Coming out after everyone else has lined up for the national anthems, not shaking hands after the game… Dude needs to grow up. Sometimes I wonder if he plays for the nation of Italy or for the Republic of Mario Balotelli.