Where Are They Now? The Healy 2100

A Times Higher Education
piece
making the rounds last week found that most published philosophy papers
are never cited. More exactly, of the studied philosophy papers
published in 2012, more than half had no citations indexed in Web of
Science five years
later.

At Daily Nous, the discussion of that
finding
turned up some interesting follow-up questions and findings. In
particular, Brian Weatherson found quite different
figures
for papers published in prestigious philosophy journals. In the journals
he looked at, 89% of the papers published in 2012 had at least one
citation in Web of Science five years later.1 And more than half had five
or more citations.

But this got me thinking about Kieran Healy’s
analysis
from a few years back. He found an “uncitation” rate higher than
Weatherson’s 11%—almost 20%—even though he was looking at just four of
philosophy’s most prominent journals: Journal of Philosophy, Mind,
Noûs, and Philosophical Review. (He found that around half of the
papers in these journals had five citations or fewer.)

So I wondered: what’s with the discrepancy? Do these journals not
necessarily get the most citations? Or is it that Healy was looking at
papers from 1993 to 2013, and things changed somehow over those two
decades, so that papers published in 2012 tend to get discussed more
than papers from 1993. Or is it just a symptom of when Healy collected
his data? Papers published in, say, 2011 wouldn’t have had much time to
gather citations by 2013 when Healy (apparently?) gathered his data.

Let’s take a look.

The Healy 2100

Since I don’t have Healy’s raw data, I went to Web of Science and
grabbed their data for all papers published over 1993–2013 in the
“Healy 4” journals. I ended up with a couple hundred more papers than
Healy looked at—not sure why. But the list of 2,100 papers he studied
is available on GitHub. So I
focused on just those to get more of an apples-to-apples comparison.

Then I tried to reproduce his original findings, especially this
histogram. Here’s my version:

I got pretty close, but I didn’t manage to reproduce his results
exactly. I found about 19.9% of the papers had no citations by the end
of 2013, compared to Healy’s ~18.5%. And I found about 58.6% with five
or fewer citations, compared with Healy’s “just over half”.

Still, the match is pretty close, so let’s go on to see how these papers
have aged since 2013.

Where Are They Now?

If we include citations up to the present day, only about 9.7% of these
2,100 papers have no citations, and about 39.1% have five or fewer.
Here’s the updated histogram:

So it looks like the discrepancy with Weatherson’s result is (partly?)
down to the obvious thing. There hadn’t been enough time for the later
papers in Healy’s data set to accrue citations.

Cutting Out Supplements

A lot of these 2,100 papers are actually from the two supplements to
Noûs: Philosophical Issues and Philosophical Perspectives. And it
turns out they’re making a big difference.

Here’s what things look like when we cut supplements out:

Now only 3.8% of our 1,677 papers have no citations to date, and just
30.9% have five or fewer.

Sliding Windows

We’ve been looking at all citations accumulated to date for these
papers, which for older papers means 25 years’ worth of opportunity for
discussion. For more direct comparison to the THE analysis mentioned
at the outset, we can look at just the five-year window following each
paper’s publication.

So, how many citations did these papers accrue just within five years of
being published? Looking at only the “core” papers again (no
supplements), 14.4% had no citations within five years of publication,
and 69.6% had five or fewer.

That’s still a bit higher than the 11% (respectively 50%) found by
Weatherson. So we have to ask: are things changing? Are recent papers in
these journals accruing citations faster?

It seems so. Here are the “uncitation” rates for the years 1993–2013:

And here are the “five or fewer citations” rates:

Update: Eric Schliesser pointed out to me what should have been obvious here, namely the internet. When Google took over the search engine industry around the year 2000, there was a citation boom across the board. So you’d naturally expect 2012 to be a very different year than 1993 as far as uncitation goes.

At first I thought the data plainly vindicated the “Google effect” hypothesis, and I had some plots up here to show as much. But it turned out Web of Science had snuck a bunch of supplemental Phil Issues/Phil Perspectives papers past me unmarked!

With those removed, the Google effect isn’t looking so big. Here’s the long view, from 1970 to 2013:

Apparently papers with zero citations after five years have been declining for a while now. There does still seem to be a significant Google effect in the “five or fewer” measure:

But it still looks like there were changes before Google. Does this reflect a growing profession? A faster-moving profession? An increasing number of publications? Longer bibliographies? Just idiosyncracies of Web of Science’s indexing? Something else? I’m not sure.