Tuesday, 24 February 2015

The H-index is a metric that was devised to identify talented individuals whose published work had made a significant impact on the field (Hirsch, 2005). One of its apparent virtues was that it was relatively difficult to game. However, analysis of publications in a group of journals in the field of developmental disabilities suggest there has been a systematic and audacious attempt at gaming the H-index by a cabal of editors.

What's the evidence for this claim? Let's start by briefly explaining what the H-index is. It's computed by rank ordering a set of publications in terms of their citation count, and identifying the point where the rank order exceeds the number of citations. So if a person has an H-index of 20 this means that they've published 20 papers with at least 20 citations – but their 21st paper (if there is one) had fewer than 21 citations.

The reason this is reckoned to be relatively impervious to gaming is that authors don't, in general, have much too much control over whether their papers get published in the first place, and how many citations their published papers get: that's down to other people. You can, of course, cite yourself, but most reviewers and editors would spot if an author was citing themselves inappropriately, and would tell them to remove otiose references. Nevertheless, self-citation is an issue. Another issue is superfluous authorship: if I were to have an agreement with another author that we'd always put each other down as authors on our papers, then both our H-indices would benefit from any citations that our papers attracted. In principle, both these tricks could be dealt with: e.g, by omitting self-citations from the H-index computation, and by dividing the number of citations by the number of authors before computing the H-index. In practice, though, this is not usually done, and the H-index is widely used when making hiring and promotion decisions.

In my previous blogpost, I described unusual editorial practices at two journals – Research in Developmental Disabilities and Research in Autism Spectrum Disorders – that had led to the editor, Johnny Matson achieving an H-index on Web of Science of 59. (Since I wrote that blogpost it's risen to 60). That impressive H-index was based in part, however, on Matson publishing numerous papers in his own journals, and engaging in self-citation at a rate that was at least five times higher than typical for productive researchers in his field.

It seems, though, that this is just the tip of a large and ugly iceberg. When looking at Matson's publications, I found two other journals where he published an unusual number of papers: Developmental Neurorehabilitation (DN) and Journal of Developmental and Physical Disabilities (JDPD). JDPD does not publish dates of submission and acceptance for its papers, but DN does, and I found that for 32 papers co-authored by Matson in this journal between 2010 and 2014 for which the information was available, the median lag from a paper being received and it being accepted was one day. So it seemed a good idea to look at the editors of DN and JDPD. What I found was a very cosy relationship between editors of all four journals.

Figure 1 shows associate editors and editorial board members who have published a lot in some or all of the four journals. It is clear that, just as Matson published frequently in DN and JDPD, so too did the editors of DN and JDPD publish frequently in RASD and RIDD. Looking at some of the publications, it was also evident that these other editors also frequently co-authored papers. For instance, over a four-year period (2010-2014) there were 140 papers co-authored by Mark O'Reilly, Jeff Sigafoos, and Guiliano Lancioni. Interestingly, Matson did not co-author with this group, but he frequently accepted their papers in his journals.

Figure 1: N papers authored by each individual 2010-2014 for 4 journals.
Orange denotes main editor, yellow associate editor, and tan a member of editorial board. Sigafoos moved from editor to editorial board of DN in this period.

Figure 2 shows the distribution of publication lags for the 140 papers in RASD and RISS where authors included the O'Reilly, Sigafoos and Lancioni trio. This shows the number of days between the paper being received by the journal and its acceptance. For anything less than a fortnight it is implausible that there could have been peer review.

Figure 2
Lag from paper received to acceptance (days) for 73 papers co-authored by Sigafoos, Lancioni and O'Reilly, 2010-2014

Papers by this trio of authors were not only accepted with breathtaking speed: they were also often on topics that seem rather remote from 'developmental disabilities', such as post-coma behaviour, amyotrophic lateral sclerosis, and Alzheimer's disease. Many were review papers and others were in effect case reports based on two or three patients. The content was so slender that it was often hard to see why the input of three experts from different continents was required. Although none of these three authors achieved Matson's astounding rate of 54% self-citations, they all self-cited at well above normal rates: Lancioni at 32%, O'Reilly at 31% and Sigafoos at 26%. It's hard to see what explanation there could be for this pattern of behaviour other than a deliberate attempt to boost the H-index. All three authors have a lifetime H-index of 24 or over.

One has to ask whether the publishers of these journals were asleep on the job, not to notice the remarkable turnaround of papers from the same small group of people. In the Comments on my previous post, Michael Osuch, a representative of Elsevier, reassured me that "Under Dr Matson’s editorship of both RIDD and RASD all accepted papers were reviewed, and papers on which Dr Matson was an author were handled by one of the former Associate Editors." I queried this because I was aware of cases of papers being accepted without peer review and asked if the publisher had checked the files: something that should be easy in these days of electronic journal management. I was told "Yes, we have looked at the files. In a minority of cases, Dr Matson acted as sole referee." My only response to this is, see Figure 2.

ReferenceHirsch, J. (2005). An index to quantify an individual's scientific research output Proceedings of the National Academy of Sciences, 102 (46), 16569-16572 DOI: 10.1073/pnas.0507655102

P. S. I have now posted the raw data on which these analyses are based here.

P.P.S. 1st March 2015

Some of those commenting on this blogpost have argued
that I am behaving unfairly in singling out specific authors for criticism.
Their argument is that many people were getting papers published in RASD
and RIDD with very short time lags, so why should I pick on Sigafoos, O'Reilly,
and Lancioni?

I should note first of all that the argument 'everyone's
doing it' is not a very good one. It would seem that this field has some pretty
weird standards if it is regarded as normal to have papers published without
peer review in journals that are thought to be peer-reviewed.

Be this as it may, since some people don't seem to have understood the blogpost, let me state more explicitly the reasons why I have singled out these
three individuals. Their situation is different from others who have achieved
easy reviewer-free publication in that:

1. Their publications don't only seem to get into RIDD and
RASD very quickly; they also are of a quite staggering quantity – they eclipse
all other authors other than Matson. It's easy to get the stats from Scopus, so
I am showing the relevant graphs here, for RIDD/RASD together and also for the
other two journals I focused on, Developmental Neurorehabilitation and JDPD.

2. All three have played editorial roles for some of these four
journals. Sigafoos was previously editor at Developmental Neurorehabilitation,
and until recently was listed as associate editor at RASD and JDPD. O'Reilly is editor of JDPD, is on the
editorial board of Developmental Neurorehabilitation and was until 2015 on the
editorial board of RIDD.Lancioni was
until 2015 an associate editor of RIDD.

Now if it is the case that Matson was accepting papers for
RASD/RIDD without peer review (and even my critics seem to accept that was
happening), as well as publishing a high volume of his own papers in those
journals, then that is something that is certainly not normally accepted
behaviour by an editor. The reasons why journals have editorial boards is
precisely to ensure that the journal is run properly. If these editors were
aware that you could get loads of papers into RASD/RIDD without peer review,
then their reaction should have been to query the practice, not to take
advantage of it. Allowing it to continue has put the reputation of these journals at risk. You might say why didn't I include other associate editors or
board members? Well, for a start none of them was quite so prolific in using
RASD/RIDD as a publication outlet, and, if my own experience is anything to go
by, it seems possible that some of them were unaware that they were even listed
as playing an editorial role.

Far from raising questions with Matson about the lack of
peer review in his journals, O'Reilly and Sigafoos appear to have encouraged
him to publish in the journals they edited. Information about publication lag
is not available for JDPD; in Developmental
Neurorehabilitation, Matson's papers were being accepted with such lightning speed
as to preclude peer review.

Being an editor is a high status role that provides many
opportunities but also carries responsibilities. My case is that these were not
taken seriously and this has caused this whole field of study to suffer a major
loss of credibility.

I note that there are plans to take complaints about my
behaviour to the Vice Chancellor at the University of Oxford. I'm sure he'll be
very interested to hear from complainants and astonished to learn about what
passes for acceptable publication practices in this field.

P.P.P.S 7th March 2015I note from the comments that there are those who think that I should not criticise the trio of Sigafoos, O'Reilly and Lancioni for having numerous papers published in RASD and RIDD with remarkably short acceptance times, because others had papers accepted in these journals with equally short lags between submission and acceptance. I've been accused of cherry-picking data to try and make a case that these three were gaming the system.As noted above, I think that to repeatedly submit work to a journal knowing that it will be published without peer review, while giving the impression that it is peer-reviewed (and hence eligible for inclusion in metrics such as H-index) is unacceptable in absolute terms, regardless of who else is doing it. It is particularly problematic in someone who has been given editorial responsibility. However, it is undoubtedly true that rapid acceptance of papers was not uncommon under Matson's editorship. I know this both from people who've emailed me personally about it, and there are also brave people who mention this in the Comments. However, most of these people were not gaming the system: they were surprised to find such easy acceptance, but didn't go on to submit numerous papers to RASD and RIDD once they became aware of it. So should I do an analysis to show that, even by the lax editorial standards of RASD/RIDD, Sigafoos/O'Reilly/Lancioni (SOL) papers had preferential treatment? Personally I don't think it is necessary, but to satisfy complainants, I have done such an analysis. Here's the logic. If SOL are given preferential treatment, then we should find that the acceptance lag for their papers is less than for papers by other authors published around the same time. Accordingly, I searched on Web of Science for papers published in RASD during the period 2010-2014. For each paper authored by Sigafoos, I took as a 'control' paper the next paper in the Web of Science list that was not authored by any of the six individuals listed in Table 1 above, and checked its acceptance lag. There were 20 papers by Sigafoos: 18 of these were also co-authored by O'Reilly and Lancioni, and so I had already got their acceptance lag data. I added the data for the two additional Sigafoos papers. For one paper, data on acceptance lag was not provided, so this left 19 matched pairs of papers. For those authored by Sigafoos and colleagues, the median lag was 4 days. For papers with other authors, the median lag was 65 days. The difference is highly significant on matched pairs t-test; t = 3.19, p = .005. The data on which this analysis was based can be found here.I dare say someone will now say I have cherrypicked data because I only analysed RASD and not RIDD papers. To that I would reply, the evidence for preferential treatment is so strong that if you want to argue it did not occur, it is up to you to do the analysis. Be warned, checking acceptance lags is very tedious work.

Sunday, 1 February 2015

Last October, I was surprised to see a tweet from
@autismcrisis (Michelle Dawson) saying "Belatedly noticed: @deevybee is on the
editorial board of Johnny Matson's RASD?! Well I'm speechless. Wow."

Many of those reading this tweet were puzzled as to what
this was all about, but I was aware that Michelle had earlier drawn attention
to an odd feature of the journal Research in Autism Spectrum Disorders (RASD):
an unusually high proportion of the papers in the journal were authored by the
Editor, Johnny Matson*.

People were even more puzzled by my reaction, which was to
express surprise and to thank Michelle for telling me. Surely, you might think,
you'd know if you were on the Editorial Board of a journal. But, actually,
that's not always the case. I had no recollection of having joined the
Editorial Board, but my memory for the past is not good. I did remember having
some correspondence with Johnny Matson around 2008, about a nice article he'd
written concerning problems with some of the 'gold standard' diagnostic
instruments (see this
blogpost for more on this). I suspected that I'd agreed to serve on the
Editorial Board in the course of this email exchange, but I had no details of
this on file. Journals vary considerably in how far they treat their Editorial Board
as emblematic, and how far they actually make use of the board in decision-making.
I had barely had any interaction with Research in Autism Spectrum Disorders
over the years, and had remained sufficiently unaware of my Editorial Board
role to omit this from my curriculum vitae. But there I was, as Michelle had
noted, listed as a member of the Editorial Board on the journal website.

I realised that I had to take action for two reasons. First,
RASD was an Elsevier journal. I had been convinced by the arguments
of Tim Gowers to sign the Cost of Knowledge
statement, pledging not to support Elsevier, in the light of their exorbitant
pricing of journals. In 2012 I had resigned from the editorial boards of other
Elsevier journals. But I'd been unaware that I was on the Editorial Board of
RASD, so I had not written to them. Second, I was interested in Michelle's
concerns about editorial practices at the journal. I thought I had better check
her claims out for myself. What I found was disturbing.Matson had been editor of Research in
Developmental Disabilities (RIDD) since the journal's inception in 1987, and he
then took up editorship of the sister journal, Research in Autism Spectrum
Disorders when it started in 2007. As can be seen in Figure 1, his publication
rate shot up around 2007, and an analysis using Web of Science reveals that many
of these papers were published in RASD or RIDD. Indeed, Matson is an author on
over ten percent of the papers published in RASD since 2007.

Figure 1: Publications identified from Web of Science**

Around 2007, Matson's papers also started being highly cited
as can be seen in Figure 2.

Figure 2: Matson citations from Web of Science**

On Web of Science he currently has an H-index of 59:
those of you who know about these things will recognise this as an impressive
value that would usually be indicative of a scientist who does highly
influential work. His University webpage proudly displays the 'Highly Cited' badge from Thompson Reuters. However, there is something unusual about Matson's citation
profile: just over half of the citations are self-citations. I did some comparisons
with other top scientists in the field of autism/intellectual disabilities, and
it is clear that Matson is an outlier in terms of his rate of self-citation
(see Figure 3).

I wrote to Matson to query my editorial board status and to
explain why I wished to resign and received a curt reply, confirming that I had
agreed to be on the Editorial Board, but he would remove me. Nevertheless, my
name remained on the Editorial Board list of the journal website for a while. But
then, a few weeks ago, there was a new development. For both
RIDD and RASD, the pages on the journal
website showing information about the Editors and Editorial Board disappeared.
What, I wonder, is going on?

* (update 22 Mar 2015): Michelle's tweets on this topic can be accessed from the sidebar on her blogpost
**Search terms were AUTHOR: (matson j*) ANDTOPIC: (autism OR intellectual)

***Here I simply selected well-known researchers whose names were distinctive enough to make a search unchallenging. Where necessary to disambiguate authors I added the same Topic search term as for Matson

Update 5th February 2015In the past few days there have been some new developments. On 2nd Feb, Alicia Wise (@wisealic) of Elsevier responded to a concerned tweet from @DavidPriceUCL to say Hi, David - colleagues have been speaking to community about new Editors-in-Chief; appointments will be announced shortly.

Then on 4th Feb, Michelle Dawson (@autismcrisis) tweeted:RASD finally has (cryptic) editorial board info for March 2015 issue http://j.mp/1BZQC1D compare to Feb 2015 issue http://j.mp/1zQNY0DYou have to download the pdfs to see the information. The Feb issue list Matson as editor and gives the full editorial board. The March issue deletes the editorial board but retains Matson as editor. We might, from Alicia Wise's tweet, have expected the opposite.

I have, of course, no idea if these changes relate to anything in this blogpost.

Update 7th February 2015Michelle Dawson pointed out on Twitter that back in June 2013 she suggested that someone should look at how far the impact factor of RASD and RIDD is affected by Matson's self-citations. I doubt there'd be much effect on RIDD, where Matson's papers constitute only around 1% of articles, but I did the sums for RASD. Readers who have access to Web of Science can see the historical impact factor data here. I simply recomputed the data after searching for papers in a 2-year period with the search term NOT Matson J* as author. I had to read the N citation data from the bar plot you get from citation reports in Web of Science, so precision not guaranteed, but here's what it looks like:

P.S. I am hearing on the grapevine that people have had papers accepted in RASD and RIDD without reviewing, but nobody seems willing to say that publicly. If there are any brave souls out there who are prepared to speak out, please can you do so via Comments. Thanks.

Update 14th February 2015

In the comments below, Michael Osuch, Publishing Director for Neuroscience & Psychology journals at Elsevier, has added some notes of clarification. He states: "Under Dr Matson’s editorship of both RIDD and RASD all accepted papers
were reviewed, and papers on which Dr Matson was an author were handled
by one of the former Associate Editors. Dr Matson and his team of
Associate Editors stepped down at the end of 2014."
The issue of Matson's papers being handled by an Associate Editor
is key. When discussing this point, some people have argued that
editors should never publish in their own journal. I disagree, and think it is
reasonable provided (a) it is a relatively rare occurrence and (b) the paper is
handled by an Associate Editor (AE) who ensures that the paper is subjected to
a rigorous review. In such cases it is crucial that the AE is
someone who will not be unduly influenced by their relationship with the editor.

In this regard it is worrying to see that two people who
were, until 2015, the first listed AEs on both RASD and RIDD are relatively
junior with close links to Matson.Thompson Davis III is an associate professor in Matson's department with
expertise in autism and anxiety disorders. To identify his
publications on Web of Science I searched for AUTHOR:
(Davis T*) AND TOPIC: (autism OR anxiety) AND ADDRESS: (Louisiana State
University). This brought up 41 publications. Only a few of these are published in RASD or RIDD (five papers from 2007 to 2011), but he has eleven
co-authored papers with Matson. Jill Fodstad's CV indicates she took the
Clinical Psychology program at Louisiana State University, where it seems
that Matson encouraged students to co-author papers with him. (This is evident
from an analysis of his coauthors in RASD/RIDD). She is now based at Indiana
State University. 55 of her 59 publications
are co-authored with Matson, and 30 of them are published in RASD or RIDD.

Good editorial practice would dictate that neither of these AEs should be assigned
papers authored by Matson because the unequal power relationship with him would make it extremely difficult
to give a dispassionate appraisal of the work.

Of course, I do not know which AEs did handle Matson's
papers, but I would be surprised if an experienced senior editor would
have accepted his manuscripts without challenging the high rate of
self-citation.

Update 15th February 2015
An email from a colleague reads as follows re another Associate Editor at RASD, Jeff Sigafoos:"I have heard of several colleagues having papers accepted extremely quickly at Developmental Neurorehabilitation. The editor at that time was Jeff Sigafoos, but I think he has since stepped down from that position. Anyway, I looked up Jeff Sigafoos website...look at his publication list: http://www.victoria.ac.nz/education/about/staff/publications-jeff-sigafoos ...an amazing number of papers in RIDD and RASD. "

I also checked the submission/acceptance lag for Sigafoos' papers in RIDD. For the first 20 I found, 7 were accepted within one day and a total of 14 within one week. This does rather question the claim by Michael Osuch that Matson acted as sole referee 'in a minority of cases'.
I should add that Developmental Neurorehabilitation is not an Elsevier journal: it is published by Informa Healthcare.
Finally, you might ask whether anyone is hurt by this. I think the answer is that it is damaging to other people who published in any of these journals in good faith, and who now will have the validity of their work questioned because of inadequate peer review.