Main menu

The value of negative citations — Christian Catalini

Understanding which papers attract critical citations, and what effect they have, gives an insight into how science progresses, says Christian Catalini.

Science advances through researchers sharing their work for others to extend or improve. As Isaac Newton once said, he could see further by “standing on the shoulders of giants”.

But what happens when those shoulders aren’t as sturdy as we thought? Sometimes, citations are negative, pointing out a study’s flaws or even disproving its findings. What role, relevance and impact do these negative citations have on a field as a whole?

There has been little research in this area, because of the difficulty in identifying and classifying such citations. Thanks to advancements in the ability of computers to understand human language, known as natural-language processing, and in the ability to sort and analyse large bodies of text, this is changing. We can now identify such citations and reconstruct the context in which they were made to understand the author’s intentions better. Using such techniques, my colleagues and I have found evidence to suggest that negative citations play an important role in the advancement of science.

We hypothesised that negative citations help science progress through their role in limiting and correcting previous results. To test this idea, we looked at 762,355 citations from 15,731 articles in The Journal of Immunology. Using a combination of natural-language processing and experts in the field, we identified 18,304 negative citations, or about 2.4 per cent of the total. We also found that about 7 per cent—not a trivial proportion—of these papers received at least one negative citation.

Several features of these negative citations support our hypothesis. A paper is most likely to receive a negative citation in the first few years after publication. This is probably because this is when the science is potentially newer and untested, and thus attracts more attention and scrutiny.

We also found that negatively cited studies were of higher quality and prominence, as captured by the overall number of citations received, a broadly used proxy for scientific impact. This might be because scientists pay more attention to recent, high-impact studies, and so they are also more likely to provide criticisms, extensions, and qualifications to their results.

Christian Catalini is the Fred Kayne (1960) Career Development Professor of Entrepreneurship and Assistant Professor of Technological Innovation, Entrepreneurship, and Strategic Management at the MIT Sloan School of Management.