Assessing the role of negative citations in science

October 26, 2015

The number of times academic articles are cited by subsequent publications is among the time-honored measures used to assess scholarly standing and evaluate academic productivity. But not all of these citations are positive ones, and a paper published this week in the journal Proceedings of the National Academy of Sciences finds that as many as one in 50 citations in a top immunology journal were critical in nature.

These negative citations may point out limitations, inconsistences or flaws in previous work. The study found that these negative citations were more likely to criticize highly-read papers, and that the criticisms focused on specific sections of the papers. Negative citations tended to originate from scholars who were close to the authors of the original articles in academic discipline and social distance - but at least 150 miles away geographically.

The research, by authors at the Georgia Institute of Technology, University of Toronto and Massachusetts Institute of Technology, may be the first to systematically quantify and examine these negative citations. The authors hope to expand their study to other disciplines, and believe their work could ultimately lead to a re-examination of how citations are used in academia.

"Given that we rely so heavily on these citation metrics as measures of quality, it's important to note that the intent of these citations isn't homogeneous," said Alexander Oettl, an assistant professor in Georgia Tech's Scheller College of Business and one of the study's three co-authors. "Criticism is really the cornerstone of the scientific enterprise, yet it's difficult to get accurate measures of the extent to which criticism of academic papers truly occurs. The negative citations really are different from objective citations."

The researchers, who also include Nicola Lacetera from the University of Toronto and Christian Catalini from MIT, began with 15,731 full-text articles that had been published in the Journal of Immunology, the top academic journal in that field. From those articles, they extracted 762,355 citations, which referred to 146,891 unique published papers.

Beginning with a manual process, they created a set of citations that was used to train a natural language processing program which classified the remainder of the citations as either objective or negative. The process identified 18,304 negative citations, about 2.4 percent of the total citations studied.

"The majority of these negative citations appear to take issue with the 'results' and 'discussion' sections of the papers, so they are really not attempting to overturn theory," said Oettl. "They can be attempting to constrain results, note inconsistencies with other research, point out statistical flaws or correct other issues."

In their study, the authors point out the relative rarity of these negative citations, and speculate that they could either demonstrate a "limited, uninfluential role" for criticism in science - or show a hesitancy to criticize.

"Making these negative citations isn't without cost," Oettl noted. "There could be reputational harm from making these negative citations, and if your criticism turns out to be false, this could heavily impact your reputation within the field. Without this cost, perhaps we might see more overt criticism appearing in the journals."

The researchers found that while negative citations tended to come from scientists close to the narrow academic topic, the criticism was more likely to involve geographic distance.

"We see that the probability of making a negative citation is much, much lower if you are co-located with the scientist whose work is being critiqued," Oettl said. "That potentially speaks to the social component, the social cost of criticism - you don't want to criticize someone you may run into on campus. Another possible interpretation is that these issues may be aired face-to-face among scientists who are located near each other geographically."

In the study, frequently-cited papers were more likely than less-read papers to get negative citations, which stands to reason, Oettl said. "These are typically more influential papers, so this may have more to do with more people reading them and more of an incentive for scholars to take issue with important papers," he explained. "Furthermore, less important papers may receive less scrutiny, as pointing out limitations and shortcomings of trivial work will not drastically shift the scientific frontier."

Replication of previous work helps ensure the accuracy of research, and criticism of published studies can lead to correction that makes science more robust. But does the current scientific publishing system encourage enough discussion?

"The pessimistic view is that only one in 50 citations is negative in nature, so possibly there isn't as much debate as we would want for a healthy discussion," Oettl said. "But on the other hand, by the time a major manuscript makes it into print, it has undergone a tremendous amount of criticism. We now have some evidence of the extent to which criticism, in the form of negative citations, occurs within scientific manuscripts."

The researchers chose immunology because they had access to the papers, which were consistent in format. They found similar numbers of negative citations in a small set of mathematics-related papers, but hope to expand their work to determine if their findings apply to other disciplines. Ultimately, they hope the work could change the way citations are used in science.

"We wanted to put forward a methodology to help future scholars think about how we can possibly expand this classification infrastructure and attract additional metadata to the citations that we make," Oettl said. "Our paper is very descriptive, but it identifies something that nobody has really demonstrated before: that these negative citations behave very differently from others."

Though the current study focused on a single discipline, immunology, "we would really like to see this work and methodology expanded to other disciplines," Oettl said. "Ultimately, though, we care about what role these negative citations play in aiding scientific debate and in the advancement of the scientific frontier."
The paper was published October 26 in the Early Edition of the journal.

CITATION: Christian Catalini, Nicola Lacetera, Alexander Oettl, "Beyond citation counts: Exploring negative citations in science," (Proceedings of the National Academy of Sciences, 2015).

Georgia Institute of Technology

Related Immunology Articles from Brightsurf:

Call for immunology to return to the wild
In an article published today in Science, a multidisciplinary research team from more than 10 universities and research institutes outlines how integrating a more diverse set of species and environments could enhance the biomedical research cycle.

What underlies differing disease severity in COVID-19?
Researchers have reported a significant and positive relationship between the amount of virus present in a throat swab sample and the severity of COVID disease.

AGA and joint task force on allergy-immunology practice parameters release EoE guidelines
Eosinophilic esophagitis (EoE) is a chronic inflammatory disease of the esophagus triggered by allergens that causes difficulty with swallowing in adults, which grows more frequent and intense over time, affecting patients' quality of life.

Immunology -- not just supporting actors
Non-classical monocytes were long thought to play a purely surveillance role in the immune system.

Electronic consultations can streamline, simplify care in allergy and immunology
A study from Massachusetts General Hospital researchers finds that electronic consultations in allergy and immunology can simplify the process of providing the most appropriate care, often reducing the need for in-person specialist visits.

Combatting brain infections in special issue of Viral Immunology
A special issue of Viral Immunology, a peer-reviewed journal from Mary Ann Liebert, Inc., publishers, contains a rich collection of the latest research and reviews focusing on Viral Neuroimmunology and the intricacies of viral brain infection.

Pig immunology comes of age: Killer T cell responses to influenza
Researchers from The Pirbright Institute, University of Bristol, Cardiff University and University of Oxford have generated tools that allow scientists to understand a vital area of the pig immune system which was previously inaccessible.

The skinny on lipid immunology
In a new study published in Science Immunology, researchers from Brigham and Women's Hospital and Monash University in Australia reveal new insights into the basis for T cell receptor (TCR) autoreactivity to self-phospholipids, with implications for autoimmune diseases.

How a few drops of blood led to a breakthrough in immunology
Scientists from the Research Institute of the McGill University Health Centre may have cracked the code to understanding the function of special cells called regulatory T Cells.

Immunology: How ancestry shapes our immune cells
A genetic variant that is particularly prevalent in people of African ancestry confers protection against malaria.

Read More: Immunology News and Immunology Current Events is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to