Exacerbating the replication crisis in science: Replication studies are often unwelcome

April 11, 2017

Researchers in London have investigated 1151 psychology journals and found that just 3% state that they welcome scientists to submit replication studies for publication. In replication studies, scientists try to replicate the findings of previous studies to verify that their results are robust and correct.

Publishing research findings in scientific journals is a career yardstick for many scientists, where the number and perceived impact of their publications are considered important measures of career success. This view is also frequently taken by the funding bodies and tenure committees that provide academic researchers with hotly contested research funding and job security.

Scientific journals provide a platform for scientists to communicate their research. The impact of published papers is often gauged by how many times they are cited (or referenced) by other papers, the idea being that such citations indicate that other scientists are building on the original work.

Scientific journals are also judged on the citations their published manuscripts receive, leading to metrics such as the impact factor. Put simply, the impact factor represents the average number of citations an article in a journal receives in a year. Journals with higher impact factors are often considered to be more desirable places to publish, although the impact factor has sometimes been criticized as an inaccurate measurement of quality.

The tendency for many journals to accept only papers that report positive and original findings has been termed publication bias. In traditional academic publishing, a small minority of submitted studies are accepted for publication, based on their perceived significance or originality, or if they confirm an existing hypothesis. Studies that are thought to provide only a small advance, or those that present "non-impactful" or negative results are frequently rejected.

In fact, many journals explicitly state in their aims or guide to authors that high levels of significance or originality are a prerequisite for publication. Journal editors often enforce extremely high rejection rates, the idea being that selective publishing will increase the journal's impact factor.

However, this traditional publishing model has drawn criticism that it may exclude studies with perceived low impact, but which are valuable for scientific integrity and development. Replications are one such type of study, where researchers attempt to replicate the findings of previous studies to verify that their results are robust and correct.

It is important that scientific experiments are repeatable and produce identical or similar results if repeated. Otherwise, it is difficult to know if experimental results are correct and reveal a real phenomenon, or if they are merely a one-off caused by experimental error or highly specific conditions that are difficult to recreate.

"Science progresses through replication and contradiction. The former builds the body of evidence, the latter determines whether such a body exists," explains Professor Neil Martin, of Regent's University London, lead author on the study, recently published in Frontiers in Psychology.

Recently, a so-called "replication crisis" has been brewing in science. This is occurring across various fields, but the issue has recently come to a head following some high-profile failed replications in psychology. "Researchers have been accused of various creative methodological misdemeanors which may have led to false-positive results being published," says Martin. "We're still uncovering questionable research practices in some well-known historical studies, and I would not be surprised to see many others emerging."

There is a growing awareness and discussion in the wider scientific community that replications are not performed or published enough. This could potentially result in whole areas of scientific research being constructed on foundations of sand.

There are many reasons for the lack of replication studies in various fields of science. The overriding scientific culture is one of innovation, originality and discovery, and scientists may be reluctant to conduct "housekeeping" replication studies, when resources are limited.

Another potential reason is the perception that replications have limited impact, and will therefore be difficult to publish. "Journals have been criticized for not readily accepting replications, but the basis for this criticism is anecdotal," explains Martin.

To begin to quantify this phenomenon, Martin and Richard Clarke, a research student at the London School of Hygiene and Tropical Medicine, investigated how welcoming psychology journals are to publishing replications. "We wanted to investigate whether journals specifically rejected (or did not recommend) the submission of replications. We did this by examining the aims and instructions to authors of 1151 journals in psychology," says Martin.

The team found that only 3% of the psychology journals on their list explicitly stated that they accepted replications. There was no difference between high and low impact journals and no difference between the different branches of psychology.

33% of the journals emphasized the need for scientific originality in submissions, which discourages scientists to submit replications, while 63% neither encouraged nor discouraged replications. The remaining 1% actively discouraged replications.

So how do we make psychology journals more welcoming to replications? "We've suggested that all journals in psychology should state that they accept replications that are positive and negative," explains Martin. "Researchers could also submit two papers for publication when they submit original research: one which reports the original results and one replication which acts as a test of the original findings."

An alternative approach to traditional publishing, in the form of impact-neutral publishers, could also help to increase the number of published replications. Impact-neutral publishers, such as Frontiers, PLOS ONE and many BMC journals, don't make value judgements on perceived impact, and instead assess the scientific validity of a submitted study, when deciding whether to publish it. "If a study tests a hypothesis based on sound reasoning, using sound methodology with appropriate data analysis, those should be the absolute criteria for publishing," says Martin.

Despite not publishing articles on the basis of perceived impact and having a low overall rejection rate (~27%), the impact of Frontiers journals is significant and increasing. For example, Frontiers journals rank in the top 15% of impact factors for all journals, based on Journal Citation Reports published in 2016. The success of newer impact-neutral publishers is challenging the traditional belief that impact-based selectivity is the best way to publish impactful and quality research. Impact-neutral publishing may help researchers to address the replication crisis, by creating a positive atmosphere in which to publish replications.


Related Psychology Articles from Brightsurf:

More than one cognition: A call for change in the field of comparative psychology
In a paper published in the Journal of Intelligence, researchers argue that cognitive studies in comparative psychology often wrongly take an anthropocentric approach, resulting in an over-valuation of human-like abilities and the assumption that cognitive skills cluster in animals as they do in humans.

Psychology research: Antivaxxers actually think differently than other people
As vaccine skepticism has become increasingly widespread, two researchers in the Texas Tech University Department of Psychological Sciences have suggested a possible explanation.

In court, far-reaching psychology tests are unquestioned
Psychological tests are important instruments used in courts to aid legal decisions that profoundly affect people's lives.

Psychology program for refugee children improves wellbeing
A positive psychology program created by researchers at Queen Mary University of London focuses on promoting wellbeing in refugee children.

Psychology can help prevent deadly childhood accidents
Injuries have overtaken infectious disease as the leading cause of death for children worldwide, and psychologists have the research needed to help predict and prevent deadly childhood mishaps, according to a presentation at the annual convention of the American Psychological Association.

Raising the standard for psychology research
Researchers from Stanford University, Arizona State University, and Dartmouth College used Texas Advanced Computing Center supercomputers to apply more rigorous statistical methods to psychological studies of self-regulation.

Psychology: Robot saved, people take the hit
To what extent are people prepared to show consideration for robots?

Researchers help to bridge the gap between psychology and gamification
A multi-disciplinary research team is bridging the gap between psychology and gamification that could significantly impact learning efforts in user experience design, healthcare, and government.

Virtual reality at the service of psychology
Our environment is composed according to certain rules and characteristics which are so obvious to us that we are scarcely aware of them.

Modeling human psychology
A human being's psychological make-up depends on an array of emotional and motivational parameters.

Read More: Psychology News and Psychology Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.