Nav: Home

Does 'publication bias' affect the 'canonization' of facts in science?

December 20, 2016

Researchers model how 'publication bias' does -- and doesn't -- affect the 'canonization' of facts in science

Arguing in a Boston courtroom in 1770, John Adams famously pronounced, "Facts are stubborn things," which cannot be altered by "our wishes, our inclinations or the dictates of our passion."

But facts, however stubborn, must pass through the trials of human perception before being acknowledged -- or "canonized" -- as facts. Given this, some may be forgiven for looking at passionate debates over the color of a dress and wondering if facts are up to the challenge.

Carl Bergstrom believes facts stand a fighting chance, especially if science has their back. A professor of biology at the University of Washington, he has used mathematical modeling to investigate the practice of science, and how science could be shaped by the biases and incentives inherent to human institutions.

"Science is a process of revealing facts through experimentation," said Bergstrom. "But science is also a human endeavor, built on human institutions. Scientists seek status and respond to incentives just like anyone else does. So it is worth asking -- with precise, answerable questions -- if, when and how these incentives affect the practice of science."

In an article published Dec. 20 in the journal eLife, Bergstrom and co-authors present a mathematical model that explores whether "publication bias" -- the tendency of journals to publish mostly positive experimental results -- influences how scientists canonize facts. Their results offer a warning that sharing positive results comes with the risk that a false claim could be canonized as fact. But their findings also offer hope by suggesting that simple changes to publication practices can minimize the risk of false canonization.

These issues have become particularly relevant over the past decade, as prominent articles have questioned the reproducibility of scientific experiments -- a hallmark of validity for discoveries made using the scientific method. But neither Bergstrom nor most of the scientists engaged in these debates are questioning the validity of heavily studied and thoroughly demonstrated scientific truths, such as evolution, anthropogenic climate change or the general safety of vaccination.

"We're modeling the chances of 'false canonization' of facts on lower levels of the scientific method," said Bergstrom. "Evolution happens, and explains the diversity of life. Climate change is real. But we wanted to model if publication bias increases the risk of false canonization at the lowest levels of fact acquisition."

Bergstrom cites a historical example of false canonization in science that lies close to our hearts -- or specifically, below them. Biologists once postulated that bacteria caused stomach ulcers. But in the 1950s, gastroenterologist E.D. Palmer reported evidence that bacteria could not survive in the human gut.

"These findings, supported by the efficacy of antacids, supported the alternative 'chemical theory of ulcer development,' which was subsequently canonized," said Bergstrom. "The problem was that Palmer was using experimental protocols that would not have detected Helicobacter pylori, the bacteria that we know today causes ulcers. It took about a half century to correct this falsehood."

While the idea of false canonization itself may cause dyspepsia, Bergstrom and his team -- lead author Silas Nissen of the Niels Bohr Institute in Denmark and co-authors Kevin Gross of North Carolina State University and UW undergraduate student Tali Magidson -- set out to model the risks of false canonization given the fact that scientists have incentives to publish only their best, positive results. The so-called "negative results," which show no clear, definitive conclusions or simply do not affirm a hypothesis, are much less likely to be published in peer-reviewed journals.

"The net effect of publication bias is that negative results are less likely to be seen, read and processed by scientific peers," said Bergstrom. "Is this misleading the canonization process?"

For their model, Bergstrom's team incorporated variables such as the rates of error in experiments, how much evidence is needed to canonize a claim as fact and the frequency with which negative results are published. Their mathematical model showed that the lower the publication rate is for negative results, the higher the risk for false canonization. And according to their model, one possible solution -- raising the bar for canonization -- didn't help alleviate this risk.

"It turns out that requiring more evidence before canonizing a claim as fact did not help," said Bergstrom. "Instead, our model showed that you need to publish more negative results -- at least more than we probably are now."

Since most negative results live out their obscurity in the pages of laboratory notebooks, it is difficult to quantify the ratio that are published. But clinical trials, which must be registered with the U.S. Food and Drug Administration before they begin, offer a window into how often negative results make it into the peer-reviewed literature. A 2008 analysis of 74 clinical trials for antidepressant drugs showed that scarcely more than 10 percent of negative results were published, compared to over 90 percent for positive results.

"Negative results are probably published at different rates in other fields of science," said Bergstrom. "And new options today, such as self-publishing papers online and the rise of journals that accept some negative results, may affect this. But in general, we need to share negative results more than we are doing today."

Their model also indicated that negative results had the biggest impact as a claim approached the point of canonization. That finding may offer scientists an easy way to prevent false canonization.

"By more closely scrutinizing claims as they achieve broader acceptance, we could identify false claims and keep them from being canonized," said Bergstrom.

To Bergstrom, the model raises valid questions about how scientists choose to publish and share their findings -- both positive and negative. He hopes that their findings pave the way for more detailed exploration of bias in scientific institutions, including the effects of funding sources and the different effects of incentives on different fields of science. But he believes a cultural shift is needed to avoid the risks of publication bias.

"As a community, we tend to say, 'Damn it, this didn't work, and I'm not going to write it up,'" said Bergstrom. "But I'd like scientists to reconsider that tendency, because science is only efficient if we publish a reasonable fraction of our negative findings."
-end-
The research was funded by the Danish National Research Foundation, the University of Washington and the John Templeton Foundation.

For more information, contact Bergstrom at cbergst@u.washington.edu or 206-685-3847.

University of Washington

Related Climate Change Articles:

Fairy-wrens change breeding habits to cope with climate change
Warmer temperatures linked to climate change are having a big impact on the breeding habits of one of Australia's most recognisable bird species, according to researchers at The Australian National University (ANU).
Believing in climate change doesn't mean you are preparing for climate change, study finds
Notre Dame researchers found that although coastal homeowners may perceive a worsening of climate change-related hazards, these attitudes are largely unrelated to a homeowner's expectations of actual home damage.
Older forests resist change -- climate change, that is
Older forests in eastern North America are less vulnerable to climate change than younger forests, particularly for carbon storage, timber production, and biodiversity, new research finds.
Could climate change cause infertility?
A number of plant and animal species could find it increasingly difficult to reproduce if climate change worsens and global temperatures become more extreme -- a stark warning highlighted by new scientific research.
Predicting climate change
Thomas Crowther, ETH Zurich identifies long-disappeared forests available for restoration across the world.
Historical climate important for soil responses to future climate change
Researchers at Lund University in Sweden, in collaboration with colleagues from the University of Amsterdam, examined how 18 years of drought affect the billions of vital bacteria that are hidden in the soil beneath our feet.
Can forests save us from climate change?
Additional climate benefits through sustainable forest management will be modest and local rather than global.
From crystals to climate: 'Gold standard' timeline links flood basalts to climate change
Princeton geologists used tiny zircon crystals found in volcanic ash to rewrite the timeline for the eruptions of the Columbia River flood basalts, a series of massive lava flows that coincided with an ancient global warming period 16 million years ago.
Think pink for a better view of climate change
A new study says pink noise may be the key to separating out natural climate variability from climate change that is influenced by human activity.
Climate taxes on agriculture could lead to more food insecurity than climate change itself
New IIASA-led research has found that a single climate mitigation scheme applied to all sectors, such as a global carbon tax, could have a serious impact on agriculture and result in far more widespread hunger and food insecurity than the direct impacts of climate change.
More Climate Change News and Climate Change Current Events

Best Science Podcasts 2019

We have hand picked the best science podcasts for 2019. Sit back and enjoy new science podcasts updated daily from your favorite science news services and scientists.
Now Playing: TED Radio Hour

Rethinking Anger
Anger is universal and complex: it can be quiet, festering, justified, vengeful, and destructive. This hour, TED speakers explore the many sides of anger, why we need it, and who's allowed to feel it. Guests include psychologists Ryan Martin and Russell Kolts, writer Soraya Chemaly, former talk radio host Lisa Fritsch, and business professor Dan Moshavi.
Now Playing: Science for the People

#538 Nobels and Astrophysics
This week we start with this year's physics Nobel Prize awarded to Jim Peebles, Michel Mayor, and Didier Queloz and finish with a discussion of the Nobel Prizes as a way to award and highlight important science. Are they still relevant? When science breakthroughs are built on the backs of hundreds -- and sometimes thousands -- of people's hard work, how do you pick just three to highlight? Join host Rachelle Saunders and astrophysicist, author, and science communicator Ethan Siegel for their chat about astrophysics and Nobel Prizes.