Nav: Home

Does 'publication bias' affect the 'canonization' of facts in science?

December 20, 2016

Researchers model how 'publication bias' does -- and doesn't -- affect the 'canonization' of facts in science

Arguing in a Boston courtroom in 1770, John Adams famously pronounced, "Facts are stubborn things," which cannot be altered by "our wishes, our inclinations or the dictates of our passion."

But facts, however stubborn, must pass through the trials of human perception before being acknowledged -- or "canonized" -- as facts. Given this, some may be forgiven for looking at passionate debates over the color of a dress and wondering if facts are up to the challenge.

Carl Bergstrom believes facts stand a fighting chance, especially if science has their back. A professor of biology at the University of Washington, he has used mathematical modeling to investigate the practice of science, and how science could be shaped by the biases and incentives inherent to human institutions.

"Science is a process of revealing facts through experimentation," said Bergstrom. "But science is also a human endeavor, built on human institutions. Scientists seek status and respond to incentives just like anyone else does. So it is worth asking -- with precise, answerable questions -- if, when and how these incentives affect the practice of science."

In an article published Dec. 20 in the journal eLife, Bergstrom and co-authors present a mathematical model that explores whether "publication bias" -- the tendency of journals to publish mostly positive experimental results -- influences how scientists canonize facts. Their results offer a warning that sharing positive results comes with the risk that a false claim could be canonized as fact. But their findings also offer hope by suggesting that simple changes to publication practices can minimize the risk of false canonization.

These issues have become particularly relevant over the past decade, as prominent articles have questioned the reproducibility of scientific experiments -- a hallmark of validity for discoveries made using the scientific method. But neither Bergstrom nor most of the scientists engaged in these debates are questioning the validity of heavily studied and thoroughly demonstrated scientific truths, such as evolution, anthropogenic climate change or the general safety of vaccination.

"We're modeling the chances of 'false canonization' of facts on lower levels of the scientific method," said Bergstrom. "Evolution happens, and explains the diversity of life. Climate change is real. But we wanted to model if publication bias increases the risk of false canonization at the lowest levels of fact acquisition."

Bergstrom cites a historical example of false canonization in science that lies close to our hearts -- or specifically, below them. Biologists once postulated that bacteria caused stomach ulcers. But in the 1950s, gastroenterologist E.D. Palmer reported evidence that bacteria could not survive in the human gut.

"These findings, supported by the efficacy of antacids, supported the alternative 'chemical theory of ulcer development,' which was subsequently canonized," said Bergstrom. "The problem was that Palmer was using experimental protocols that would not have detected Helicobacter pylori, the bacteria that we know today causes ulcers. It took about a half century to correct this falsehood."

While the idea of false canonization itself may cause dyspepsia, Bergstrom and his team -- lead author Silas Nissen of the Niels Bohr Institute in Denmark and co-authors Kevin Gross of North Carolina State University and UW undergraduate student Tali Magidson -- set out to model the risks of false canonization given the fact that scientists have incentives to publish only their best, positive results. The so-called "negative results," which show no clear, definitive conclusions or simply do not affirm a hypothesis, are much less likely to be published in peer-reviewed journals.

"The net effect of publication bias is that negative results are less likely to be seen, read and processed by scientific peers," said Bergstrom. "Is this misleading the canonization process?"

For their model, Bergstrom's team incorporated variables such as the rates of error in experiments, how much evidence is needed to canonize a claim as fact and the frequency with which negative results are published. Their mathematical model showed that the lower the publication rate is for negative results, the higher the risk for false canonization. And according to their model, one possible solution -- raising the bar for canonization -- didn't help alleviate this risk.

"It turns out that requiring more evidence before canonizing a claim as fact did not help," said Bergstrom. "Instead, our model showed that you need to publish more negative results -- at least more than we probably are now."

Since most negative results live out their obscurity in the pages of laboratory notebooks, it is difficult to quantify the ratio that are published. But clinical trials, which must be registered with the U.S. Food and Drug Administration before they begin, offer a window into how often negative results make it into the peer-reviewed literature. A 2008 analysis of 74 clinical trials for antidepressant drugs showed that scarcely more than 10 percent of negative results were published, compared to over 90 percent for positive results.

"Negative results are probably published at different rates in other fields of science," said Bergstrom. "And new options today, such as self-publishing papers online and the rise of journals that accept some negative results, may affect this. But in general, we need to share negative results more than we are doing today."

Their model also indicated that negative results had the biggest impact as a claim approached the point of canonization. That finding may offer scientists an easy way to prevent false canonization.

"By more closely scrutinizing claims as they achieve broader acceptance, we could identify false claims and keep them from being canonized," said Bergstrom.

To Bergstrom, the model raises valid questions about how scientists choose to publish and share their findings -- both positive and negative. He hopes that their findings pave the way for more detailed exploration of bias in scientific institutions, including the effects of funding sources and the different effects of incentives on different fields of science. But he believes a cultural shift is needed to avoid the risks of publication bias.

"As a community, we tend to say, 'Damn it, this didn't work, and I'm not going to write it up,'" said Bergstrom. "But I'd like scientists to reconsider that tendency, because science is only efficient if we publish a reasonable fraction of our negative findings."
-end-
The research was funded by the Danish National Research Foundation, the University of Washington and the John Templeton Foundation.

For more information, contact Bergstrom at cbergst@u.washington.edu or 206-685-3847.

University of Washington

Related Climate Change Articles:

The black forest and climate change
Silver and Douglas firs could replace Norway spruce in the long run due to their greater resistance to droughts.
For some US counties, climate change will be particularly costly
A highly granular assessment of the impacts of climate change on the US economy suggests that each 1°Celsius increase in temperature will cost 1.2 percent of the country's gross domestic product, on average.
Climate change label leads to climate science acceptance
A new Cornell University study finds that labels matter when it comes to acceptance of climate science.
Was that climate change?
A new four-step 'framework' aims to test the contribution of climate change to record-setting extreme weather events.
It's more than just climate change
Accurately modeling climate change and interactive human factors -- including inequality, consumption, and population -- is essential for the effective science-based policies and measures needed to benefit and sustain current and future generations.
Climate change scientists should think more about sex
Climate change can have a different impact on male and female fish, shellfish and other marine animals, with widespread implications for the future of marine life and the production of seafood.
Climate change prompts Alaska fish to change breeding behavior
A new University of Washington study finds that one of Alaska's most abundant freshwater fish species is altering its breeding patterns in response to climate change, which could impact the ecology of northern lakes that already acutely feel the effects of a changing climate.
Uncertainties related to climate engineering limit its use in curbing climate change
Climate engineering refers to the systematic, large-scale modification of the environment using various climate intervention techniques.
Public holds polarized views about climate change and trust in climate scientists
There are gaping divisions in Americans' views across every dimension of the climate debate, including causes and cures for climate change and trust in climate scientists and their research, according to a new Pew Research Center survey.
The psychology behind climate change denial
In a new thesis in psychology, Kirsti Jylhä at Uppsala University has studied the psychology behind climate change denial.

Related Climate Change Reading:

Best Science Podcasts 2019

We have hand picked the best science podcasts for 2019. Sit back and enjoy new science podcasts updated daily from your favorite science news services and scientists.
Now Playing: TED Radio Hour

Climate Crisis
There's no greater threat to humanity than climate change. What can we do to stop the worst consequences? This hour, TED speakers explore how we can save our planet and whether we can do it in time. Guests include climate activist Greta Thunberg, chemical engineer Jennifer Wilcox, research scientist Sean Davis, food innovator Bruce Friedrich, and psychologist Per Espen Stoknes.
Now Playing: Science for the People

#527 Honey I CRISPR'd the Kids
This week we're coming to you from Awesome Con in Washington, D.C. There, host Bethany Brookshire led a panel of three amazing guests to talk about the promise and perils of CRISPR, and what happens now that CRISPR babies have (maybe?) been born. Featuring science writer Tina Saey, molecular biologist Anne Simon, and bioethicist Alan Regenberg. A Nobel Prize winner argues banning CRISPR babies won’t work Geneticists push for a 5-year global ban on gene-edited babies A CRISPR spin-off causes unintended typos in DNA News of the first gene-edited babies ignited a firestorm The researcher who created CRISPR twins defends...