Scientists unknowingly tweak experiments: ANU media release

March 17, 2015

A new study has found some scientists are unknowingly tweaking experiments and analysis methods to increase their chances of getting results that are easily published.

The study conducted by scientists at The Australian National University (ANU) is the most comprehensive investigation into a type of publication bias called p-hacking.

P-hacking happens when researchers either consciously or unconsciously analyse their data multiple times or in multiple ways until they get a desired result. If p-hacking is common, the exaggerated results could lead to misleading conclusions, even when evidence comes from multiple studies.

"We found evidence that p-hacking is happening throughout the life sciences," said lead author Dr Megan Head from the ANU Research School of Biology.

The study used text mining to extract p-values - a number that indicates how likely it is that a result occurs by chance - from more than 100,000 research papers published around the world, spanning many scientific disciplines, including medicine, biology and psychology.

"Many researchers are not aware that certain methods could make some results seem more important than they are. They are just genuinely excited about finding something new and interesting," Dr Head said.

"I think that pressure to publish is one factor driving this bias. As scientists we are judged by how many publications we have and the quality of the scientific journals they go in.

"Journals, especially the top journals, are more likely to publish experiments with new, interesting results, creating incentive to produce results on demand."

Dr Head said the study found a high number of p-values that were only just over the traditional threshold that most scientists call statistically significant.

"This suggests that some scientists adjust their experimental design, datasets or statistical methods until they get a result that crosses the significance threshold," she said.

"They might look at their results before an experiment is finished, or explore their data with lots of different statistical methods, without realising that this can lead to bias."

The concern with p-hacking is that it could get in the way of forming accurate scientific conclusions, even when scientists review the evidence by combining results from multiple studies.

For example, if some studies show a particular drug is effective in treating hypertension, but other studies find it is not effective, scientists would analyse all the data to reach an overall conclusion. But if enough results have been p-hacked, the drug would look more effective than it is.

"We looked at the likelihood of this bias occurring in our own specialty, evolutionary biology, and although p-hacking was happening it wasn't common enough to drastically alter general conclusions that could be made from the research," she said.

"But greater awareness of p-hacking and its dangers is important because the implications of p-hacking may be different depending on the question you are asking."
-end-


Australian National University

Related Biology Articles from Brightsurf:

Experimental Biology press materials available now
Though the Experimental Biology (EB) 2020 meeting was canceled in response to the COVID-19 outbreak, EB research abstracts are being published in the April 2020 issue of The FASEB Journal.

Structural biology: Special delivery
Bulky globular proteins require specialized transport systems for insertion into membranes.

Cell biology: All in a flash!
Scientists of Ludwig-Maximilians-Universitaet (LMU) in Munich have developed a tool to eliminate essential proteins from cells with a flash of light.

A biology boost
Assistance during the first years of a biology major leads to higher retention of first-generation students.

Cell biology: Compartments and complexity
Ludwig-Maximilians-Universitaet (LMU) in Munich biologists have taken a closer look at the subcellular distribution of proteins and metabolic intermediates in a model plant.

Cell biology: The complexity of division by two
Ludwig-Maximilians-Universitaet (LMU) in Munich researchers have identified a novel protein that plays a crucial role in the formation of the mitotic spindle, which is essential for correct segregation of a full set of chromosomes to each daughter cell during cell division.

Cell biology: Dynamics of microtubules
Filamentous polymers called microtubules play vital roles in chromosome segregation and molecular transport.

The biology of color
Scientists are on a threshold of a new era of color science with regard to animals, according to a comprehensive review of the field by a multidisciplinary team of researchers led by professor Tim Caro at UC Davis.

Kinky biology
How and why proteins fold is a problem that has implications for protein design and therapeutics.

A new tool to decipher evolutionary biology
A new bioinformatics tool to compare genome data has been developed by teams from the Max F.

Read More: Biology News and Biology Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.