Nav: Home

Study: Larger sample sizes needed to increase reproducibility in neuroscience studies

June 07, 2018

CHAMPAIGN, Ill. -- Small sample sizes in studies using functional MRI to investigate brain connectivity and function are common in neuroscience, despite years of warnings that such studies likely lack sufficient statistical power. A new analysis reveals that task-based fMRI experiments involving typical sample sizes of about 30 participants are only modestly replicable. This means that independent efforts to repeat the experiments are as likely to challenge as to confirm the original results.

The study, reported in the journal Nature Communications Biology, also finds that task-based fMRI studies with sample sizes of up to 100 also fall short of being perfectly replicable.

Task-based fMRI studies track changes in blood oxygen levels in the brain while study subjects are engaged in cognitive tasks. The technique allows researchers to see which brain regions are recruited to perform specific tasks.

But those in the field of cognitive neuroscience have not agreed on specific standards for the design of task-based fMRI studies - in particular, how many study subjects are needed to ensure reliable findings. The new research aims to address this shortfall, researchers said.

"This study is the largest investigation of the role of sample size in the reproducibility of task-based fMRI methods," said University of Illinois psychology professor Aron Barbey, who conducted the research with Erick Paul, a former postdoctoral fellow at the U. of I.; Benjamin Turner, of Nanyang Technological University in Singapore; and Michael Miller, of the University of California, Santa Barbara.

"We systematically examined the reproducibility of studies at different sample sizes, from as few as 16 to as many as 121 participants, and across many different task-based fMRI approaches," Barbey said. "We found that reproducibility is modest in studies involving the standard number of 30 participants, and that much larger sample sizes may be needed to improve replicability."

The problem is widespread, Paul said. A majority fMRI studies published in cognitive neuroscience journals in 2011-14 involved fewer than 100 participants. The median number is about 30. The problem stems in part from the fact that fMRI studies are expensive.

"Even small studies can cost several tens of thousands of dollars, and funding systems are not generally set up to enable the routine collection of data from more than 100 participants," Paul said.

Other factors also contribute to the challenge. For example, researchers in neuroscience and psychological science may not understand how statistical analyses for fMRI differ from those in behavioral studies, and so may not accurately evaluate the statistical power of their fMRI analyses, Turner said.

"They may confuse the high retest accuracy of fMRI with high replicability," he said. "But that sort of accuracy refers only to retests of the same individuals engaged in the same tasks, not to similarity across separate small groups."

To overcome the problem of expense, Miller suggests that researchers pool their resources to conduct fMRI studies across numerous sites involving greater numbers of participants. This would likely result in more diverse participant pools. It also would advance the work of many scientists, laboratories and institutions at once, yielding more data per participant with potentially greater impact.

"Replicability is the foundation of scientific progress," the authors wrote. "Unfortunately, for a variety of reasons, many scientific fields are gripped by a crisis of irreproducibility. While some of the causes of the crisis are deeply woven into the academic landscape - incentives related to publication, funding and tenure - the most straightforward solution relates to statistical power."

"Our study points to a major problem in the field of cognitive neuroscience, a field in which fMRI studies are a commonly used tool," Barbey said. "But this is a problem that we can solve by increasing the size of the samples we study."
-end-
Barbey is a professor in the Beckman Institute for Advanced Science and Technology and an affiliate of the Carl R. Woese Institute for Genomic Biology at the U. of I. He directs the Center for Brain Plasticity, a partnership between the Beckman Institute and the Interdisciplinary Health Sciences Institute at Illinois.

This research is based upon work supported by the Office of the Director of National Intelligence, Intelligence Advanced Research Projects Activity.

Editor's notes:

To reach Aron Barbey, call 217-244-2551; email barbey@illinois.edu.

The paper "Small sample sizes reduce the replicability of task-based fMRI studies" is available from the U. of I. News Bureau. DOI: 10.1038/s42003-018-0073-z

University of Illinois at Urbana-Champaign

Related Neuroscience Articles:

Researchers rebuild the bridge between neuroscience and artificial intelligence
In an article in the journal Scientific Reports, researchers reveal that they have successfully rebuilt the bridge between experimental neuroscience and advanced artificial intelligence learning algorithms.
The evolution of neuroscience as a research
When the first issue of the JDR was published, the field of neuroscience did not exist but over subsequent decades neuroscience has emerged as a scientific field that has particular relevance to dentistry.
Diabetes-Alzheimer's link explored at Neuroscience 2019
Surprising links exist between diabetes and Alzheimer's disease, and researchers are beginning to unpack the pathology that connects the two.
Organoid research revealed at Neuroscience 2019
Mini-brains, also called organoids, may offer breakthroughs in clinical research by allowing scientists to study human brain cells without a human subject.
The neuroscience of autism: New clues for how condition begins
UNC School of Medicine scientists found that a gene mutation linked to autism normally works to organize the scaffolding of brain cells called radial progenitors necessary for the orderly formation of the brain.
Harnessing reliability for neuroscience research
Neuroscientists are amassing the large-scale datasets needed to study individual differences and identify biomarkers.
Blue Brain solves a century-old neuroscience problem
In a front-cover paper published in Cerebral Cortex, EPFL's Blue Brain Project, a Swiss Brain Research Initiative, explains how the shapes of neurons can be classified using mathematical methods from the field of algebraic topology.
Characterizing pig hippocampus could improve translational neuroscience
Researchers have taken further steps toward developing a superior animal model of neurological conditions such as traumatic brain injury and epilepsy, according to a study of miniature pigs published in eNeuro.
The neuroscience of human vocal pitch
Among primates, humans are uniquely able to consciously control the pitch of their voices, making it possible to hit high notes in singing or stress a word in a sentence to convey meaning.
Study tackles neuroscience claims to have disproved 'free will'
For several decades, some researchers have argued that neuroscience studies prove human actions are driven by external stimuli -- that the brain is reactive and free will is an illusion.
More Neuroscience News and Neuroscience Current Events

Trending Science News

Current Coronavirus (COVID-19) News

Top Science Podcasts

We have hand picked the top science podcasts of 2020.
Now Playing: TED Radio Hour

Listen Again: The Power Of Spaces
How do spaces shape the human experience? In what ways do our rooms, homes, and buildings give us meaning and purpose? This hour, TED speakers explore the power of the spaces we make and inhabit. Guests include architect Michael Murphy, musician David Byrne, artist Es Devlin, and architect Siamak Hariri.
Now Playing: Science for the People

#576 Science Communication in Creative Places
When you think of science communication, you might think of TED talks or museum talks or video talks, or... people giving lectures. It's a lot of people talking. But there's more to sci comm than that. This week host Bethany Brookshire talks to three people who have looked at science communication in places you might not expect it. We'll speak with Mauna Dasari, a graduate student at Notre Dame, about making mammals into a March Madness match. We'll talk with Sarah Garner, director of the Pathologists Assistant Program at Tulane University School of Medicine, who takes pathology instruction out of...
Now Playing: Radiolab

What If?
There's plenty of speculation about what Donald Trump might do in the wake of the election. Would he dispute the results if he loses? Would he simply refuse to leave office, or even try to use the military to maintain control? Last summer, Rosa Brooks got together a team of experts and political operatives from both sides of the aisle to ask a slightly different question. Rather than arguing about whether he'd do those things, they dug into what exactly would happen if he did. Part war game part choose your own adventure, Rosa's Transition Integrity Project doesn't give us any predictions, and it isn't a referendum on Trump. Instead, it's a deeply illuminating stress test on our laws, our institutions, and on the commitment to democracy written into the constitution. This episode was reported by Bethel Habte, with help from Tracie Hunte, and produced by Bethel Habte. Jeremy Bloom provided original music. Support Radiolab by becoming a member today at Radiolab.org/donate.     You can read The Transition Integrity Project's report here.