Nav: Home

Study: Larger sample sizes needed to increase reproducibility in neuroscience studies

June 07, 2018

CHAMPAIGN, Ill. -- Small sample sizes in studies using functional MRI to investigate brain connectivity and function are common in neuroscience, despite years of warnings that such studies likely lack sufficient statistical power. A new analysis reveals that task-based fMRI experiments involving typical sample sizes of about 30 participants are only modestly replicable. This means that independent efforts to repeat the experiments are as likely to challenge as to confirm the original results.

The study, reported in the journal Nature Communications Biology, also finds that task-based fMRI studies with sample sizes of up to 100 also fall short of being perfectly replicable.

Task-based fMRI studies track changes in blood oxygen levels in the brain while study subjects are engaged in cognitive tasks. The technique allows researchers to see which brain regions are recruited to perform specific tasks.

But those in the field of cognitive neuroscience have not agreed on specific standards for the design of task-based fMRI studies - in particular, how many study subjects are needed to ensure reliable findings. The new research aims to address this shortfall, researchers said.

"This study is the largest investigation of the role of sample size in the reproducibility of task-based fMRI methods," said University of Illinois psychology professor Aron Barbey, who conducted the research with Erick Paul, a former postdoctoral fellow at the U. of I.; Benjamin Turner, of Nanyang Technological University in Singapore; and Michael Miller, of the University of California, Santa Barbara.

"We systematically examined the reproducibility of studies at different sample sizes, from as few as 16 to as many as 121 participants, and across many different task-based fMRI approaches," Barbey said. "We found that reproducibility is modest in studies involving the standard number of 30 participants, and that much larger sample sizes may be needed to improve replicability."

The problem is widespread, Paul said. A majority fMRI studies published in cognitive neuroscience journals in 2011-14 involved fewer than 100 participants. The median number is about 30. The problem stems in part from the fact that fMRI studies are expensive.

"Even small studies can cost several tens of thousands of dollars, and funding systems are not generally set up to enable the routine collection of data from more than 100 participants," Paul said.

Other factors also contribute to the challenge. For example, researchers in neuroscience and psychological science may not understand how statistical analyses for fMRI differ from those in behavioral studies, and so may not accurately evaluate the statistical power of their fMRI analyses, Turner said.

"They may confuse the high retest accuracy of fMRI with high replicability," he said. "But that sort of accuracy refers only to retests of the same individuals engaged in the same tasks, not to similarity across separate small groups."

To overcome the problem of expense, Miller suggests that researchers pool their resources to conduct fMRI studies across numerous sites involving greater numbers of participants. This would likely result in more diverse participant pools. It also would advance the work of many scientists, laboratories and institutions at once, yielding more data per participant with potentially greater impact.

"Replicability is the foundation of scientific progress," the authors wrote. "Unfortunately, for a variety of reasons, many scientific fields are gripped by a crisis of irreproducibility. While some of the causes of the crisis are deeply woven into the academic landscape - incentives related to publication, funding and tenure - the most straightforward solution relates to statistical power."

"Our study points to a major problem in the field of cognitive neuroscience, a field in which fMRI studies are a commonly used tool," Barbey said. "But this is a problem that we can solve by increasing the size of the samples we study."
-end-
Barbey is a professor in the Beckman Institute for Advanced Science and Technology and an affiliate of the Carl R. Woese Institute for Genomic Biology at the U. of I. He directs the Center for Brain Plasticity, a partnership between the Beckman Institute and the Interdisciplinary Health Sciences Institute at Illinois.

This research is based upon work supported by the Office of the Director of National Intelligence, Intelligence Advanced Research Projects Activity.

Editor's notes:

To reach Aron Barbey, call 217-244-2551; email barbey@illinois.edu.

The paper "Small sample sizes reduce the replicability of task-based fMRI studies" is available from the U. of I. News Bureau. DOI: 10.1038/s42003-018-0073-z

University of Illinois at Urbana-Champaign

Related Neuroscience Articles:

Three ways neuroscience can advance the concussion debate
While concussion awareness has improved over the past decade, understanding the nuances of these sports injuries, their severity, symptoms, and treatment, is still a work in progress.
Study shows rapid growth in neuroscience research
A study of the impact and research topics of neuroscience papers from 2006-2015 has shown that the number of neuroscience papers and highly-productive core neuroscience journals has grown, while psychology and behavioral sciences have become more popular research areas.
CNS 2017: Big Ideas in Cognitive Neuroscience
Press registration is now open for the Cognitive Neuroscience Society annual conference, March 25-28, 2017, in San Francisco, CA, at the Hyatt Regency.
Jounrnal of Neuroscience: Highlights from the November 9 issue
Check out these newsworthy symposia featured in the Nov. 9, 2016, issue of the Journal of Neuroscience.
Awards event to recognize heroes of green and open neuroscience
On Monday, November 14, at 6:30 p.m., the Physicians Committee for Responsible Medicine and the founders of the Green Neuroscience Laboratory will recognize individuals and organizations that help forward the principles and culture of Green and Open Neuroscience.
Robotic cleaning technique could automate neuroscience research
For scientists listening in on the faint whispers of brain neurons, a first-ever robotic technique for cleaning the tiny devices that record the signals could facilitate a new level of automation in neuroscience research.
29th ECNP Congress for Applied and Translational Neuroscience
Europe's largest meeting in applied and translational neuroscience, the 29th ECNP Congress of the European College of Neuropsychopharmacology (ECNP) will take place at the Austria Center Vienna from Sept.
Neuroscience 2016 media registration now open
San Diego becomes the epicenter of neuroscience in November as 30,000 researchers, clinicians, and advocates from around the world gather November 12-16 to explore and share the latest developments in brain research.
A vision for revamping neuroscience education
The expanding scope and growing number of tools used for neuroscience is moving beyond what is taught in traditional graduate programs, say leaders in American neuroscience education, funding, and policy.
Is educational neuroscience a waste of money?
Educational neuroscience has little to offer schools or children's education, according to new research from the University of Bristol, UK.

Related Neuroscience Reading:

Best Science Podcasts 2019

We have hand picked the best science podcasts for 2019. Sit back and enjoy new science podcasts updated daily from your favorite science news services and scientists.
Now Playing: TED Radio Hour

Bias And Perception
How does bias distort our thinking, our listening, our beliefs... and even our search results? How can we fight it? This hour, TED speakers explore ideas about the unconscious biases that shape us. Guests include writer and broadcaster Yassmin Abdel-Magied, climatologist J. Marshall Shepherd, journalist Andreas Ekström, and experimental psychologist Tony Salvador.
Now Playing: Science for the People

#514 Arctic Energy (Rebroadcast)
This week we're looking at how alternative energy works in the arctic. We speak to Louie Azzolini and Linda Todd from the Arctic Energy Alliance, a non-profit helping communities reduce their energy usage and transition to more affordable and sustainable forms of energy. And the lessons they're learning along the way can help those of us further south.