Nav: Home

Tel Aviv University-led study finds high variability is result of complex data workflows

May 20, 2020

A new Tel Aviv University-led study published on May 20 in Nature offers new evidence that the complexity of contemporary analytical methods in science contributes to the variability of research outcomes.

Previous studies in the fields of psychology, cancer biology and behavioral economics revealed many failures in the attempt to reproduce methodologies and mirror results. The TAU researchers used an approach known as "Many Analysts," in which many researchers analyzed the same dataset to test variability in the analysis outcomes, explains study co-lead author Dr. Tom Schonberg of the Department of Neurobiology at TAU's George S. Wise Faculty of Life Sciences and TAU's Sagol School of Neuroscience.

"The variability in outcomes demonstrated in this study is an inherent part of the complex process of obtaining scientific results, and we must understand it in order to know how to tackle it," he adds. "Science is conducted by humans, and there is no way to escape variability. But we must acknowledge this in order to self-correct and attain the most reliable answers."

The Neuroimaging Analysis, Replication and Prediction Study (NARPS) was also led by Dr. Schonberg's former PhD student Dr. Rotem Botvinik-Nezer, today a post-doctoral researcher at Dartmouth College, together with co-investigators Prof. Russel Poldrack of Stanford University and Prof. Thomas Nichols of Oxford University.

Overall, 180 researchers from 70 teams of scientists around the world analyzed the same brain imaging dataset of 108 subjects. These subjects participated in a task that tested their decision-making regarding gambles of potential gains and losses. Each group chose a distinct analysis method, and the different methods led to different conclusions.

"Science is being often criticized," adds Dr. Schonberg. "But it is not a belief like a religion, as some have argued. It has rules and a method -- the scientific method. We are constantly trying to improve this method in a process of constant self-questioning. We believe our study takes this process forward."

The research teams were presented with the same data -- fMRI scans of subjects performing a value-based decision-making task -- and told to test the same nine different set hypotheses.

The large neuroimaging dataset had been collected over the course of a year at the Alfredo Federico Strauss Center for Computational Neuroimaging at TAU by Roni Iwanir, a former Sagol School MSc student from Schonberg's lab. While the study participants engaged in the monetary decision-making task, fMRI scans were used to test whether the activity of specific brain regions involved in value processing changed in relation to the amount of money won or lost on a gamble. Some 70 international teams independently analyzed these datasets over the course of three months.

"The processing you have to go through from raw data to a result with fMRI is really complicated," adds Prof. Poldrack. "There are a lot of choices you have to make at each place in the analysis workflow."

Each team of researchers arrived at their own final conclusions regarding the data, with results varying significantly across the teams in five out of nine hypotheses.

"Our new study demonstrated high analytical flexibility as occurs 'in the wild'," says Dr. Schonberg. "The participating researchers modeled the hypotheses differently and used different software for the analysis. They also used different techniques and definitions in different aspects of the analysis."

Another part of the study involved experts from the field as well as researchers from the analysis teams trading with other researchers in what are called "prediction markets" on what they thought would be the outcomes of the research. This part of the study was run by economists and behavioral finance experts, who provided the initial idea for the the study, and revealed marked over-optimism about the ability to replicate previous findings, even by researchers who analyzed the data themselves.

"While the final reported outcomes varied substantially, earlier stage analysis results actually showed a consensus among most research teams," says Dr. Schonberg. "This is very encouraging and was in fact a somewhat surprising result. Despite the large variability in final reported results, the underlying analysis was similar, meaning we need to find methods to express this convergence.

"For example, the study suggests that researchers could perform and report multiple analyses with the same data, to find the results to which different reliable methods converge."

Dr. Schonberg believes the findings can help scientists advance their methodology and improve the quality of their analyses in the future.

"I would want our findings to be used to take science forward -- toward an even further sharing of all study related information, of transparency of methods, analysis codes and data," says Dr. Schonberg. That is the only way everyone can test and "play" with results to see what holds. We have seen the importance and great need in sharing data in the recent COVID-19 pandemic in order to understand the optimal course of action.

"Naturally, the novelty of discoveries matters a great deal to scientists. But just as important is the rigor of our methodology," concludes Dr. Schonberg. "Our study reflects the ambition of a vast community of scientists to spend thousands of hours to improve our methodologies in order to get the conclusions right and to reach reliable results."
-end-
American Friends of Tel Aviv University supports Israel's most influential, comprehensive and sought-after center of higher learning, Tel Aviv University (TAU). TAU is recognized and celebrated internationally for creating an innovative, entrepreneurial culture on campus that generates inventions, startups and economic development in Israel. TAU is ranked ninth in the world, and first in Israel, for producing start-up founders of billion-dollar companies, an achievement that surpassed several Ivy League universities. To date, 2,500 US patents have been filed by Tel Aviv University researchers -- ranking TAU #1 in Israel, #10 outside of the US and #66 in the world.

American Friends of Tel Aviv University

Related Data Articles:

Ups and downs in COVID-19 data may be caused by data reporting practices
As data accumulates on COVID-19 cases and deaths, researchers have observed patterns of peaks and valleys that repeat on a near-weekly basis.
Data centers use less energy than you think
Using the most detailed model to date of global data center energy use, researchers found that massive efficiency gains by data centers have kept energy use roughly flat over the past decade.
Storing data in music
Researchers at ETH Zurich have developed a technique for embedding data in music and transmitting it to a smartphone.
Life data economics: calling for new models to assess the value of human data
After the collapse of the blockchain bubble a number of research organisations are developing platforms to enable individual ownership of life data and establish the data valuation and pricing models.
Geoscience data group urges all scientific disciplines to make data open and accessible
Institutions, science funders, data repositories, publishers, researchers and scientific societies from all scientific disciplines must work together to ensure all scientific data are easy to find, access and use, according to a new commentary in Nature by members of the Enabling FAIR Data Steering Committee.
Democratizing data science
MIT researchers are hoping to advance the democratization of data science with a new tool for nonstatisticians that automatically generates models for analyzing raw data.
Getting the most out of atmospheric data analysis
An international team including researchers from Kanazawa University used a new approach to analyze an atmospheric data set spanning 18 years for the investigation of new-particle formation.
Ecologists ask: Should we be more transparent with data?
In a new Ecological Applications article, authors Stephen M. Powers and Stephanie E.
Should you share data of threatened species?
Scientists and conservationists have continually called for location data to be turned off in wildlife photos and publications to help preserve species but new research suggests there could be more to be gained by sharing a rare find, rather than obscuring it, in certain circumstances.
Futuristic data storage
The development of high-density data storage devices requires the highest possible density of elements in an array made up of individual nanomagnets.
More Data News and Data Current Events

Trending Science News

Current Coronavirus (COVID-19) News

Top Science Podcasts

We have hand picked the top science podcasts of 2020.
Now Playing: TED Radio Hour

Warped Reality
False information on the internet makes it harder and harder to know what's true, and the consequences have been devastating. This hour, TED speakers explore ideas around technology and deception. Guests include law professor Danielle Citron, journalist Andrew Marantz, and computer scientist Joy Buolamwini.
Now Playing: Science for the People

#576 Science Communication in Creative Places
When you think of science communication, you might think of TED talks or museum talks or video talks, or... people giving lectures. It's a lot of people talking. But there's more to sci comm than that. This week host Bethany Brookshire talks to three people who have looked at science communication in places you might not expect it. We'll speak with Mauna Dasari, a graduate student at Notre Dame, about making mammals into a March Madness match. We'll talk with Sarah Garner, director of the Pathologists Assistant Program at Tulane University School of Medicine, who takes pathology instruction out of...
Now Playing: Radiolab

What If?
There's plenty of speculation about what Donald Trump might do in the wake of the election. Would he dispute the results if he loses? Would he simply refuse to leave office, or even try to use the military to maintain control? Last summer, Rosa Brooks got together a team of experts and political operatives from both sides of the aisle to ask a slightly different question. Rather than arguing about whether he'd do those things, they dug into what exactly would happen if he did. Part war game part choose your own adventure, Rosa's Transition Integrity Project doesn't give us any predictions, and it isn't a referendum on Trump. Instead, it's a deeply illuminating stress test on our laws, our institutions, and on the commitment to democracy written into the constitution. This episode was reported by Bethel Habte, with help from Tracie Hunte, and produced by Bethel Habte. Jeremy Bloom provided original music. Support Radiolab by becoming a member today at Radiolab.org/donate.     You can read The Transition Integrity Project's report here.