Nav: Home

Experimental economics: Results you can trust

March 03, 2016

Reproducibility is an important measure of validity in all fields of experimental science. If researcher A publishes a particular scientific result from his laboratory, researcher B should be able to follow the same protocol and achieve the same result in her laboratory. However, in recent years many results in a variety of disciplines have been questioned for their lack of reproducibility. A new study suggests that published results from experimental economics--a field pioneered at Caltech--are better than average when it comes to reproducibility.

The work was published in the March 3 online issue of the journal Science.

"Trying to reproduce previous results is not glamorous or creative, so it is rarely done. But being able to get the same result over and over is part of the definition of what makes knowledge scientific," says Colin Camerer, the Robert Kirby Professor of Behavioral Economics at Caltech and lead author on the paper.

The study was based on a previous method used to assess the replication of psychology experiments. In the earlier technique, called the reproducibility project psychology (RPP), researchers replicated 100 original studies published in three of the top journals in psychology--and found that although 97 percent of the original studies reported so-called "positive findings" (meaning a significant change compared to control conditions), such positive findings were reliably reproduced only 36 percent of the time.

Using this same technique, Camerer and his colleagues reproduced 18 laboratory experimental papers published in two top-tier economics journals between 2011 and 2014. Eleven of the 18--roughly 61 percent--showed a "significant effect in the same direction as in the original study." The researchers also found that the sample size and p-values--a standard measure of statistical confidence--of the original studies were good predictors for the success of replication, meaning they could serve as good indicators for the reliability of results in future experiments.

"Replicability has become a major issue in many sciences over the past few years, with often low replication rates," says paper coauthor Juergen Huber of the University of Innsbruck. "The rate we report for experimental economics is the highest we are aware of for any field."

The authors suggest that there are some methodological research practices in laboratory experimental economics that contribute to the good replication success. "It seems that the culture established in experimental economics--incentivizing subjects, publication of the experimental procedure and instructions, no deception--ensures reliable results. This is very encouraging given that it is a very young discipline," says Michael Kirchler, another coauthor and collaborator from the University of Innsbruck.

"As a journal editor myself, we are always curious whether experimental results will replicate across populations and cultures, and these results from multiple countries are really reassuring," says coauthor Teck-Hua Ho from the National University of Singapore.

Coauthor Magnus Johannesson from the Stockholm School of Economics adds, "It is extremely important to investigate to what extent we can trust published scientific findings and to implement institutions that promote scientific reproducibility."

"For the past half century, Caltech has been a leader in the development of social science experimental methods. It is no surprise that Caltech scholars are part of a group that use replication studies to demonstrate the validity of these methods," says Jean-Laurent Rosenthal, the Rea A. and Lela G. Axline Professor of Business Economics and chair of the Division of the Humanities and Social Sciences at Caltech.

The work was published in a paper titled, "Evaluating Replicability of Laboratory Experiments in Economics." Other coauthors are: Taisuke Imai and Gideon Nave from Caltech; Johan Almenberg from Sveriges Riksbank in Stockholm; Anna Dreber, Eskil Forsell, Adam Altmejd, Emma Heikensten, and Siri Isaksson from the Stockholm School of Economics; Taizan Chan and Hang Wu from the National University of Singapore; Felix Holzmeister and Michael Razen from the University of Innsbruck; and Thomas Pfeiffer from the New Zealand Institute for Advanced Study.

The study was funded by the Austrian Science Fund, the Austrian National Bank, the Behavioral and Neuroeconomics Discovery Fund, the Jan Wallander and Tom Hedelius Foundation, the Knut and Alice Wallenberg Foundation, the Swedish Foundation For Humanities and Social Sciences, and the Sloan Foundation.

California Institute of Technology

Related Economics Articles:

Life data economics: calling for new models to assess the value of human data
After the collapse of the blockchain bubble a number of research organisations are developing platforms to enable individual ownership of life data and establish the data valuation and pricing models.
SCAI and ACVP release consensus statement on cardiovascular catheterization laboratory economics
A newly released expert consensus statement provides recommendations for optimizing the financial operations of the cardiovascular catheterization laboratory (CCL) while providing cutting-edge patient care.
Shocking economics
Understanding economies in times of crises? Modern macroeconomics failed so far.
When does one of the central ideas in economics work?
Many situations in economics are complicated and competitive; this research raises the question of whether many theories in economics may suffer from the very fundamental problem that the key behavioral assumption of equilibrium is wrong.
From property damage to lost production: How natural disasters impact economics
When a natural disaster strikes, major disaster databases tend to compile information about losses such as damages to property or cost of repairs, but other economic impacts after the disaster are often overlooked--such as how a company's lost ability to produce products may affect the entire supply-chain within the affected region and in other regions.
More Economics News and Economics Current Events

Best Science Podcasts 2019

We have hand picked the best science podcasts for 2019. Sit back and enjoy new science podcasts updated daily from your favorite science news services and scientists.
Now Playing: TED Radio Hour

Erasing The Stigma
Many of us either cope with mental illness or know someone who does. But we still have a hard time talking about it. This hour, TED speakers explore ways to push past — and even erase — the stigma. Guests include musician and comedian Jordan Raskopoulos, neuroscientist and psychiatrist Thomas Insel, psychiatrist Dixon Chibanda, anxiety and depression researcher Olivia Remes, and entrepreneur Sangu Delle.
Now Playing: Science for the People

#537 Science Journalism, Hold the Hype
Everyone's seen a piece of science getting over-exaggerated in the media. Most people would be quick to blame journalists and big media for getting in wrong. In many cases, you'd be right. But there's other sources of hype in science journalism. and one of them can be found in the humble, and little-known press release. We're talking with Chris Chambers about doing science about science journalism, and where the hype creeps in. Related links: The association between exaggeration in health related science news and academic press releases: retrospective observational study Claims of causality in health news: a randomised trial This...