Nav: Home

Experimental economics: Results you can trust

March 03, 2016

Reproducibility is an important measure of validity in all fields of experimental science. If researcher A publishes a particular scientific result from his laboratory, researcher B should be able to follow the same protocol and achieve the same result in her laboratory. However, in recent years many results in a variety of disciplines have been questioned for their lack of reproducibility. A new study suggests that published results from experimental economics--a field pioneered at Caltech--are better than average when it comes to reproducibility.

The work was published in the March 3 online issue of the journal Science.

"Trying to reproduce previous results is not glamorous or creative, so it is rarely done. But being able to get the same result over and over is part of the definition of what makes knowledge scientific," says Colin Camerer, the Robert Kirby Professor of Behavioral Economics at Caltech and lead author on the paper.

The study was based on a previous method used to assess the replication of psychology experiments. In the earlier technique, called the reproducibility project psychology (RPP), researchers replicated 100 original studies published in three of the top journals in psychology--and found that although 97 percent of the original studies reported so-called "positive findings" (meaning a significant change compared to control conditions), such positive findings were reliably reproduced only 36 percent of the time.

Using this same technique, Camerer and his colleagues reproduced 18 laboratory experimental papers published in two top-tier economics journals between 2011 and 2014. Eleven of the 18--roughly 61 percent--showed a "significant effect in the same direction as in the original study." The researchers also found that the sample size and p-values--a standard measure of statistical confidence--of the original studies were good predictors for the success of replication, meaning they could serve as good indicators for the reliability of results in future experiments.

"Replicability has become a major issue in many sciences over the past few years, with often low replication rates," says paper coauthor Juergen Huber of the University of Innsbruck. "The rate we report for experimental economics is the highest we are aware of for any field."

The authors suggest that there are some methodological research practices in laboratory experimental economics that contribute to the good replication success. "It seems that the culture established in experimental economics--incentivizing subjects, publication of the experimental procedure and instructions, no deception--ensures reliable results. This is very encouraging given that it is a very young discipline," says Michael Kirchler, another coauthor and collaborator from the University of Innsbruck.

"As a journal editor myself, we are always curious whether experimental results will replicate across populations and cultures, and these results from multiple countries are really reassuring," says coauthor Teck-Hua Ho from the National University of Singapore.

Coauthor Magnus Johannesson from the Stockholm School of Economics adds, "It is extremely important to investigate to what extent we can trust published scientific findings and to implement institutions that promote scientific reproducibility."

"For the past half century, Caltech has been a leader in the development of social science experimental methods. It is no surprise that Caltech scholars are part of a group that use replication studies to demonstrate the validity of these methods," says Jean-Laurent Rosenthal, the Rea A. and Lela G. Axline Professor of Business Economics and chair of the Division of the Humanities and Social Sciences at Caltech.

The work was published in a paper titled, "Evaluating Replicability of Laboratory Experiments in Economics." Other coauthors are: Taisuke Imai and Gideon Nave from Caltech; Johan Almenberg from Sveriges Riksbank in Stockholm; Anna Dreber, Eskil Forsell, Adam Altmejd, Emma Heikensten, and Siri Isaksson from the Stockholm School of Economics; Taizan Chan and Hang Wu from the National University of Singapore; Felix Holzmeister and Michael Razen from the University of Innsbruck; and Thomas Pfeiffer from the New Zealand Institute for Advanced Study.

The study was funded by the Austrian Science Fund, the Austrian National Bank, the Behavioral and Neuroeconomics Discovery Fund, the Jan Wallander and Tom Hedelius Foundation, the Knut and Alice Wallenberg Foundation, the Swedish Foundation For Humanities and Social Sciences, and the Sloan Foundation.

California Institute of Technology

Related Economics Articles:

Research group focuses on economics of transportation needs for rural elderly
A multidisciplinary team of researchers is examining economic issues associated with providing transportation for the rural elderly and other socially disadvantaged populations.
Understanding decisions: The power of combining psychology and economics
A new paper published in the Proceedings of the National Academy of Sciences shows how collaborations between psychologists and economists lead to better understanding of such decisions than either discipline can on its own.
Three Harvard experts explain how economics can shape precision medicines
Many public and private efforts focus on research in precision medicine.
Denmark to honor late Nobel Prize winner Dale T. Mortensen
Three years after the passing of Nobel Prize winner Dale T.
Economics made simple with physics models
Both physical and economic phenomena may possess universal features that could be uncovered using the tools of physics.
Economics of forest biomass raise hurdles for rural development
The use of residual forest biomass for rural development faces significant economic hurdles that make it unlikely to be a source of jobs in the near future, according to an analysis by economists.
'Super yeast' has the power to improve economics of biofuels
Scientists at the University of Wisconsin-Madison and the Great Lakes Bioenergy Research Center have found a way to nearly double the efficiency with which a commonly used industrial yeast strain converts plant sugars to biofuel.
'The Nobel Prize in Economics' and society
In 1968, the scientific status of economics was strengthened by the creation of the 'Nobel Prize in Economic Sciences.' After financial crises, low economic growth and increasing social tension, many are questioning this scientific status.
Tapping into behavioral economics to boost clinical trial participation
Behavioral economics may offer a powerful tool for improving patient enrollment in clinical trials, argue Eric VanEpps, Kevin Volpp, and Scott Halpern in this Focus.
Economics study finds volume discounts don't increase profitability for video game
A study by University of Chicago economists finds discounts tied to buying large quantities of virtual goods have little impact on profitability and do not increase the number of customers making purchases.

Related Economics Reading:

Best Science Podcasts 2019

We have hand picked the best science podcasts for 2019. Sit back and enjoy new science podcasts updated daily from your favorite science news services and scientists.
Now Playing: TED Radio Hour

Moving Forward
When the life you've built slips out of your grasp, you're often told it's best to move on. But is that true? Instead of forgetting the past, TED speakers describe how we can move forward with it. Guests include writers Nora McInerny and Suleika Jaouad, and human rights advocate Lindy Lou Isonhood.
Now Playing: Science for the People

#527 Honey I CRISPR'd the Kids
This week we're coming to you from Awesome Con in Washington, D.C. There, host Bethany Brookshire led a panel of three amazing guests to talk about the promise and perils of CRISPR, and what happens now that CRISPR babies have (maybe?) been born. Featuring science writer Tina Saey, molecular biologist Anne Simon, and bioethicist Alan Regenberg. A Nobel Prize winner argues banning CRISPR babies won’t work Geneticists push for a 5-year global ban on gene-edited babies A CRISPR spin-off causes unintended typos in DNA News of the first gene-edited babies ignited a firestorm The researcher who created CRISPR twins defends...