Experimental economics: Results you can trust

March 03, 2016

Reproducibility is an important measure of validity in all fields of experimental science. If researcher A publishes a particular scientific result from his laboratory, researcher B should be able to follow the same protocol and achieve the same result in her laboratory. However, in recent years many results in a variety of disciplines have been questioned for their lack of reproducibility. A new study suggests that published results from experimental economics--a field pioneered at Caltech--are better than average when it comes to reproducibility.

The work was published in the March 3 online issue of the journal Science.

"Trying to reproduce previous results is not glamorous or creative, so it is rarely done. But being able to get the same result over and over is part of the definition of what makes knowledge scientific," says Colin Camerer, the Robert Kirby Professor of Behavioral Economics at Caltech and lead author on the paper.

The study was based on a previous method used to assess the replication of psychology experiments. In the earlier technique, called the reproducibility project psychology (RPP), researchers replicated 100 original studies published in three of the top journals in psychology--and found that although 97 percent of the original studies reported so-called "positive findings" (meaning a significant change compared to control conditions), such positive findings were reliably reproduced only 36 percent of the time.

Using this same technique, Camerer and his colleagues reproduced 18 laboratory experimental papers published in two top-tier economics journals between 2011 and 2014. Eleven of the 18--roughly 61 percent--showed a "significant effect in the same direction as in the original study." The researchers also found that the sample size and p-values--a standard measure of statistical confidence--of the original studies were good predictors for the success of replication, meaning they could serve as good indicators for the reliability of results in future experiments.

"Replicability has become a major issue in many sciences over the past few years, with often low replication rates," says paper coauthor Juergen Huber of the University of Innsbruck. "The rate we report for experimental economics is the highest we are aware of for any field."

The authors suggest that there are some methodological research practices in laboratory experimental economics that contribute to the good replication success. "It seems that the culture established in experimental economics--incentivizing subjects, publication of the experimental procedure and instructions, no deception--ensures reliable results. This is very encouraging given that it is a very young discipline," says Michael Kirchler, another coauthor and collaborator from the University of Innsbruck.

"As a journal editor myself, we are always curious whether experimental results will replicate across populations and cultures, and these results from multiple countries are really reassuring," says coauthor Teck-Hua Ho from the National University of Singapore.

Coauthor Magnus Johannesson from the Stockholm School of Economics adds, "It is extremely important to investigate to what extent we can trust published scientific findings and to implement institutions that promote scientific reproducibility."

"For the past half century, Caltech has been a leader in the development of social science experimental methods. It is no surprise that Caltech scholars are part of a group that use replication studies to demonstrate the validity of these methods," says Jean-Laurent Rosenthal, the Rea A. and Lela G. Axline Professor of Business Economics and chair of the Division of the Humanities and Social Sciences at Caltech.

The work was published in a paper titled, "Evaluating Replicability of Laboratory Experiments in Economics." Other coauthors are: Taisuke Imai and Gideon Nave from Caltech; Johan Almenberg from Sveriges Riksbank in Stockholm; Anna Dreber, Eskil Forsell, Adam Altmejd, Emma Heikensten, and Siri Isaksson from the Stockholm School of Economics; Taizan Chan and Hang Wu from the National University of Singapore; Felix Holzmeister and Michael Razen from the University of Innsbruck; and Thomas Pfeiffer from the New Zealand Institute for Advanced Study.

The study was funded by the Austrian Science Fund, the Austrian National Bank, the Behavioral and Neuroeconomics Discovery Fund, the Jan Wallander and Tom Hedelius Foundation, the Knut and Alice Wallenberg Foundation, the Swedish Foundation For Humanities and Social Sciences, and the Sloan Foundation.

California Institute of Technology

Related Economics Articles from Brightsurf:

A century of misunderstanding of a key tool in the economics of natural resources
In the past few weeks, oil prices have fallen to record lows.

What comes after COVID-19? Special issue in the journal Population and Economics
At this alarming time, when the COVID-19 pandemic is on everyone's mind, a new special issue in the open-access peer-reviewed journal Population and Economics provides a platform for discussion on the impact of the pandemic on the population and economics, both in Russia and worldwide.

Challenges for Russia's agriculture: new special issue in Russian Journal of Economics
While Russia seems to have tackled its historic problem: food shortage -- with the agri-food sector becoming one of the most steadily developing of the national economy in the last decade -- there is a new set of challenges.

Brief entrance test can predict academic success within first year of study in economics
German researchers at Johannes Gutenberg University Mainz (JGU) and Humboldt-Universit├Ąt zu Berlin found that even a short test can reliably predict students' success within their first year of study in economics -- much better than an intelligence test or predictions based on school grades.

The case for economics -- by the numbers
In recent years, criticism has been levelled at economics for being insular and unconcerned about real-world problems.

Scientists develop open-source software to analyze economics of biofuels, bioproducts
Perennial grasses can be converted into everything from ethanol to bioplastics, but it's unclear which bioproducts hold the greatest potential.

Life data economics: calling for new models to assess the value of human data
After the collapse of the blockchain bubble a number of research organisations are developing platforms to enable individual ownership of life data and establish the data valuation and pricing models.

SCAI and ACVP release consensus statement on cardiovascular catheterization laboratory economics
A newly released expert consensus statement provides recommendations for optimizing the financial operations of the cardiovascular catheterization laboratory (CCL) while providing cutting-edge patient care.

Shocking economics
Understanding economies in times of crises? Modern macroeconomics failed so far.

When does one of the central ideas in economics work?
Many situations in economics are complicated and competitive; this research raises the question of whether many theories in economics may suffer from the very fundamental problem that the key behavioral assumption of equilibrium is wrong.

Read More: Economics News and Economics Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.