An analysis of psychological meta-analyses reveals a reproducibility problem

May 27, 2020

Meta-analysis research studies in psychology aren't always reproducible due to a lack of transparency of reporting in the meta-analysis process, according to a new study published May 27, 2020 in the open-access journal PLOS ONE by Esther Maassen of Tilburg University, the Netherlands, and colleagues.

Meta-analysis is a widely used method to combine and compare quantitative data from multiple primary studies. The statistical approach used in meta-analyses can reveal whether study outcomes differ based on particular study characteristics, and help compute an overall effect size--for instance, the magnitude of a treatment effect--for the topic of interest. However, many steps of a meta-analysis involve decisions and judgements that can be arbitrary or differ by researcher.

In the new study, researchers analyzed 33 meta-analysis articles in the field of psychology. The meta-analytical studies were all published in 2011 and 2012, all had data tables with primary studies, and all included at least ten primary studies. For each meta-analysis, the team searched for the corresponding primary study articles, followed any methods detailed in the meta-analysis article, and recomputed a total of 500 effect sizes reported in the meta-analyses.

Out of 500 primary study effect sizes, the researchers were able to reproduce 276 (55%) without any problems. (In this case, reproducibility was defined as arriving at the same result after reanalyzing the same data following the reported procedures.) However, in some cases, the meta-analyses did not contain enough information to reproduce the study effect size, while in others a different effect than stated was calculated. 114 effect sizes (23%) showed discrepancies compared to what was reported in the meta-analytical article. 30 of the 33 meta-analyses contained at least one effect size that could not be easily reproduced.

When the erroneous or unreproducible effect sizes were integrated into each meta-analysis itself, the team found that 13 of the 33 (39%) meta-analyses had discrepancies in their results, although many were negligible. The researchers recommend adding to existing guidelines for the publication of psychological meta-analyses to make them more reproducible.

The authors add: Individual effect sizes from meta-analyses in psychology are difficult to reproduce due to inaccurate and incomplete reporting in the meta-analysis. To increase the trustworthiness of meta-analytic results, it is essential that researchers explicitly document their data handling practices and workflow, as well as publish their data and code online.
Citation: Maassen E, van Assen MALM, Nuijten MB, Olsson-Collentine A, Wicherts JM (2020) Reproducibility of individual effect sizes in meta-analyses in psychology. PLoS ONE 15(5): e0233107.

Funding: This research was supported by a Consolidator Grant 726361 (IMPROVE) from the European Research Council (ERC,, awarded to J.M. Wicherts. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing Interests: I have read the journal's policy and the authors of this manuscript have the following competing interests: Jelte M. Wicherts is a PLOS ONE Editorial Board member. This does not alter the authors' adherence to PLOS ONE Editorial policies and criteria.

In your coverage please use this URL to provide access to the freely available article in PLOS ONE:


Related Psychology Articles from Brightsurf:

More than one cognition: A call for change in the field of comparative psychology
In a paper published in the Journal of Intelligence, researchers argue that cognitive studies in comparative psychology often wrongly take an anthropocentric approach, resulting in an over-valuation of human-like abilities and the assumption that cognitive skills cluster in animals as they do in humans.

Psychology research: Antivaxxers actually think differently than other people
As vaccine skepticism has become increasingly widespread, two researchers in the Texas Tech University Department of Psychological Sciences have suggested a possible explanation.

In court, far-reaching psychology tests are unquestioned
Psychological tests are important instruments used in courts to aid legal decisions that profoundly affect people's lives.

Psychology program for refugee children improves wellbeing
A positive psychology program created by researchers at Queen Mary University of London focuses on promoting wellbeing in refugee children.

Psychology can help prevent deadly childhood accidents
Injuries have overtaken infectious disease as the leading cause of death for children worldwide, and psychologists have the research needed to help predict and prevent deadly childhood mishaps, according to a presentation at the annual convention of the American Psychological Association.

Raising the standard for psychology research
Researchers from Stanford University, Arizona State University, and Dartmouth College used Texas Advanced Computing Center supercomputers to apply more rigorous statistical methods to psychological studies of self-regulation.

Psychology: Robot saved, people take the hit
To what extent are people prepared to show consideration for robots?

Researchers help to bridge the gap between psychology and gamification
A multi-disciplinary research team is bridging the gap between psychology and gamification that could significantly impact learning efforts in user experience design, healthcare, and government.

Virtual reality at the service of psychology
Our environment is composed according to certain rules and characteristics which are so obvious to us that we are scarcely aware of them.

Modeling human psychology
A human being's psychological make-up depends on an array of emotional and motivational parameters.

Read More: Psychology News and Psychology Current Events is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to