Highly cited studies on biomarkers often report larger effect size estimates than subsequent studies

May 31, 2011

Frequently cited studies involving associations of biomarkers report effect sizes that are often larger when compared to summary estimates from meta-analyses evaluating the same associations, according to a study in the June 1 issue of JAMA.

"Many new biomarkers are continuously proposed as potential determinants of disease risk, prognosis, or response to treatment. The plethora of statistically significant associations increases expectations for improvements in risk appraisal. However, many markers get evaluated only in 1 or a few studies. Among those evaluated more extensively, few reach clinical practice. This translational attrition requires better study. Are the effect sizes proposed in the literature accurate or overestimated?" the authors write.

John P. A. Ioannidis, M.D., D.Sc., of the Stanford University School of Medicine, Stanford, Calif., and University of Ioannina School of Medicine, Ioannina, Greece, and Orestis A. Panagiotou, M.D., of the University of Ioannina School of Medicine, conducted a review of the medical literature to examine biomarkers that had been evaluated in at least 1 highly cited study and for which at least 1 meta-analysis had been performed for that same association to compare the effect size of these associations. Eligible articles were those that had received more than 400 citations in the ISI Web of Science and that had been published in any of 24 highly cited biomedical journals. MEDLINE was searched for subsequent meta-analyses on the same associations (same biomarker and same outcome). The researchers identified 35 highly cited studies published between 1991 and 2006 for inclusion in the study. The highly cited biomarkers included genetic risk factors, blood proteins, other blood biomarkers, and infectious agent biomarkers. Cancer-related and cardiovascular-related outcomes predominated.

The researchers found that for 30 of the 35 associations (86 percent), the highly cited studies had a stronger effect estimate than the largest study; for 3 the largest and highly cited study coincided; and only twice was the effect estimate stronger in the largest than in the highly cited study. The relative risk estimate was in the opposite direction in the highly cited than in the largest study in 5 associations, and the increase was more than 2-fold greater in another 20 associations (more than 4-fold in 13 associations).

For 29 of the 35 highly cited studies (83 percent), the corresponding meta-analysis found a smaller effect. The relative risk estimate was in the opposite direction in the highly cited than in the meta-analysis in 4 associations and the increase was more than 2-fold greater in another 14 associations (more than 4-fold in 7 associations), the authors write. Only 15 associations were nominally statistically significant based on the largest studies. Thirty-two of the 35 associations showed nominally statistically significant increased risk based on the meta-analyses.

The researchers write that several reasons could explain false-positive and inflated results among the examined highly cited investigations. "Many of these studies were relatively small and among the first to report on the association of interest. Discoveries made in small studies are prone to overestimate or underestimate the actual association. Interest in publishing major discoveries leads to selective reporting from chasing significance."

"... our study documents that results in highly cited biomarker studies often significantly overestimate the findings seen from meta-analyses. Evidence from multiple studies, in particular large investigations, is necessary to appreciate the discriminating ability of these emerging risk factors. Rapid clinical adoption in the absence of such evidence may lead to wasted resources."

(JAMA. 2011;305[21]2200-2210. Available pre-embargo to the media at www.jamamedia.org)

Editor's Note: Please see the article for additional information, including other authors, author contributions and affiliations, financial disclosures, funding and support, etc.

Editorial: The Thin Line Between Hope and Hype in Biomarker Research

In an accompanying editorial, Patrick M. M. Bossuyt, Ph.D., of the University of Amsterdam, the Netherlands, writes that it would be premature to doubt all scientific efforts at marker discovery and unwise to discount all future biomarker evaluation studies.

"However, the analysis presented by Ioannidis and Panagiotou should convince clinicians and researchers to be careful to match personal hope with professional skepticism, to apply critical appraisal of study design and close scrutiny of findings where indicated, and to be aware of the findings of well-conducted systematic reviews and meta-analyses when evaluating the evidence on biomarkers."

(JAMA. 2011;305[21]2229-2230. Available pre-embargo to the media at www.jamamedia.org)

Editor's Note: Please see the article for additional information, financial disclosures, funding and support, etc.
-end-
To contact John P. A. Ioannidis, M.D., D.Sc., call Krista Conger at 650-725-5371 or email kristac@stanford.edu. To contact editorial author Patrick M. M. Bossuyt, Ph.D., email p.m.bossuyt@amc.uva.nl.

The JAMA Network Journals

Related Biomarkers Articles from Brightsurf:

Urgent need for blood-based biomarkers to diagnosis concussion
There is an urgent need for objective markers for diagnosing concussion, or mild traumatic brain injury.

Engineered bacteria churn out cancer biomarkers
A Cornell lab has created these very tools by commandeering simple, single-celled microorganisms - namely E. coli bacteria - and engineering them to explore the complex process of glycosylation and the functional role that protein-linked glycans play in health and disease.

Exercise induces secretion of biomarkers into sweat
The aim was to reveal the potential of microRNAs in sweat extracellular vesicles in monitoring exercise performance.

Phosphoprotein biomarkers to guide cancer therapy are identified
Researchers led by James Bibb, Ph.D., professor of surgery at the University of Alabama at Birmingham, suggest using a broader lens of post-translational modification analysis to identify new biomarkers of cancer drivers that may allow a much more precise prediction of patient responses to treatments.

Exhaled biomarkers can reveal lung disease
Using specialized nanoparticles, MIT engineers have developed a way to diagnose pneumonia or other lung diseases by analyzing the breath exhaled by the patient.

Race-specific lupus nephritis biomarkers
A University of Houston biomedical researcher has discovered a difference in urinary biomarker proteins of lupus nephritis in patients according to race.

Semen miRNAs could be non-invasive biomarkers for prostate cancer
Researchers of the Human Molecular Genetics group at the Bellvitge Biomedical Research Institute (IDIBELL), led by Dr.

Scientists have found longevity biomarkers
An international group of scientists studied the effects of 17 different lifespan-extending interventions on gene activity in mice and discovered genetic biomarkers of longevity.

After concussion, biomarkers in the blood may help predict recovery time
A study of high school and college football players suggests that biomarkers in the blood may have potential use in identifying which players are more likely to need a longer recovery time after concussion, according to a study published in the July 3, 2019, online issue of Neurology, the medical journal of the American Academy of Neurology.

3D-printed device detects biomarkers of preterm birth
Preterm birth (PTB) -- defined as birth before the 37th week of gestation -- is the leading complication of pregnancy.

Read More: Biomarkers News and Biomarkers Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.