Nav: Home

Bringing a 'trust but verify' model to journal peer review

July 31, 2017

Academic journals are increasingly asking authors to use transparent reporting practices to "trust, but verify" that outcomes are not being reported in a biased way and to enable other researchers to reproduce the results.

To implement these reporting practices, most journals rely on the process of peer review -- in which other scholars review research findings before publication -- but relatively few journals measure the quality and effectiveness of the process.

In a commentary published July 20 in the journal Science, lead author Carole Lee and co-author David Moher identify incentives that could encourage journals to "open the black box of peer review" for the sake of improving transparency, reproducibility, and trust in published research. Lee is an associate professor of philosophy at the University of Washington; Moher is a senior scientist at the Ottawa Hospital and associate professor at the University of Ottawa.

Lee and Moher see this as a collective action problem requiring leadership and investment by publishers.

"Science would be better off if journals allowed for and participated in the empirical study and quality assurance of their peer review processes," they write. "However, doing so is resource-intensive and comes at considerable risk for individual journals in the form of unfavorable evidence and bad press."

To help journals manage the reputational risk associated with auditing their own peer review processes, Lee and Moher suggest revising the Transparency and Openness (TOP) Guidelines, a set of voluntary reporting standards to which 2,900 journals and organizations are now signatories. These guidelines were published in Science in 2015 by a committee of researchers and representatives from nonprofit scientific organizations, grant agencies, philanthropic organizations and elite journals.

Lee and Moher suggest adding a new category to the TOP guidelines "indicating a journal's willingness to facilitate meta-research on the effectiveness of its own peer review practices." With these, journals can choose which tier or level they take on. Higher levels of transparency would involve higher risk.
  • For the lowest tier, journals would publicly disclose whether they are conducting internal evaluations of peer review, in which they are able to retain the study results for internal use.
  • At the middle tier, journals would disclose the results of their internal evaluations of peer review, but could maintain flexibility in how they report their results for external use. For example, results could be aggregated across several journals to reduce risk to any single journal.
  • At the upper tier, journals could agree to relinquish data and analyses to researchers outside their institution for third-party verification. This is an option, Lee and Moher write, "that might appeal especially to publishers with fewer resources, as it places the financial burden on those conducting the meta-research." Journals conducting their own analyses could preregister their study designs then deposit their data publicly online.
By agreeing to these more stringent guidelines, the authors write, publishers and journals would have the chance to legitimize and advertise the relative quality of their peer review process in an age when predatory journals, which falsely claim to use peer review, continue to proliferate.

"Illegitimate journals are becoming a big problem for science," said Moher. "True scientific journals can distinguish themselves with transparence about their peer review processes."

Investing in research on journal peer review will be costly, they agree. Lee and Moher suggest that large experimental studies are needed to judge the effectiveness of different web-based peer review templates to enforce reporting standards, and of ways one might train authors, reviewers and editors to use such tools and evaluate research.

Also needed, they say, are ways to detect shortcomings in statistical and methodological reporting on a research paper, and to understand how the number and relative expertise of peer reviewers can improve assessment.

The largest publishers, whose profit margins compete with those of pharmaceutical and tech giants, can afford to invest in the requisite technology and resources needed to carry out these audits, the researchers say.

"Publishers should invest in their own brands and reputations by investing in the quality of their peer review processes," said Lee. "Ultimately, this would improve the quality of the published scientific literature."
-end-
For more information, contact Lee at 206-543-9888 or c3@uw.edu, or Moher at dmoher@ohri.ca, or through Jennifer Ganton, the Ottawa Hospital Research Institute's public relations officer, at 613-614-5253 or jganton@ohri.ca.

University of Washington

Related Peer Review Articles:

Peer support reduces carer burden
In a world first, La Trobe University research has shown how peer-led support programs for family and friends who provide regular support to an adult diagnosed with a mental health condition can significantly improve carer well-being.
New research to explore technology needed for peer-to-peer 'free trade' in excess energy
People who generate their own power through solar panels and wind turbines may soon be able to decide where to distribute their excess energy, rather than back to the national grid.
Infants are more likely to learn when with a peer
Researchers at the University of Connecticut and University of Washington looked at the mechanisms involved in language learning among nine-month-olds, the youngest population known to be studied in relation to on-screen learning.
Potential gender bias against female researchers in peer review of research grants
Is peer review biased? Female health researchers who applied for grants from Canada's major health research funder were funded less often than male counterparts because of potential bias, and characteristics of peer reviewers can also affect the result, found a study in CMAJ (Canadian Medical Association Journal).
Bringing a 'trust but verify' model to journal peer review
In a commentary published July 20 in the journal Science, lead author Carole Lee and co-author David Moher identify incentives that could encourage journals to 'open the black box of peer review' for the sake of improving transparency, reproducibility and trust in published research.
Breast-feeding peer support services are lacking in many UK regions
Peer support is recommended by the World Health Organization for the initiation and continuation of breastfeeding, but in a survey of 136 service managers with jobs related to infant feeding across UK NHS Trust and Health Board areas, breastfeeding peer supporters were available in only 56 percent of NHS areas.
Peer-led self-management programs may not help teenagers with asthma
A study from the University of Warwick suggests peer-led self-management programs have little impact on the quality of life or lung function of adolescents with asthma.
FDA approves drugs more quickly than peer agency in Europe
The US Food and Drug Administration (FDA) reviews and approves new medicines in a shorter timeframe than its peer agency in Europe, the European Medicines Agency (EMA), says a Yale researcher.
Peer milk-sharing participants generally keep it clean
Mothers who want the benefits of breast milk for their babies but can't produce the substance often turn to milk-sharing networks.
New study highlights need to increase female peer reviewers
Publishers of scholarly journals should take additional steps to get more women to serve as peer reviewers of manuscripts, according to a new analysis by the American Geophysical Union showing women across all age groups have fewer opportunities than men to participate in this career-building activity.
More Peer Review News and Peer Review Current Events

Best Science Podcasts 2019

We have hand picked the best science podcasts for 2019. Sit back and enjoy new science podcasts updated daily from your favorite science news services and scientists.
Now Playing: TED Radio Hour

Rethinking Anger
Anger is universal and complex: it can be quiet, festering, justified, vengeful, and destructive. This hour, TED speakers explore the many sides of anger, why we need it, and who's allowed to feel it. Guests include psychologists Ryan Martin and Russell Kolts, writer Soraya Chemaly, former talk radio host Lisa Fritsch, and business professor Dan Moshavi.
Now Playing: Science for the People

#538 Nobels and Astrophysics
This week we start with this year's physics Nobel Prize awarded to Jim Peebles, Michel Mayor, and Didier Queloz and finish with a discussion of the Nobel Prizes as a way to award and highlight important science. Are they still relevant? When science breakthroughs are built on the backs of hundreds -- and sometimes thousands -- of people's hard work, how do you pick just three to highlight? Join host Rachelle Saunders and astrophysicist, author, and science communicator Ethan Siegel for their chat about astrophysics and Nobel Prizes.