Innovating the peer-review research process

April 16, 2020

A team of scientists led by a Michigan State University astronomer has found that a new process of evaluating proposed scientific research projects is as effective - if not more so - than the traditional peer-review method.

Normally, when a researcher submits a proposal, the funding agency then asks a number of researchers in that particular field to evaluate and make funding recommendations. A system that can sometimes be a bit bulky and slow - not quite an exact science.

"As in all human endeavors, this one has it flaws," said Wolfgang Kerzendorf, an assistant professor in MSU's departments of Physics and Astronomy, and Computational Mathematics, Science and Engineering.

Detailed in the publication Nature Astronomy, Kerzendorf and colleagues tested a new system that distributes the work load of reviewin¬¬g project proposals among the proposers, known as the "distributed peer review" approach.

However, the team enhanced it by using two other novel features: Using machine learning to match reviewers with proposals and the inclusion of a feedback mechanism on the review.

Essentially, this process consists of three different features designed to improve the peer-review process.

First, when a scientist submits a proposal for evaluation, he or she is first asked to review several of their competitors' papers, a way of lessening the amount of papers one is asked to review.

"If you lower the number of reviews that every person has to do, they may spend a little more time with each one of the proposals," Kerzendorf said.

Second, by using computers - machine learning - funding agencies can match up the reviewer with proposals of fields in which they are experts. This process can take human bias out of the equation, resulting in a more accurate review.

"We essentially look at the papers that potential readers have written and then give these people proposals they are probably good at judging," Kerzendorf said. "Instead of a reviewer self-reporting their expertise, the computer does the work."

And third, the team introduced a feedback system in which the person who submitted the proposal can judge if the feedback they received was helpful. Ultimately, this might help the community reward scientists that consistently provide constructive criticism.

"This part of the process is not unimportant," Kerzendorf said. "A good, constructive review is a bit of a bonus, a reward for the work you put in reviewing other proposals."

To do the experiment, Kerzendorf and his team considered 172 submitted proposals that each requested use of the telescopes on the European Southern Observatory, a 16-nation ground-based observatory in Germany.

The proposals were reviewed in both the traditional manner and using distributed peer review. The results? From a statistical standpoint, it was seemingly indistinguishable

However, Kerzendorf said this was a novel experiment testing a new approach to evaluating peer-review research, one that could make a difference in the scientific world.

"While we think very critically about science, we sometimes do not take the time to think critically about improving the process of allocating resources in science," he said. "This is an attempt to do this."
-end-
Other members of the research team included Ferdinando Patat, Dominic Bordelon and Glenn van de Ven of the European Southern Observatory; and Tyler Pritchard, Center for Cosmology and Particle Physics, New York University.

(Note for media: Please include a link to the original paper in online coverage: https://www.nature.com/articles/s41550-020-1038-y)

Michigan State University

Related Astronomy Articles from Brightsurf:

Spitzer space telescope legacy chronicled in Nature Astronomy
A national team of scientists Thursday published in the journal Nature Astronomy two papers that provide an inventory of the major discoveries made possible thanks to Spitzer and offer guidance on where the next generation of explorers should point the James Webb Space Telescope (JWST) when it launches in October 2021.

New technology is a 'science multiplier' for astronomy
A new study has tracked the long-term impact of early seed funding obtained from the National Science Foundation on many key advances in astronomy over the past three decades.

Powerful new AI technique detects and classifies galaxies in astronomy image data
Researchers at UC Santa Cruz have developed a powerful new computer program called Morpheus that can analyze astronomical image data pixel by pixel to identify and classify all of the galaxies and stars in large data sets from astronomy surveys.

Astronomy student discovers 17 new planets, including Earth-sized world
University of British Columbia astronomy student Michelle Kunimoto has discovered 17 new planets, including a potentially habitable, Earth-sized world, by combing through data gathered by NASA's Kepler mission.

Task force recommends changes to increase African-American physics and astronomy students
Due to long-term and systemic issues leading to the consistent exclusion of African-Americans in physics and astronomy, a task force is recommending sweeping changes and calling for awareness into the number and experiences of African-American students studying the fields.

How to observe a 'black hole symphony' using gravitational wave astronomy
New research led by Vanderbilt astrophysicist Karan Jani presents a compelling roadmap for capturing intermediate-mass black hole activity.

Graphene sets the stage for the next generation of THz astronomy detectors
Researchers from Chalmers University of Technology have demonstrated a detector made from graphene that could revolutionize the sensors used in next-generation space telescopes.

3D holograms bringing astronomy to life
Scientists unravelling the mysteries of star cluster formation have taken inspiration from a 19th century magic trick, to help explain their work to the public.

The vibrating universe: Making astronomy accessible to the deaf
Astronomers at the University of California, Riverside, have teamed with teachers at the California School for the Deaf, Riverside, or CSDR, to design an astronomy workshop for students with hearing loss that can be easily used in classrooms, museums, fairs, and other public events.

Prehistoric cave art reveals ancient use of complex astronomy
As far back as 40,000 years ago, humans kept track of time using relatively sophisticated knowledge of the stars

Read More: Astronomy News and Astronomy Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.