Nav: Home

Aiming for gold: improving reproducibility in hydrology studies

February 27, 2019

LOGAN, UT - In six well-regarded hydrology and water resources journals published in 2017, the estimated percentage of studies whose results could be fully reproduced was only between 0.06 and 6.8 percent. This low level of reproducibility is not uncommon in hydrology studies -- a fact many scientists readily acknowledge. However, a team of researchers at Utah State University may have found a solution to make these studies more reproducible.

In their paper, "Assessing data availability and research reproducibility in hydrology and water resources," published Feb. 26 in Nature's Scientific Data, David Rosenberg co-authors developed an online survey tool to assess the reproducibility of published research. The team reviewed 360 articles from six water resource journals published in 2017. Out of 360 articles, they could only fully reproduce results from four articles.

"Our survey tool breaks down the concept of scientific reproducibility into specific components of data availability, reproducibility of results and replicability of findings," said Rosenberg, an associate professor of civil and environmental engineering at USU. "We then suggest how authors, journals, funders and institutions can use the survey tool to increase low rates of reproducibility."

The authors say reproducibility can be broken down into three components:
  • Are the data, models, code, directions for use and other artifacts used in the work available?
  • Can artifacts be used to reproduce published results?
  • Can the findings be replicated with new datasets?
The team's online survey tool consists of 15 questions and provides a checklist of the essential items needed for artifact availability and result reproducibility. Artifacts is an umbrella term for all data, software, models, code, directions and other materials needed to reproduce results within a study.

The team found that about 70 percent of the sampled articles stated some materials were available but only around 48 percent of the materials could be accessed online. Only around six percent of the sampled articles made artifacts publicly available, and only one percent of sampled articles made artifacts available and could be fully reproduced.

The authors said many articles were missing directions to generate results. If authors provide directions, they say, the number of articles that could be tested for reproducibility would double. Articles that made all artifacts available had a six in 10 chance of having some or all of their results reproduced. Two journals surveyed by the team required articles to state how artifacts can be accessed, and four journals encouraged statements. No journals required authors to make all artifacts available.

The survey tool can help recognize and encourage authors to achieve certain reproducibility levels. For example, authors can use the survey tool to self-assess the reproducibility of their results. Rosenberg and his team also recommend a medal system to recognize different levels of reproducibility:
  • Bronze Medal: All artifacts are made available within the article or in open repositories
  • Silver Medal: All artifacts are made available and results are fully reproducible
  • Gold Medal: Results are fully reproducible and overall findings can be replicated in different settings with the same or different artifacts
Rosenberg recommend that medal icons be posted next to online articles to recognize authors for their reproducibility work and make it easier for readers to find top reproducibility practices. Rosenberg and his team assigned four silver medals and six bronze medals out of the 360 articles they reviewed. Awarding gold medals for replicability of findings remains an important line of future work.

"We hope the survey tool will help nudge authors, journals, funders and institutions to make scientific work more reproducible," said Rosenberg. "We welcome discussion to improve the survey tool and to improve the reproducibility of our science."
Research Contact: David Rosenberg | | 435-797-8689

Writer: Lexie Richins |

Utah State University

Related Water Resources Articles:

Research links decline in hemlock forests to changes in water resources
An insect infestation that is killing hemlock trees in New England forests is having a significant impact on the water resources of forested ecosystems that provide essential water supplies to one of the nation's most populous regions.
It's not too late to conserve water resources in rapidly urbanizing areas
As climate change and population pressure intensify in suburbia, a new study by watershed scientist Timothy Randhir at UMass Amherst suggests that threats such as water shortages and poor quality can be met if managers begin to act now.
USU scientist explores 'big picture' of hydrologic modeling for water resources management
Using water resources systems analysis and physical geography, Utah State University researcher Sarah Null is developing mathematical models to explore processes and interactions of both built and natural water systems.
Utah State University researchers receive American Water Resources Association Award
Utah State University researchers Enjie Li, Joanna Endter-Wada and Shujuan Li were honored with the 2016 William R.
Jumping water striders know how to avoid breaking of the water surface
When escaping from attacking predators, different water strider species adjust their jump performance to their mass and morphology in order to jump off the water as fast and soon as possible without breaking of the water surface.
Syrian crisis altered region's land and water resources, Stanford study finds
Using remote sensing tools to uncover the environmental impacts of war, researchers introduce novel approaches for hard-to-reach areas.
Water, water -- the two types of liquid water
There are two types of liquid water, according to research carried out by an international scientific collaboration.
US taps NCAR technology for new water resources forecasts
As NOAA launches a comprehensive system this month for forecasting water resources, it's turning to NCAR technology.
Just add water? New MRI technique shows what drinking water does to your appetite, stomach and brain
Stomach MRI images combined with functional fMRI of the brain activity have provided scientists new insight into how the brain listens to the stomach during eating.
Climate change redistributes global water resources
Rising temperatures worldwide are changing not only weather systems, but -- just as importantly -- the distribution of water around the globe, according to a study published today (March 14, 2016) in the journal, 'Scientific Reports.'

Related Water Resources Reading:

Best Science Podcasts 2019

We have hand picked the best science podcasts for 2019. Sit back and enjoy new science podcasts updated daily from your favorite science news services and scientists.
Now Playing: TED Radio Hour

Digital Manipulation
Technology has reshaped our lives in amazing ways. But at what cost? This hour, TED speakers reveal how what we see, read, believe — even how we vote — can be manipulated by the technology we use. Guests include journalist Carole Cadwalladr, consumer advocate Finn Myrstad, writer and marketing professor Scott Galloway, behavioral designer Nir Eyal, and computer graphics researcher Doug Roble.
Now Playing: Science for the People

#529 Do You Really Want to Find Out Who's Your Daddy?
At least some of you by now have probably spit into a tube and mailed it off to find out who your closest relatives are, where you might be from, and what terrible diseases might await you. But what exactly did you find out? And what did you give away? In this live panel at Awesome Con we bring in science writer Tina Saey to talk about all her DNA testing, and bioethicist Debra Mathews, to determine whether Tina should have done it at all. Related links: What FamilyTreeDNA sharing genetic data with police means for you Crime solvers embraced...