Nav: Home

The Forecaster's Dilemma: Evaluating forecasts of extreme events

April 10, 2017

When it comes to extreme events, public discussion of forecasts often focuses on predictive performance. After the international financial crisis of 2007, for example, the public paid a great deal of attention to economists who had correctly predicted the crisis, attributing it to their superior predictive ability. However, restricting forecast evaluation to subsets of extreme observations has unexpected and undesired effects, and is bound to discredit even the most expert forecasts. In a recent article, statisticians Dr. Sebastian Lerch and Prof. Tilmann Gneiting (both affiliated with HITS and the Karlsruhe Institute of Technology), together with their coauthors from Norway and Italy, analyzed and explained this phenomenon and suggested potential remedies. The research team used theoretical arguments, simulation experiments and a real data study on economic variables. The article has just been published in the peer-reviewed journal Statistical Science. It is based on research funded by the Volkswagen Foundation.

Predicting calamities every time - a worthwhile strategy?

Forecast evaluation is often only conducted in the public arena if an extreme event has been observed; in particular, if forecasters have failed to predict an event with high economic or societal impact. An example of what this can mean for forecasters is the devastating L'Aquila earthquake in 2009 that caused 309 deaths. In the aftermath, six Italian seismologists were put on trial for not predicting the earthquake. They were found guilty of involuntary manslaughter and sentenced to six years in prison until the Supreme Court in Rome acquitted them of the charges.

But how can scientists and outsiders, such as the media, evaluate the accuracy of forecasts predicting extreme events? At first sight, the practice of selecting extreme observations while discarding non-extreme ones and proceeding on the basis of standard evaluation tools seems quite logical. Intuitively, accurate predictions on the subset of extreme observations may suggest superior predictive abilities. But limiting the analyzed data to selected subsets can be problematic. "In a nutshell, if forecast evaluation is conditional on observing a catastrophic event, predicting a disaster every time becomes a worthwhile strategy," says Sebastian Lerch, member of the "Computational Statistics" group at HITS. Given that media attention tends to focus on extreme events, expert forecasts are bound to fail in the public eye, and it becomes tempting to base decision making on misguided inferential procedures. "We refer to this critical issue as the 'forecaster's dilemma,'" adds Tilmann Gneiting.

Avoiding the forecaster's dilemma: Method is everything.

This predicament can be avoided if forecasts take the form of probability distributions, for which standard evaluation methods can be generalized to allow for specifically emphasizing extreme events. The paper uses economic forecasts of GDP growth and inflation rates in the United States between 1985 and 2011 to illustrate the forecaster's dilemma and how these tools can be used to address it.

The results of the study are especially relevant for scientists seeking to evaluate the forecasts of their own methods and models, and for external third parties who need to choose between competing forecast providers, for example to produce hazard warnings or make financial decisions.

Although the research paper focused on an economic data set, the conclusions are important for many other applications: The forecast evaluation tools are currently being tested for use by national and international weather services.
-end-
Publication:

Lerch, S., Thorarinsdottir, T. L., Ravazzolo, F. and Gneiting, T. (2017). Forecaster's dilemma: Extreme events and forecast evaluation. Statistical Science, in press.

DOI: 10.1214/16-STS588

Link:http://projecteuclid.org/euclid.ss/1491465630

Scientific Contact:

Prof. Dr. Tilmann Gneiting
Computational Statistics (CST) group
HITS Heidelberg Institute for Theoretical Studies
Tilmann.gneiting@h.its.org
Schloss-Wolfsbrunnenweg 35
69118 Heidelberg

Press Contact:

Dr. Peter Saueressig
Head of Communications
Heidelberg Institute for Theoretical Studies (HITS)
Phone: +49 6221 533 245
peter.saueressig@h-its.org

About HITS

The Heidelberg Institute for Theoretical Studies (HITS) was established in 2010 by the physicist and SAP co-founder Klaus Tschira (1940-2015) and the Klaus Tschira Foundation as a private, non-profit research institute. HITS conducts basic research in the natural sciences, mathematics and computer science, with a focus on the processing, structuring, and analyzing of large amounts of complex data and the development of computational methods and software. The research fields range from molecular biology to astrophysics. The shareholders of HITS are the HITS Stiftung, which is a subsidiary of the Klaus Tschira Foundation, Heidelberg University and the Karlsruhe Institute of Technology (KIT). HITS also cooperates with other universities and research institutes and with industrial partners. The base funding of HITS is provided by the HITS Stiftung with funds received from the Klaus Tschira Foundation. The primary external funding agencies are the Federal Ministry of Education and Research (BMBF), the German Research Foundation (DFG), and the European Union.

Heidelberg Institute for Theoretical Studies (HITS)

Related Media Articles:

Teens must 'get smart' about social media
New research indicates that social media is leading young adolescent girls and boys down a worrying path towards developing body image issues and eating disorder behaviours - even though they are smartphone savvy.
Cynical social media voices can erode trust in news media
Amid rising concerns about low public trust in mainstream media institutions, a Rutgers study found that real-life and online social interactions can strongly influence a person's trust in newspaper, TV and online journalism -- but when it comes to online interactions, cynical views are the most influential.
Social media stress can lead to social media addiction
Social network users risk becoming more and more addicted to social media platforms even as they experience stress from their use.
'Fake news,' diminishing media trust and the role of social media
Exploring the perception of the 'fake news' phenomenon is critical to combating the ongoing global erosion of trust in the media according to a study co-authored by a University of Houston researcher.
Journalists can restore media trust
In a first-of-its-kind study, researchers discovered journalists can increase media trust by speaking out in defense of their profession, while also doing more fact checking.
Media research: Time not up for newsprint
Youth spend more time with newspapers in print than online, shows a new study.
Cosmetic surgery on social media -- Patients rate preferred social media sites and content
Plastic surgeons using social media to attract patients should know their audience's preferred social media platforms and the types of posts of greatest interest, according to a survey study in the November issue of Plastic and Reconstructive Surgery®, the official medical journal of the American Society of Plastic Surgeons (ASPS).
Where there's fire, there's smoke -- and social media
The fact that people reliably flock to social media to discuss smoke and fire was the inspiration for a new study by Colorado State University atmospheric scientists.
Social media: Simplifying surveillance
The controversial Snap Map app enables Snapchat users to track their friends.
Your (social media) votes matter
Tim Weninger, assistant professor of computer science and engineering at the University of Notre Dame, conducted two large-scale experiments on Reddit and the results provide insight into how a single up/down vote can influence what content users see on the site.
More Media News and Media Current Events

Top Science Podcasts

We have hand picked the top science podcasts of 2019.
Now Playing: TED Radio Hour

In & Out Of Love
We think of love as a mysterious, unknowable force. Something that happens to us. But what if we could control it? This hour, TED speakers on whether we can decide to fall in — and out of — love. Guests include writer Mandy Len Catron, biological anthropologist Helen Fisher, musician Dessa, One Love CEO Katie Hood, and psychologist Guy Winch.
Now Playing: Science for the People

#543 Give a Nerd a Gift
Yup, you guessed it... it's Science for the People's annual holiday episode that helps you figure out what sciency books and gifts to get that special nerd on your list. Or maybe you're looking to build up your reading list for the holiday break and a geeky Christmas sweater to wear to an upcoming party. Returning are pop-science power-readers John Dupuis and Joanne Manaster to dish on the best science books they read this past year. And Rachelle Saunders and Bethany Brookshire squee in delight over some truly delightful science-themed non-book objects for those whose bookshelves are already full. Since...
Now Playing: Radiolab

An Announcement from Radiolab