Nav: Home

New method peeks inside the 'black box' of artificial intelligence

October 31, 2018

Artificial intelligence--specifically, machine learning--is a part of daily life for computer and smartphone users. From autocorrecting typos to recommending new music, machine learning algorithms can help make life easier. They can also make mistakes.

It can be challenging for computer scientists to figure out what went wrong in such cases. This is because many machine learning algorithms learn from information and make their predictions inside a virtual "black box," leaving few clues for researchers to follow.

A group of computer scientists at the University of Maryland has developed a promising new approach for interpreting machine learning algorithms. Unlike previous efforts, which typically sought to "break" the algorithms by removing key words from inputs to yield the wrong answer, the UMD group instead reduced the inputs to the bare minimum required to yield the correct answer. On average, the researchers got the correct answer with an input of less than three words.

In some cases, the researchers' model algorithms provided the correct answer based on a single word. Frequently, the input word or phrase appeared to have little obvious connection to the answer, revealing important insights into how some algorithms react to specific language. Because many algorithms are programmed to give an answer no matter what--even when prompted by a nonsensical input--the results could help computer scientists build more effective algorithms that can recognize their own limitations.

The researchers will present their work on November 4, 2018 at the 2018 Conference on Empirical Methods in Natural Language Processing.

"Black-box models do seem to work better than simpler models, such as decision trees, but even the people who wrote the initial code can't tell exactly what is happening," said Jordan Boyd-Graber, the senior author of the study and an associate professor of computer science at UMD. "When these models return incorrect or nonsensical answers, it's tough to figure out why. So instead, we tried to find the minimal input that would yield the correct result. The average input was about three words, but we could get it down to a single word in some cases."

In one example, the researchers entered a photo of a sunflower and the text-based question, "What color is the flower?" as inputs into a model algorithm. These inputs yielded the correct answer of "yellow." After rephrasing the question into several different shorter combinations of words, the researchers found that they could get the same answer with "flower?" as the only text input for the algorithm.

In another, more complex example, the researchers used the prompt, "In 1899, John Jacob Astor IV invested $100,000 for Tesla to further develop and produce a new lighting system. Instead, Tesla used the money to fund his Colorado Springs experiments."

They then asked the algorithm, "What did Tesla spend Astor's money on?" and received the correct answer, "Colorado Springs experiments." Reducing this input to the single word "did" yielded the same correct answer.

The work reveals important insights about the rules that machine learning algorithms apply to problem solving. Many real-world issues with algorithms result when an input that makes sense to humans results in a nonsensical answer. By showing that the opposite is also possible--that nonsensical inputs can also yield correct, sensible answers--Boyd-Graber and his colleagues demonstrate the need for algorithms that can recognize when they answer a nonsensical question with a high degree of confidence.

"The bottom line is that all this fancy machine learning stuff can actually be pretty stupid," said Boyd-Graber, who also has co-appointments at the University of Maryland Institute for Advanced Computer Studies (UMIACS) as well as UMD's College of Information Studies and Language Science Center. "When computer scientists train these models, we typically only show them real questions or real sentences. We don't show them nonsensical phrases or single words. The models don't know that they should be confused by these examples."

Most algorithms will force themselves to provide an answer, even with insufficient or conflicting data, according to Boyd-Graber. This could be at the heart of some of the incorrect or nonsensical outputs generated by machine learning algorithms--in model algorithms used for research, as well as real-world algorithms that help us by flagging spam email or offering alternate driving directions. Understanding more about these errors could help computer scientists find solutions and build more reliable algorithms.

"We show that models can be trained to know that they should be confused," Boyd-Graber said. "Then they can just come right out and say, 'You've shown me something I can't understand.'"
-end-
In addition to Boyd-Graber, UMD-affiliated researchers involved with this work include undergraduate researcher Eric Wallace; graduate students Shi Feng and Pedro Rodriguez; and former graduate student Mohit Iyyer (M.S. '14, Ph.D. '17, computer science).

The research presentation, "Pathologies of Neural Models Make Interpretation Difficult," Shi Feng, Eric Wallace, Alvin Grissom II, Pedro Rodriguez, Mohit Iyyer, and Jordan Boyd-Graber, will be presented at the 2018 Conference on Empirical Methods in Natural Language Processing on November 4, 2018.

This work was supported by the Defense Advanced Research Projects Agency (Award No. HR0011-15-C-011) and the National Science Foundation (Award No. IIS1652666). The content of this article does not necessarily reflect the views of these organizations.

Media Relations Contact: Matthew Wright, 301-405-9267, mewright@umd.edu

University of Maryland
College of Computer, Mathematical, and Natural Sciences
2300 Symons Hall
College Park, MD 20742
http://www.cmns.umd.edu
@UMDscience

About the College of Computer, Mathematical, and Natural Sciences

The College of Computer, Mathematical, and Natural Sciences at the University of Maryland educates more than 9,000 future scientific leaders in its undergraduate and graduate programs each year. The college's 10 departments and more than a dozen interdisciplinary research centers foster scientific discovery with annual sponsored research funding exceeding $175 million.

University of Maryland

Related Algorithms Articles:

Developing a new AI breast cancer diagnostic tool
Scientists are developing a new way to identify the unique chemical 'fingerprints' for different types of breast cancers.
Artificial intelligence-based algorithm for intensive care of traumatic brain injury
A recent Finnish study, published in Scientific Reports, presents the first artificial intelligence (AI) based algorithm that may be utilized in the intensive care unit for treating patients with severe traumatic brain injury.
New algorithms train AI to avoid specific bad behaviors
Robots, self-driving cars and other intelligent machines could become better-behaved if machine-learning designers adopt a new framework for building AI with safeguards against specific undesirable outcomes.
New machine learning algorithms offer safety and fairness guarantees
Writing in Science, Thomas and his colleagues Yuriy Brun, Andrew Barto and graduate student Stephen Giguere at UMass Amherst, Bruno Castro da Silva at the Federal University of Rio Grande del Sol, Brazil, and Emma Brunskill at Stanford University this week introduce a new framework for designing machine learning algorithms that make it easier for users of the algorithm to specify safety and fairness constraints.
Algorithm for preventing 'undesirable behavior' works in gender fairness and health tests
A new framework for designing machine learning algorithms helps to prevent intelligent machines from exhibiting undesirable behavior, researchers report.
Are hiring algorithms fair? They're too opaque to tell, study finds
New research from a team of Computing and Information Science scholars at Cornell University raises questions about hiring algorithms and the tech companies who develop and use them: How unbiased is the automated screening process?
New algorithm can distinguish cyberbullies from normal Twitter users with 90% accuracy
A team of researchers, including faculty at Binghamton University, have developed machine learning algorithms which can successfully identify bullies and aggressors on Twitter with 90 percent accuracy.
AI learns complex gene-disease patterns
A deep learning model improves the ability to identify genes potentially involved in disease.
New brain map could improve AI algorithms for machine vision
An international team of researchers led by neuroscientists from CSHL and University of Sydney published an updated view on the primate brain's visual system organization, and they found that parts of the primate visual system may work differently than previously thought.
Two new algorithms can identify patients at risk of HIV
Two new studies developed algorithms that can identify patients who are at risk of acquiring HIV and may benefit from preventive care.
More Algorithms News and Algorithms Current Events

Top Science Podcasts

We have hand picked the top science podcasts of 2019.
Now Playing: TED Radio Hour

Accessing Better Health
Essential health care is a right, not a privilege ... or is it? This hour, TED speakers explore how we can give everyone access to a healthier way of life, despite who you are or where you live. Guests include physician Raj Panjabi, former NYC health commissioner Mary Bassett, researcher Michael Hendryx, and neuroscientist Rachel Wurzman.
Now Playing: Science for the People

#544 Prosperity Without Growth
The societies we live in are organised around growth, objects, and driving forward a constantly expanding economy as benchmarks of success and prosperity. But this growing consumption at all costs is at odds with our understanding of what our planet can support. How do we lower the environmental impact of economic activity? How do we redefine success and prosperity separate from GDP, which politicians and governments have focused on for decades? We speak with ecological economist Tim Jackson, Professor of Sustainable Development at the University of Surrey, Director of the Centre for the Understanding of Sustainable Propserity, and author of...
Now Playing: Radiolab

An Announcement from Radiolab