Nav: Home

Early Bird uses 10 times less energy to train deep neural networks

May 18, 2020

HOUSTON -- (May 18, 2020) -- Rice University's Early Bird could care less about the worm; it's looking for megatons of greenhouse gas emissions.

Early Bird is an energy-efficient method for training deep neural networks (DNNs), the form of artificial intelligence (AI) behind self-driving cars, intelligent assistants, facial recognition and dozens more high-tech applications.

Researchers from Rice and Texas A&M University unveiled Early Bird April 29 in a spotlight paper at ICLR 2020, the International Conference on Learning Representations. A study by lead authors Haoran You and Chaojian Li of Rice's Efficient and Intelligent Computing (EIC) Lab showed Early Bird could use 10.7 times less energy to train a DNN to the same level of accuracy or better than typical training. EIC Lab director Yingyan Lin led the research along with Rice's Richard Baraniuk and Texas A&M's Zhangyang Wang.

"A major driving force in recent AI breakthroughs is the introduction of bigger, more expensive DNNs," Lin said. "But training these DNNs demands considerable energy. For more innovations to be unveiled, it is imperative to find 'greener' training methods that both address environmental concerns and reduce financial barriers of AI research."

Training cutting-edge DNNs is costly and getting costlier. A 2019 study by the Allen Institute for AI in Seattle found the number of computations needed to train a top-flight deep neural network increased 300,000 times between 2012-2018, and a different 2019 study by researchers at the University of Massachusetts Amherst found the carbon footprint for training a single, elite DNN was roughly equivalent to the lifetime carbon dioxide emissions of five U.S. automobiles.

DNNs contain millions or even billions of artificial neurons that learn to perform specialized tasks. Without any explicit programming, deep networks of artificial neurons can learn to make humanlike decisions -- and even outperform human experts -- by "studying" a large number of previous examples. For instance, if a DNN studies photographs of cats and dogs, it learns to recognize cats and dogs. AlphaGo, a deep network trained to play the board game Go, beat a professional human player in 2015 after studying tens of thousands of previously played games.

"The state-of-art way to perform DNN training is called progressive prune and train," said Lin, an assistant professor of electrical and computer engineering in Rice's Brown School of Engineering. "First, you train a dense, giant network, then remove parts that don't look important -- like pruning a tree. Then you retrain the pruned network to restore performance because performance degrades after pruning. And in practice you need to prune and retrain many times to get good performance."

Pruning is possible because only a fraction of the artificial neurons in the network can potentially do the job for a specialized task. Training strengthens connections between necessary neurons and reveals which ones can be pruned away. Pruning reduces model size and computational cost, making it more affordable to deploy fully trained DNNs, especially on small devices with limited memory and processing capability.

"The first step, training the dense, giant network, is the most expensive," Lin said. "Our idea in this work is to identify the final, fully functional pruned network, which we call the 'early-bird ticket,' in the beginning stage of this costly first step."

By looking for key network connectivity patterns early in training, Lin and colleagues were able to both discover the existence of early-bird tickets and use them to streamline DNN training. In experiments on various benchmarking data sets and DNN models, Lin and colleagues found Early Bird could emerge as little as one-tenth or less of the way through the initial phase of training.

"Our method can automatically identify early-bird tickets within the first 10% or less of the training of the dense, giant networks," Lin said. "This means you can train a DNN to achieve the same or even better accuracy for a given task in about 10% or less of the time needed for traditional training, which can lead to more than one order savings in both computation and energy."

Developing techniques to make AI greener is the main focus of Lin's group. Environmental concerns are the primary motivation, but Lin said there are multiple benefits.

"Our goal is to make AI both more environmentally friendly and more inclusive," she said. "The sheer size of complex AI problems has kept out smaller players. Green AI can open the door enabling researchers with a laptop or limited computational resources to explore AI innovations."
-end-
Additional co-authors include Pengfei Xu, Yonggan Fu and Yue Wang, all of Rice, and Xiaohan Chen of Texas A&M.

The research was supported by the National Science Foundation (1937592, 1937588).

Links and resources:

ICLR paper on Early Bird DNN training: openreview.net/pdf?id=BJxsrgStvr

ICLR presentation on Early Bird DNN training: iclr.cc/virtual_2020/poster_BJxsrgStvr.html

Yingyan Lin: ece.rice.edu/people/yingyan-lin

Electrical and Computer Engineering at Rice: ece.rice.edu

George R. Brown School of Engineering at Rice: engineering.rice.edu

High-resolution IMAGES are available for download at:

https://news-network.rice.edu/news/files/2020/05/0518_EARLYBIRD-ylin-med.jpg

CAPTION: Yingyan Lin (Photo courtesy of Rice University)

https://news-network.rice.edu/news/files/2020/05/0518_EARLYBIRD-fig-lg.jpg

CAPTION: Rice University's Early Bird method for training deep neural networks finds key connectivity patterns early in training, reducing the computations and carbon footprint for the increasingly popular form of artificial intelligence known as deep learning. (Graphic courtesy of Y. Lin/Rice University)

This release can be found online at news.rice.edu.

Follow Rice News and Media Relations via Twitter @RiceUNews.

Located on a 300-acre forested campus in Houston, Rice University is consistently ranked among the nation's top 20 universities by U.S. News & World Report. Rice has highly respected schools of Architecture, Business, Continuing Studies, Engineering, Humanities, Music, Natural Sciences and Social Sciences and is home to the Baker Institute for Public Policy. With 3,962 undergraduates and 3,027 graduate students, Rice's undergraduate student-to-faculty ratio is just under 6-to-1. Its residential college system builds close-knit communities and lifelong friendships, just one reason why Rice is ranked No. 1 for lots of race/class interaction and No. 4 for quality of life by the Princeton Review. Rice is also rated as a best value among private universities by Kiplinger's Personal Finance.

Rice University

Related Neurons Articles:

Paying attention to the neurons behind our alertness
The neurons of layer 6 - the deepest layer of the cortex - were examined by researchers from the Okinawa Institute of Science and Technology Graduate University to uncover how they react to sensory stimulation in different behavioral states.
Trying to listen to the signal from neurons
Toyohashi University of Technology has developed a coaxial cable-inspired needle-electrode.
A mechanical way to stimulate neurons
Magnetic nanodiscs can be activated by an external magnetic field, providing a research tool for studying neural responses.
Extraordinary regeneration of neurons in zebrafish
Biologists from the University of Bayreuth have discovered a uniquely rapid form of regeneration in injured neurons and their function in the central nervous system of zebrafish.
Dopamine neurons mull over your options
Researchers at the University of Tsukuba have found that dopamine neurons in the brain can represent the decision-making process when making economic choices.
Neurons thrive even when malnourished
When animal, insect or human embryos grow in a malnourished environment, their developing nervous systems get first pick of any available nutrients so that new neurons can be made.
The first 3D map of the heart's neurons
An interdisciplinary research team establishes a new technological pipeline to build a 3D map of the neurons in the heart, revealing foundational insight into their role in heart attacks and other cardiac conditions.
Mapping the neurons of the rat heart in 3D
A team of researchers has developed a virtual 3D heart, digitally showcasing the heart's unique network of neurons for the first time.
How to put neurons into cages
Football-shaped microscale cages have been created using special laser technologies.
A molecule that directs neurons
A research team coordinated by the University of Trento studied a mass of brain cells, the habenula, linked to disorders like autism, schizophrenia and depression.
More Neurons News and Neurons Current Events

Trending Science News

Current Coronavirus (COVID-19) News

Top Science Podcasts

We have hand picked the top science podcasts of 2020.
Now Playing: TED Radio Hour

Warped Reality
False information on the internet makes it harder and harder to know what's true, and the consequences have been devastating. This hour, TED speakers explore ideas around technology and deception. Guests include law professor Danielle Citron, journalist Andrew Marantz, and computer scientist Joy Buolamwini.
Now Playing: Science for the People

#576 Science Communication in Creative Places
When you think of science communication, you might think of TED talks or museum talks or video talks, or... people giving lectures. It's a lot of people talking. But there's more to sci comm than that. This week host Bethany Brookshire talks to three people who have looked at science communication in places you might not expect it. We'll speak with Mauna Dasari, a graduate student at Notre Dame, about making mammals into a March Madness match. We'll talk with Sarah Garner, director of the Pathologists Assistant Program at Tulane University School of Medicine, who takes pathology instruction out of...
Now Playing: Radiolab

What If?
There's plenty of speculation about what Donald Trump might do in the wake of the election. Would he dispute the results if he loses? Would he simply refuse to leave office, or even try to use the military to maintain control? Last summer, Rosa Brooks got together a team of experts and political operatives from both sides of the aisle to ask a slightly different question. Rather than arguing about whether he'd do those things, they dug into what exactly would happen if he did. Part war game part choose your own adventure, Rosa's Transition Integrity Project doesn't give us any predictions, and it isn't a referendum on Trump. Instead, it's a deeply illuminating stress test on our laws, our institutions, and on the commitment to democracy written into the constitution. This episode was reported by Bethel Habte, with help from Tracie Hunte, and produced by Bethel Habte. Jeremy Bloom provided original music. Support Radiolab by becoming a member today at Radiolab.org/donate.     You can read The Transition Integrity Project's report here.