Nav: Home

Behind the magic: Making moving photos a reality

June 11, 2019

People moving in and out of photographs used to be reserved for the world of Harry Potter. But now computer scientists at the University of Washington have brought that magic to real life.

Their algorithm, Photo Wake-Up, can take a person from a 2D photo or a work of art and make them run, walk or jump out of the frame. The system also allows users to view the animation in three dimensions using augmented reality tools. The researchers will be presenting their results June 19 at the Conference on Computer Vision and Pattern Recognition in Long Beach, California. This research first attracted media attention when it was posted in preprint form in December on ArXiv.

"This is a very hard fundamental problem in computer vision," said co-author Ira Kemelmacher-Shlizerman, an associate professor at the UW's Paul G. Allen School of Computer Science & Engineering. "The big challenge here is that the input is only from a single camera position, so part of the person is invisible. Our work combines technical advancement on an open problem in the field with artistic creative visualization."

Previously, researchers thought it would be impossible to animate a person running out of a single photo.

"There is some previous work that tries to create a 3D character using multiple viewpoints," said co-author Brian Curless, a professor in the Allen School. "But you still couldn't bring someone to life and have them run out of a scene, and you couldn't bring AR into it. It was really surprising that we could get some compelling results with using just one photo."

The applications of Photo Wake-Up are numerous, the team says. The researchers envision this could lead to a new way for gamers to create avatars that actually look like them, a method for visitors to interact with paintings in an art museum -- say sitting down to have tea with Mona Lisa -- or something that lets children to bring their drawings to life. Examples in the research paper include animating the Golden State Warriors' Stephen Curry to run off the court, Paul McCartney to leap off the cover of the "Help!" album and Matisse's "Icarus" to leave his frame.

To make the magic a reality, Photo Wake-Up starts by identifying a person in an image and making a mask of the body's outline. From there, it matches a 3D template to the subject's body position. Then the algorithm does something surprising: In order to warp the template so that it actually looks like the person in the photo, it projects the 3D person back into 2D.

"It's very hard to manipulate in 3D precisely," said co-author Chung-Yi Weng, a doctoral student in the Allen School. "Maybe you can do it roughly, but any error will be obvious when you animate the character. So we have to find a way to handle things perfectly, and it's easier to do this in 2D."

Photo Wake-Up stores 3D information for each pixel: its distance from the camera or artist and how a person's joints are connected together. Once the template has been warped to match the person's shape, the algorithm pastes on the texture -- the colors from the image. It also generates the back of the person by using information from the image and the 3D template. Then the tool stitches the two sides together to make a 3D person who will be able to turn around.

Once the 3D character is ready to run, the algorithm needs to set up the background so that the character doesn't leave a blank space behind. Photo Wake-Up fills in the hole behind the person by borrowing information from other parts of the image.



Right now Photo Wake-Up works best with images of people facing forward, and can animate both artistic creations and photographs of real people. The algorithm can also handle some photos where people's arms are blocking part of their bodies, but it is not yet capable of animating people who have their legs crossed or who are blocking large parts of themselves.

"Photo Wake-Up is a new way to interact with photos," Weng said. "It can't do everything yet, but this is just the beginning."
-end-
This research was funded by the National Science Foundation, UW Animation Research, UW Reality Lab, Facebook, Huawei and Google.

For more information, contact Weng at chungyi@cs.washington.edu, Kemelmacher-Shlizerman at kemelmi@cs.washington.edu or Curless at curless@cs.washington.edu.

Grant number: #VEC1538618

Video on YouTube: https://www.youtube.com/watch?v=G63goXc5MyUSarah McQuate
University of Washington
206-543-2580
smcquate@uw.edu

University of Washington

Related Algorithm Articles:

Algorithm personalizes which cancer mutations are best targets for immunotherapy
As tumor cells multiply, they often spawn tens of thousands of genetic mutations.
Universal algorithm set to boost microscopes
EPFL scientists have developed an algorithm that can determine whether a super-resolution microscope is operating at maximum resolution based on a single image.
Algorithm designed to map universe, solve mysteries
Cornell University researchers have developed an algorithm designed to visualize models of the universe in order to solve some of physics' greatest mysteries.
Algorithm tells robots where nearby humans are headed
A new tool for predicting a person's movement trajectory may help humans and robots work together in close proximity.
Algorithm to transform investment banking with higher returns
A University of Bath researcher has created an algorithm which aims to remove the elements of chance, bias or emotion from investment banking decisions, a development which has the potential to reduce errors in financial decision making and improve financial returns in global markets.
Algorithm provides customized caffeine strategy for alertness
A web-based caffeine optimization tool successfully designs effective strategies to maximize alertness while avoiding excessive caffeine consumption, according to preliminary results from a new study.
New algorithm optimizes quantum computing problem-solving
Tohoku University researchers have developed an algorithm that enhances the ability of a Canadian-designed quantum computer to more efficiently find the best solution for complicated problems, according to a study published in the journal Scientific Reports.
Machine learning algorithm helps in the search for new drugs
Researchers have designed a machine learning algorithm for drug discovery which has been shown to be twice as efficient as the industry standard, which could accelerate the process of developing new treatments for disease.
Researchers create algorithm to predict PEDV outbreaks
Researchers from North Carolina State University have developed an algorithm that could give pig farms advance notice of porcine epidemic diarrhea virus (PEDV) outbreaks.
New algorithm provides a more detailed look at urban heat islands
Urban areas are warmer than the adjacent undeveloped land, a phenomenon known as the urban heat island effect.

Best Science Podcasts 2019

We have hand picked the best science podcasts for 2019. Sit back and enjoy new science podcasts updated daily from your favorite science news services and scientists.
Now Playing: TED Radio Hour

Rethinking Anger
Anger is universal and complex: it can be quiet, festering, justified, vengeful, and destructive. This hour, TED speakers explore the many sides of anger, why we need it, and who's allowed to feel it. Guests include psychologists Ryan Martin and Russell Kolts, writer Soraya Chemaly, former talk radio host Lisa Fritsch, and business professor Dan Moshavi.
Now Playing: Science for the People

#538 Nobels and Astrophysics
This week we start with this year's physics Nobel Prize awarded to Jim Peebles, Michel Mayor, and Didier Queloz and finish with a discussion of the Nobel Prizes as a way to award and highlight important science. Are they still relevant? When science breakthroughs are built on the backs of hundreds -- and sometimes thousands -- of people's hard work, how do you pick just three to highlight? Join host Rachelle Saunders and astrophysicist, author, and science communicator Ethan Siegel for their chat about astrophysics and Nobel Prizes.