Green-screen keying method cuts time, boosts quality in film compositing

November 14, 2016

Filming an actor in front of a green screen and then superimposing the actor over another background is commonplace in feature film production, but getting rid of all traces of the green screen remains a chore. A new "keying" method devised by Disney Research and ETH Zurich, however, improves the results substantially.

What's more, these superior results can be achieved in about one-tenth the time it takes a professional compositor to perform this task manually.

"Keying is a crucial intermediate step in adding synthetic backgrounds and objects seamlessly into live-action shots," said Markus Gross, vice president at Disney Research. "It's also a major bottleneck in postproduction, so a keying method that is both fast and produces high-quality results will have a major impact on the production of feature films."

A report on the method, which is based on a novel algorithm that computes underlying colors and their mixing ratios, will be presented at the Association for Computing Machinery's (ACM) Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 2017) next summer in Los Angeles and has been published in the August issue of ACM Transactions on Graphics journal (ACM TOG).

A number of commercial software tools are available to help a compositing artist perform keying, said Tunç Aydin, research scientist at Disney Research. But those tools have difficulty discerning the green screen background from the foreground in regions of the image where the colors of objects mix with each other, he noted. This mixing can occur because of motion blur, intricate object boundaries, such as hair, or colors cast onto an object from the green screen itself. Whatever the cause, these imperfections usually force the artist to painstakingly paint those regions manually.

The research team opted to create a color unmixing algorithm, which can be used to extract multiple color layers using a color model of the image, said Yagiz Aksoy, a Ph.D. student at Disney Research and ETH Zurich. A subset of these colors is then combined to produce the final keying result.

The team created a user interface that enables the artist to create a color model of the image simply by scribbling over each distinct-colored area of the image. The shape of the scribble doesn't matter; the program simply needs to identify each area of color so it can create a representation of all the colors in the scene. The algorithm computes the underlying colors and their mixing ratios for each area.

The researchers showed that the method would be applicable for keying scenes with non-green-screen backgrounds. Also, the ability to represent the image with multiple color layers, rather than just foreground and background layers as in conventional methods, creates opportunities for new applications, Aydin said. Colors of objects in a scene, for instance, could be changed.

In comprehensive comparisons with commercial keying tools and with other published methods, the researchers showed that the results for their method were superior in quality as it required only one-tenth of the manual editing time it would take a professional artist to process the same content.

Combining creativity and innovation, this research continues Disney's rich legacy of inventing new ways to tell great stories and leveraging technology required to build the future of entertainment.
-end-
In addition to Aksoy and Aydin, the research team included Marc Pollefeys, professor of computer science at ETH Zurich, and Aljoša Smolic, a former senior research scientist at Disney Research, currently a professor at Trinity College in Dublin.

For more information and a video, visit the project web site at https://www.disneyresearch.com/publication/green-screen/.

About Disney Research

Disney Research is a network of research laboratories supporting The Walt Disney Company. Its purpose is to pursue scientific and technological innovation to advance the company's broad media and entertainment efforts. Vice Presidents Jessica Hodgins and Markus Gross manage Disney Research facilities in Los Angeles, Pittsburgh and Zürich, and work closely with the Pixar and ILM research groups in the San Francisco Bay Area. Research topics include computer graphics, animation, video processing, computer vision, robotics, wireless & mobile computing, human-computer interaction, displays, behavioral economics, and machine learning.

Website: http://www.disneyresearch.com
Twitter: @DisneyResearch
Facebook: http://www.facebook.com/DisneyResearch

Disney Research

Related Algorithm Articles from Brightsurf:

CCNY & partners in quantum algorithm breakthrough
Researchers led by City College of New York physicist Pouyan Ghaemi report the development of a quantum algorithm with the potential to study a class of many-electron quantums system using quantum computers.

Machine learning algorithm could provide Soldiers feedback
A new machine learning algorithm, developed with Army funding, can isolate patterns in brain signals that relate to a specific behavior and then decode it, potentially providing Soldiers with behavioral-based feedback.

New algorithm predicts likelihood of acute kidney injury
In a recent study, a new algorithm outperformed the standard method for predicting which hospitalized patients will develop acute kidney injury.

New algorithm could unleash the power of quantum computers
A new algorithm that fast forwards simulations could bring greater use ability to current and near-term quantum computers, opening the way for applications to run past strict time limits that hamper many quantum calculations.

QUT algorithm could quash Twitter abuse of women
Online abuse targeting women, including threats of harm or sexual violence, has proliferated across all social media platforms but QUT researchers have developed a sophisticated statistical model to identify misogynistic content and help drum it out of the Twittersphere.

New learning algorithm should significantly expand the possible applications of AI
The e-prop learning method developed at Graz University of Technology forms the basis for drastically more energy-efficient hardware implementations of Artificial Intelligence.

Algorithm predicts risk for PTSD after traumatic injury
With high precision, a new algorithm predicts which patients treated for traumatic injuries in the emergency department will later develop posttraumatic stress disorder.

New algorithm uses artificial intelligence to help manage type 1 diabetes
Researchers and physicians at Oregon Health & Science University have designed a method to help people with type 1 diabetes better manage their glucose levels.

A new algorithm predicts the difficulty in fighting fire
The tool completes previous studies with new variables and could improve the ability to respond to forest fires.

New algorithm predicts optimal materials among all possible compounds
Skoltech researchers have offered a solution to the problem of searching for materials with required properties among all possible combinations of chemical elements.

Read More: Algorithm News and Algorithm Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.