Disney Research leverages redundancy in casually shot videos to enable scene-space effects

August 05, 2015

The same sort of video processing effects that usually require video to be shot in controlled environments where 3-D positions of cameras and objects are precisely known can be achieved with real-world, handheld video shot from consumer-grade cameras using a new approach pioneered by Disney Research.

The technique, developed with Braunschweig University of Technology, compensates for the lack of exact 3-D information about a scene by taking advantage of the fact that most elements of a scene are visible many times in a video. The researchers found they could sample pixels of scene structure from multiple frames of a video and add a filtering process that compensates for inaccuracies in 3-D positions.

Using this method, which the researchers will describe at ACM SIGGRAPH 2015, the International Conference on Computer Graphics and Interactive Techniques, in Los Angeles Aug. 9-13, they showed they could eliminate flickering and other "noise" from videos, correct the blurring of lettering caused by camera movement, and remove objects from the foreground or background of scenes.

These and other effects were demonstrated using video recorded on an iPhone 5s, a GoPro Hero 3, a Canon point-and-shoot camera, or a Sony DSLR camera.

"We believe that our novel processing approach will enable new video applications that were previously impossible, limited, or could not be fully explored because of inevitably unreliable depth information," said Oliver Wang, research scientist at Disney Research.

Processing video based on the 3-D position of pixels - known as scene-space processing - has a number of advantages over traditional, 2-D "image-space" processing. Its utility will only increase as more hardware for recording 3-D information, such as depth-enabled smartphones and Kinect game controllers, reaches the mass market.

Even so, obtaining exact 3-D information will remain elusive for the foreseeable future for video recorded under uncontrolled conditions, said Felix Klose, a researcher at Disney and at UT Braunschweig.

To leverage the high degree of redundancy in most videos and compensate for 3-D inaccuracies, the Disney team developed an efficient method for collecting all of the video pixels sampled from multiple frames that conceivably might represent the same point in a scene. They then developed a filtering process that discards outliers from this sample and computes the output pixel color as a weighted combination of the collected samples.

This framework is compatible with parallel computer processing and, despite the large amount of data accessed, reasonable runtimes can be achieved using standard desktop computers.
In addition to Klose and Wang, the research team included Jean-Charles Bazin and Alexander Sorkine-Hornung of Disney Research and Marcus Magnor, a professor of computer science at UT Braunschweig.

For more information and a video, please visit the project web page at http://www.disneyresearch.com/publication/scenespace/.

About Disney Research

Disney Research is a network of research laboratories supporting The Walt Disney Company. Its purpose is to pursue scientific and technological innovation to advance the company's broad media and entertainment efforts. Vice Presidents Jessica Hodgins and Markus Gross manage Disney Research facilities in Los Angeles, Pittsburgh, Zürich, and Boston and work closely with the Pixar and ILM research groups in the San Francisco Bay Area. Research topics include computer graphics, animation, video processing, computer vision, robotics, wireless & mobile computing, human-computer interaction, displays, behavioral economics, and machine learning.

Disney Research

Related Achieved Articles from Brightsurf:

Scientists from NUST MISIS manage to improve metallic glasses
Researchers at National University of Science and Technology MISIS (NUST MISIS) have managed to develop a unique method to process bulk metallic glasses.

Scientists propose immune cocktail therapy to boost cancer-immunity cycle in multiple aspects
A research team led by Prof. TIAN Huayu from the Changchun Institute of Applied Chemistry (CIAC) of the Chinese Academy of Sciences proposed an innovative immune cocktail therapy that combined ICT along with other therapeutic approaches.

First gene knockout in a cephalopod is achieved at Marine Biological Laboratory
A team at the Marine Biological Laboratory (MBL) has achieved the first gene knockout in a cephalopod using the squid Doryteuthis pealeii, an exceptionally important research organism in biology for nearly a century.

Site-directed mutagenesis in wheat via haploid induction by maize
Site-directed mutagenesis facilitates the experimental validation of gene function and can speed up plant breeding by producing new biodiversity or by reproducing previously known gene variants in other than their original genetic backgrounds.

USTC finds ultimate precision limit of multi-parameter quantum magnetometry
Researchers obtained the ultimate precision for the estimation of all three components of a magnetic field with entangled probe states under the parallel scheme.

Modeling population differences influences the herd immunity threshold for COVID-19
A new modeling study illustrates how accounting for factors such as age and social activity influences the predicted herd immunity threshold for COVID-19, or the level of population immunity needed to stop the disease's transmission.

BIOCAD presents results of clinical trial proving efficacy of netakimab at EULAR 2020
BIOCAD presents results of clinical trial proving efficacy of netakimab in the treatment of psoriatic arthritis at the Annual European Congress of Rheumatology (EULAR 2020).

Sustainable light achieved in living plants
This week in Nature Biotechnology, scientists have announced the feasibility of creating plants that produce their own visible luminescence.

Higher earning 'elite' political lobbyists overstate their own achievements, study shows
'Elite', high-earning political lobbyists are more likely to overstate their achievements, a new study shows.

Large meta-analysis links IVF to higher gestational diabetes risk
Women who give birth to singleton babies following assisted reproductive technologies including vitro fertilisation (IVF) are at greater risk of developing gestational diabetes than those who conceive naturally, according to a meta-analysis involving over almost 2 million singleton pregnancies.

Read More: Achieved News and Achieved Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.