System automatically converts 2-D video to 3-D

November 04, 2015

By exploiting the graphics-rendering software that powers sports video games, researchers at MIT and the Qatar Computing Research Institute (QCRI) have developed a system that automatically converts 2-D video of soccer games into 3-D.

The converted video can be played back over any 3-D device -- a commercial 3-D TV, Google's new Cardboard system, which turns smartphones into 3-D displays, or special-purpose displays such as Oculus Rift.

The researchers presented the new system last week at the Association for Computing Machinery's Multimedia conference.

"Any TV these days is capable of 3-D," says Wojciech Matusik, an associate professor of electrical engineering and computer science at MIT and one of the system's co-developers. "There's just no content. So we see that the production of high-quality content is the main thing that should happen. But sports is very hard. With movies, you have artists who paint the depth map. Here, there is no luxury of hiring 100 artists to do the conversion. This has to happen in real-time."

The system is one result of a collaboration between QCRI and MIT's Computer Science and Artificial Intelligence Laboratory. Joining Matusik on the conference paper are Kiana Calagari, a research associate at QCRI and first author; Alexandre Kaspar, an MIT graduate student in electrical engineering and computer science; Piotr Didyk, who was a postdoc in Matusik's group and is now a researcher at the Max Planck Institute for Informatics; Mohamed Hefeeda, a principal scientist at QCRI; and Mohamed Elgharib, a QCRI postdoc. QCRI also helped fund the project.

Zeroing in

In the past, researchers have tried to develop general-purpose systems for converting 2-D video to 3-D, but they haven't worked very well and have tended to produce odd visual artifacts that detract from the viewing experience.

"Our advantage is that we can develop it for a very specific problem domain," Matusik says. "We are developing a conversion pipeline for a specific sport. We would like to do it at broadcast quality, and we would like to do it in real-time. What we have noticed is that we can leverage video games."

Today's video games generally store very detailed 3-D maps of the virtual environment that the player is navigating. When the player initiates a move, the game adjusts the map accordingly and, on the fly, generates a 2-D projection of the 3-D scene that corresponds to a particular viewing angle.

The MIT and QCRI researchers essentially ran this process in reverse. They set the very realistic Microsoft soccer game "FIFA13" to play over and over again, and used Microsoft's video-game analysis tool PIX to continuously store screen shots of the action. For each screen shot, they also extracted the corresponding 3-D map.

Using a standard algorithm for gauging the difference between two images, they winnowed out most of the screen shots, keeping just those that best captured the range of possible viewing angles and player configurations that the game presented; the total number of screen shots still ran to the tens of thousands. Then they stored each screen shot and the associated 3-D map in a database.

Jigsaw puzzle

For every frame of 2-D video of an actual soccer game, the system looks for the 10 or so screen shots in the database that best correspond to it. Then it decomposes all those images, looking for the best matches between smaller regions of the video feed and smaller regions of the screen shots. Once it's found those matches, it superimposes the depth information from the screen shots on the corresponding sections of the video feed. Finally, it stitches the pieces back together.

The result is a very convincing 3-D effect, with no visual artifacts. The researchers conducted a user study in which the majority of subjects gave the 3-D effect a rating of 5 ("excellent") on a five-point ("bad" to "excellent") scale; the average score was between 4 ("good") and 5.

Currently, the researchers say, the system takes about a third of a second to process a frame of video. But successive frames could all be processed in parallel, so that the third-of-a-second delay needs to be incurred only once. A broadcast delay of a second or two would probably provide an adequate buffer to permit conversion on the fly. Even so, the researchers are working to bring the conversion time down still further.
Additional background

ARCHIVE: Customizing 3-D printing

ARCHIVE: Graphics in reverse

ARCHIVE: Glasses-free 3-D projector

Massachusetts Institute of Technology

Related Electrical Engineering Articles from Brightsurf:

Knotting semimetals in topological electrical circuits
Scientists created exotic states of matter using electrical circuit enhanced by machine-learning algorithm

Physicists make electrical nanolasers even smaller
Researchers cleared the obstacle that had prevented the creation of electrically driven nanolasers for integrated circuits.

Making plastic more transparent while also adding electrical conductivity
In an effort to improve large touchscreens, LED light panels and window-mounted infrared solar cells, researchers at the University of Michigan have made plastic conductive while also making it more transparent.

Using electrical stimulus to regulate genes
A team of researchers led by ETH professor Martin Fussenegger has succeeded in using an electric current to directly control gene expression for the first time.

2D oxide flakes pick up surprise electrical properties
Rice University researchers find evidence of piezoelectricity in lab-grown, two-dimensional flakes of molybdenum dioxide.

Electrical activity in living organisms mirrors electrical fields in atmosphere
A new Tel Aviv University study provides evidence for a direct link between electrical fields in the atmosphere and those found in living organisms, including humans.

3D-printed plastics with high performance electrical circuits
Rutgers engineers have embedded high performance electrical circuits inside 3D-printed plastics, which could lead to smaller and versatile drones and better-performing small satellites, biomedical implants and smart structures.

In and out with 10-minute electrical vehicle recharge
Electric vehicle owners may soon be able to pull into a fueling station, plug their car in, go to the restroom, get a cup of coffee and in 10 minutes, drive out with a fully charged battery, according to a team of engineers.

Electrical stimulation aids in spinal fusion
Spine surgeons in the U.S. perform more than 400,000 spinal fusions each year as a way to ease back pain and prevent vertebrae in the spine from wiggling around and doing more damage.

Fat pumps generate electrical power
A previously unknown electrical current develops in the body's cells when the vital fat pump function of the flippases transfers ('flips') lipids from the outer to the inner layer of the body's cell membranes.

Read More: Electrical Engineering News and Electrical Engineering Current Events is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to