Nav: Home

Google's new light field video research showcases high-quality experience

June 18, 2020

Google is taking immersive media technology to the next level, showing a practical system for light field video. Wide field of view scenes can be recorded and played back with the ability to move around within the video after it has been captured, revealing new perspectives. Developed by a team of leading research scientists and engineers, the new research shows the ability to record, reconstruct, compress, and deliver high-quality immersive light field videos lightweight enough to be streamed over regular Wi-Fi, advancing the state of the art in the rapidly emerging field of immersive augmented reality (AR) and virtual reality (VR) platforms.

In recent years, the immersive AR/VR field has captured mainstream attention for its promise to give people a truly authentic experience in a simulated environment. Want to really feel like you're standing among the Redwoods at Yosemite rather than sitting in the living room? Or watch an artist create a sculpture as if you're with them in the studio? That could be possible with immersive AR/VR technology.

Although the field is still nascent, the team at Google has addressed important challenges, making major research headway in immersive light field video. The research team, led by Michael Broxton, Google research scientist, and Paul Debevec, Google senior staff engineer, plans to demonstrate the new system at SIGGRAPH 2020. The conference, which will take place virtually this year, brings together a wide variety of professionals who approach computer graphics and interactive techniques from different perspectives and continues to serve as the industry's premier venue for showcasing forward-thinking ideas and research.

"This is the latest culmination of our work in light fields," Broxton, a lead author of the research, says. "We're making this technology practical, bringing us closer to delivering a truly immersive experience to more consumer devices. Photos and videos play a huge role in our day-to-day experience on mobile devices, and we are hoping that someday immersive light field images and videos will play an equally important role in future AR and VR platforms."

At SIGGRAPH 2018, Google researchers showcased similar work when they introduced photorealistic light field still images in immersive VR. This new system has added another key piece to the immersive media puzzle: video. Light field videos give users a more dynamic virtual environment with panoramic views of scenes that span more than 180 degrees. They allow users to peek around corners and enjoy a greater sense of depth while in the virtual world. And the system is able to capture content that was challenging for earlier methods, such as reflective surfaces. This translates into a more realistic environment; for instance, sunlight that naturally reflects on ocean waves or light reflecting off the shiny hood of a car shifts naturally with the user's gaze as it would in real life.

The team records immersive light field videos with a low-cost rig consisting of 46 action sports cameras mounted to a lightweight acrylic dome. Using DeepView, a machine learning algorithm developed last year by members of the same Google research team, they combine the video streams from each camera into a single 3D representation of the scene being recorded. Their paper introduces a new "layered mesh" representation that consists of a series of concentric layers with semi-transparent textures. Rendering these layers from back to front brings the scene vividly and realistically to life. This method solves the very difficult problem of synthesizing viewpoints that were never captured by the cameras in the first place, enabling the user to experience a natural range of head movement as they explore light field video content.

Another breakthrough in this work involves data compression. The idea is not only to develop a system capable of reconstructing video for a truly immersive AR/VR experience but also to access the experience via consumer AR and VR headsets and displays, and even in a web browser. The new system compresses light field video while still preserving its original visual quality, and it does so using conventional texture atlasing and widely supported video codecs. In essence, they have succeeded at bootstrapping a next generation media format off of today's image and video compression techniques.

"Users will be able to stream this light field video content over a typical, fast-speed internet connection," Broxton says. "Overcoming this problem opens up this technology to a much wider audience."

Debevec sums up the work, stating, "Completing this project feels like we've overcome a major obstacle in making virtual experiences realistic, immersive, distributable, and comfortable. I can't wait to have the experiences the creative AR and VR community will make with this."
The Google research team on "Immersive Light Field Video With a Layered Mesh Representation" includes John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew DuVall, Jason Dourgarian, Jay Busch, and Matt Whalen.

Association for Computing Machinery

Related Technology Articles:

October issue SLAS Technology now available
The October issue of SLAS Technology features the cover article, 'Role of Digital Microfl-uidics in Enabling Access to Laboratory Automation and Making Biology Programmable' by Varun B.
Robot technology for everyone or only for the average person?
Robot technology is being used more and more in health rehabilitation and in working life.
Novel biomarker technology for cancer diagnostics
A new way of identifying cancer biomarkers has been developed by researchers at Lund University in Sweden.
Technology innovation for neurology
TU Graz researcher Francesco Greco has developed ultra-light tattoo electrodes that are hardly noticeable on the skin and make long-term measurements of brain activity cheaper and easier.
April's SLAS Technology is now available
April's Edition of SLAS Technology Features Cover Article, 'CURATE.AI: Optimizing Personalized Medicine with Artificial Intelligence'.
Technology in higher education: learning with it instead of from it
Technology has shifted the way that professors teach students in higher education.
Post-lithium technology
Next-generation batteries will probably see the replacement of lithium ions by more abundant and environmentally benign alkali metal or multivalent ions.
Rethinking the role of technology in the classroom
Introducing tablets and laptops to the classroom has certain educational virtues, according to Annahita Ball, an assistant professor in the University at Buffalo School of Social Work, but her research suggests that tech has its limitations as well.
The science and technology of FAST
The Five hundred-meter Aperture Spherical radio Telescope (FAST), located in a radio quiet zone, with the targets (e.g., radio pulsars and neutron stars, galactic and extragalactic 21-cm HI emission).
AI technology could help protect water supplies
Progress on new artificial intelligence (AI) technology could make monitoring at water treatment plants cheaper and easier and help safeguard public health.
More Technology News and Technology Current Events

Trending Science News

Current Coronavirus (COVID-19) News

Top Science Podcasts

We have hand picked the top science podcasts of 2020.
Now Playing: TED Radio Hour

Listen Again: The Power Of Spaces
How do spaces shape the human experience? In what ways do our rooms, homes, and buildings give us meaning and purpose? This hour, TED speakers explore the power of the spaces we make and inhabit. Guests include architect Michael Murphy, musician David Byrne, artist Es Devlin, and architect Siamak Hariri.
Now Playing: Science for the People

#576 Science Communication in Creative Places
When you think of science communication, you might think of TED talks or museum talks or video talks, or... people giving lectures. It's a lot of people talking. But there's more to sci comm than that. This week host Bethany Brookshire talks to three people who have looked at science communication in places you might not expect it. We'll speak with Mauna Dasari, a graduate student at Notre Dame, about making mammals into a March Madness match. We'll talk with Sarah Garner, director of the Pathologists Assistant Program at Tulane University School of Medicine, who takes pathology instruction out of...
Now Playing: Radiolab

What If?
There's plenty of speculation about what Donald Trump might do in the wake of the election. Would he dispute the results if he loses? Would he simply refuse to leave office, or even try to use the military to maintain control? Last summer, Rosa Brooks got together a team of experts and political operatives from both sides of the aisle to ask a slightly different question. Rather than arguing about whether he'd do those things, they dug into what exactly would happen if he did. Part war game part choose your own adventure, Rosa's Transition Integrity Project doesn't give us any predictions, and it isn't a referendum on Trump. Instead, it's a deeply illuminating stress test on our laws, our institutions, and on the commitment to democracy written into the constitution. This episode was reported by Bethel Habte, with help from Tracie Hunte, and produced by Bethel Habte. Jeremy Bloom provided original music. Support Radiolab by becoming a member today at     You can read The Transition Integrity Project's report here.