Nav: Home

Streamlining mobile image processing

November 13, 2015

As smartphones become people's primary computers and their primary cameras, there is growing demand for mobile versions of image-processing applications.

Image processing, however, can be computationally intensive and could quickly drain a cellphone's battery. Some mobile applications try to solve this problem by sending image files to a central server, which processes the images and sends them back. But with large images, this introduces significant delays and could incur costs for increased data usage.

At the Siggraph Asia conference last week, researchers from MIT, Stanford University, and Adobe Systems presented a system that, in experiments, reduced the bandwidth consumed by server-based image processing by as much as 98.5 percent, and the power consumption by as much as 85 percent.

The system sends the server a highly compressed version of an image, and the server sends back an even smaller file, which contains simple instructions for modifying the original image.

Michaël Gharbi, a graduate student in electrical engineering and computer science at MIT and first author on the Siggraph paper, says that the technique could become more useful as image-processing algorithms become more sophisticated.

"We see more and more new algorithms that leverage large databases to take a decision on the pixel," Gharbi says. "These kinds of algorithm don't do a very complex transform if you go to a local scale on the image, but they still require a lot of computation and access to the data. So that's the kind of operation you would need to do on the cloud."

One example, Gharbi says, is recent work at MIT that transfers the visual styles of famous portrait photographers to cellphone snapshots. Other researchers, he says, have experimented with algorithms for changing the apparent time of day at which photos were taken.

Joining Gharbi on the new paper are his thesis advisor, Frédo Durand, a professor of computer science and engineering; YiChang Shih, who received his PhD in electrical engineering and computer science from MIT in March; Gaurav Chaurasia, a former postdoc in Durand's group who's now at Disney Research; Jonathan Ragan-Kelley, who has been a postdoc at Stanford since graduating from MIT in 2014; and Sylvain Paris, who was a postdoc with Durand before joining Adobe.

Bring the noise

The researchers' system works with any alteration to the style of an image, like the types of "filters" popular on Instagram. It's less effective with edits that change the image content -- deleting a figure and then filling in the background, for instance.

To save bandwidth while uploading a file, the researchers' system simply sends it as a very low-quality JPEG, the most common file format for digital images. All the cleverness is in the way the server processes the image.

The transmitted JPEG has a much lower resolution than the source image, which could lead to problems. A single reddish pixel in the JPEG, for instance, could stand in for a patch of pixels that in fact depict a subtle texture of red and purple bands. So the first thing the system does is introduce some high-frequency noise into the image, which effectively increases its resolution.

That extra resolution is basically meaningless -- just some small, random, local variation of the pixel color in the compressed file. But it prevents the system from relying too heavily on color consistency in particular regions of the image when determining how to characterize its image transformations.

Patch work

Next, the system performs the desired manipulation of the image -- heightening contrast, shifting the color spectrum, sharpening edges, or the like.

Then the system breaks the image into chunks -- of, say, 64 by 64 pixels. For each chunk, it uses a machine-learning algorithm to characterize the effects of the manipulation according to a few basic parameters, most of which concern variations in the luminance, or brightness, of the pixels in the patch. The researchers' best results came when they used about 25 parameters. So for each 64-by-64-pixel patch of the uploaded image, each pixel of which could have one of three values, the server sends back just 25 numbers.

The phone then performs the modifications described by those 25 numbers on its local, high-resolution copy of the image. To the naked eye, the results are virtually indistinguishable from direct manipulation of the high-resolution image. The bandwidth consumption, however, is only 1 to 2 percent of what it would have been.

Applying the modifications to the original image does require some extra computation on the phone, but that consumes neither as much time nor as much energy as uploading and downloading high-resolution files would. In the researchers' experiments, the energy savings were generally between 50 and 85 percent, and the time savings between 50 and 70 percent.
Additional background

ARCHIVE: Removing reflections from photos taken through windows

ARCHIVE: Spruce up your selfie

Massachusetts Institute of Technology

Related Engineering Articles:

Next frontier in bacterial engineering
A new technique overcomes a serious hurdle in the field of bacterial design and engineering.
COVID-19 and the role of tissue engineering
Tissue engineering has a unique set of tools and technologies for developing preventive strategies, diagnostics, and treatments that can play an important role during the ongoing COVID-19 pandemic.
Engineering the meniscus
Damage to the meniscus is common, but there remains an unmet need for improved restorative therapies that can overcome poor healing in the avascular regions.
Artificially engineering the intestine
Short bowel syndrome is a debilitating condition with few treatment options, and these treatments have limited efficacy.
Reverse engineering the fireworks of life
An interdisciplinary team of Princeton researchers has successfully reverse engineered the components and sequence of events that lead to microtubule branching.
New method for engineering metabolic pathways
Two approaches provide a faster way to create enzymes and analyze their reactions, leading to the design of more complex molecules.
Engineering for high-speed devices
A research team from the University of Delaware has developed cutting-edge technology for photonics devices that could enable faster communications between phones and computers.
Breakthrough in blood vessel engineering
Growing functional blood vessel networks is no easy task. Previously, other groups have made networks that span millimeters in size.
Next-gen batteries possible with new engineering approach
Dramatically longer-lasting, faster-charging and safer lithium metal batteries may be possible, according to Penn State research, recently published in Nature Energy.
What can snakes teach us about engineering friction?
If you want to know how to make a sneaker with better traction, just ask a snake.
More Engineering News and Engineering Current Events

Trending Science News

Current Coronavirus (COVID-19) News

Top Science Podcasts

We have hand picked the top science podcasts of 2020.
Now Playing: TED Radio Hour

Listen Again: Meditations on Loneliness
Original broadcast date: April 24, 2020. We're a social species now living in isolation. But loneliness was a problem well before this era of social distancing. This hour, TED speakers explore how we can live and make peace with loneliness. Guests on the show include author and illustrator Jonny Sun, psychologist Susan Pinker, architect Grace Kim, and writer Suleika Jaouad.
Now Playing: Science for the People

#565 The Great Wide Indoors
We're all spending a bit more time indoors this summer than we probably figured. But did you ever stop to think about why the places we live and work as designed the way they are? And how they could be designed better? We're talking with Emily Anthes about her new book "The Great Indoors: The Surprising Science of how Buildings Shape our Behavior, Health and Happiness".
Now Playing: Radiolab

The Third. A TED Talk.
Jad gives a TED talk about his life as a journalist and how Radiolab has evolved over the years. Here's how TED described it:How do you end a story? Host of Radiolab Jad Abumrad tells how his search for an answer led him home to the mountains of Tennessee, where he met an unexpected teacher: Dolly Parton.Jad Nicholas Abumrad is a Lebanese-American radio host, composer and producer. He is the founder of the syndicated public radio program Radiolab, which is broadcast on over 600 radio stations nationwide and is downloaded more than 120 million times a year as a podcast. He also created More Perfect, a podcast that tells the stories behind the Supreme Court's most famous decisions. And most recently, Dolly Parton's America, a nine-episode podcast exploring the life and times of the iconic country music star. Abumrad has received three Peabody Awards and was named a MacArthur Fellow in 2011.