Amplifying -- or removing -- visual variation

November 06, 2015

At the Siggraph Asia conference this week, MIT researchers presented a pair of papers describing techniques for either magnifying or smoothing out small variations in digital images.

The techniques could be used to produce more polished images for graphic-design projects, or, applied in the opposite direction, they could disclose structural defects, camouflaged objects, or movements invisible to the naked eye that could be of scientific interest.

Conceptually, the work builds on a long line of research from several groups in MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL), which sought to amplify minute motions in digital video. "In motion magnification, the deviations are over time, and the model is deviation from being perfectly static," says Tali Dekel, a postdoc in CSAIL and a co-author on both papers. "Our method takes as input only a single image, and it looks for deviation in space. We don't need to know time history to do that."

One of the two papers, on which Dekel is first author, presents an algorithm that looks for repeated forms within an image, such as the kernels of an ear of corn or bricks in a wall. It can then iron out differences across the image, producing idealized but still natural-looking corn ears or brick walls, or amplify the differences, making them more evident to the naked eye.

The algorithm works with color as well as shape. So, for instance, it can take an image in which a chameleon is concealed against the trunk of a tree and enhance subtle color differences so that the chameleon stands out blue against an orange background.

Joining Dekel on that paper are professor of computer science and engineering William Freeman, whose group she's a member of, and colleagues from Israel's Technion and Weizmann Institute.

Imperfect form

The algorithm described in the other paper amplifies deviations from ideal geometries. The roofline of a house, for instance, could look perfectly straight to the naked eye but still sag toward the middle. The algorithm can exaggerate that type of flaw, potentially drawing attention to structural problems.

In other experiments, the algorithm was able to identify a rippling in Saturn's rings that could offer information about the orbital pattern of the planet's moons, and by magnifying changes to a regular pattern projected on a screen behind a candle, it revealed thermal variations caused by the candle's flame.

The first author on that paper is Neal Wadhwa, another MIT graduate student in electrical engineering and computer science. Joining him are Dekel, Freeman, graduate student Donglai Wei, and Frédo Durand, a professor of computer science and engineering.

The first algorithm -- the one that recognizes repeated forms -- begins by comparing patches of the source image, at different scales, and identifying those that seem to be visually similar. Then it averages out all the visually similar patches and uses the averages to construct a new, highly regular version of the image. This image may look unnatural, but its purpose is just to serve as an initial target.

Then the algorithm identifies a mathematical function that moves the pixels of the source image around, producing the best possible approximation of the target image. From that function, it creates a new target image. It then iterates back and forth, producing ever more natural-looking target images and ever more regular mathematical transformations, until the two converge.

Once the algorithm has a function that produces a regular image, it can simply invert it to produce a more distorted image.


The technique works not only with geometrically simple forms like corn kernels and bricks but with more complex forms as well. So, for instance, it can take an image of a line of dancers executing the same kick and standardize their heights and the distances between them.

If it's implemented particularly aggressively, it can even cut irregularities out of an image -- for instance, standardizing the size and shape of the cells of a honeycomb while deleting the bees crawling over it. As such, it could be a useful resource for image-manipulation programs like Photoshop.

In materials science, a standard technique for identifying defects in a material's surface is to cover it with tiny soap bubbles and look for irregularities. The MIT researchers are also collaborating with materials scientists to use their algorithm to enhance that process.

The second algorithm uses existing techniques to identify the geometric shapes indicated by color gradations in an image. Then it excises a narrow band of the image that traces the curve defining each of those shapes. It then straightens the bands out, creating a uniform representation of all the shapes in the image.

At regular intervals, it considers local variations in color across the width of each band. These will typically vary, indicating deviations from the idealized geometry of the initial curve. From those deviations, the algorithm constructs a new, more erratic curve, which it can exaggerate and then reinsert into the image.
Additional background

ARCHIVE: Amplifying small motions in large motions

ARCHIVE: Gauging materials' physical properties from video

ARCHIVE: Magnifying vibrations in bridges and buildings

ARCHIVE: Extracting audio from visual information

Massachusetts Institute of Technology

Related Engineering Articles from Brightsurf:

Re-engineering antibodies for COVID-19
Catholic University of America researcher uses 'in silico' analysis to fast-track passive immunity

Next frontier in bacterial engineering
A new technique overcomes a serious hurdle in the field of bacterial design and engineering.

COVID-19 and the role of tissue engineering
Tissue engineering has a unique set of tools and technologies for developing preventive strategies, diagnostics, and treatments that can play an important role during the ongoing COVID-19 pandemic.

Engineering the meniscus
Damage to the meniscus is common, but there remains an unmet need for improved restorative therapies that can overcome poor healing in the avascular regions.

Artificially engineering the intestine
Short bowel syndrome is a debilitating condition with few treatment options, and these treatments have limited efficacy.

Reverse engineering the fireworks of life
An interdisciplinary team of Princeton researchers has successfully reverse engineered the components and sequence of events that lead to microtubule branching.

New method for engineering metabolic pathways
Two approaches provide a faster way to create enzymes and analyze their reactions, leading to the design of more complex molecules.

Engineering for high-speed devices
A research team from the University of Delaware has developed cutting-edge technology for photonics devices that could enable faster communications between phones and computers.

Breakthrough in blood vessel engineering
Growing functional blood vessel networks is no easy task. Previously, other groups have made networks that span millimeters in size.

Next-gen batteries possible with new engineering approach
Dramatically longer-lasting, faster-charging and safer lithium metal batteries may be possible, according to Penn State research, recently published in Nature Energy.

Read More: Engineering News and Engineering Current Events is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to