"X-ray vision," once the province of comic book superheroes, is coming to a mere mortal near you: A budding technology called augmented reality uses computers and cameras to mesh views from different perspectives, giving the wearer of augmented reality goggles something like superhuman vision. Now, a group of computer scientists at the University of Rochester -- one of only a handful of groups working on augmented reality -- has proposed a new way for cameras and computers to cooperate to produce an image that very convincingly merges computer-generated images and real-world videos.
This ability to seamlessly combine a visible object, like a wall, with a whole database of images you can't see, such as an infrared view or a blueprint of the wall's interior, has many applications in such fields as medicine, entertainment, maintenance, and defense. For example, instead of having to glance back and forth between a patient and an MRI view of his brain tumor, surgeons at Brigham and Women's Hospital in Boston can see on a monitor the MRI view superimposed over a video image of the patient. The next step may be augmented reality goggles -- eyewear that superimposes the MRI view directly over the naked eye's view of the patient so that surgeons can even more precisely eradicate only the tumor.
"It sounds futuristic, but this technology's a lot closer than you think," says James Vallino, a Rochester graduate student who has been working on augmented reality with Assistant Professor Kiriakos Kutulakos. "It's been evolving slowly but steadily, led by military, design, and medical projects, some of which already use augmented reality."
A version of augmented reality already makes it possible for weather forecasters to appear on the air in front of computerized maps. There are many other high-tech applications being investigated by the Rochester researchers and others at institutions such as MIT, Carnegie Mellon University, and UNC- Chapel Hill:
The biggest problem faced by the few labs now studying augmented reality is how to flawlessly coordinate the video and computer graphic images in an augmented reality sequence. "Most people want to align the video image and the computer-generated image by building a super-accurate computer model of the three- dimensional setting in which augmentation takes place," says Vallino. "If you're working in a small room or an uncomplicated space, this is easy to do. But what if you're designing goggles for a soldier on a battlefield? Things are always changing, and it's impossible to model that setting."
"In previous approaches the environment had to be tightly controlled for augmented reality to work," Kutulakos adds. "We're now trying to move outside of the lab into less structured environments."
The Rochester group takes a unique approach to this problem of camera-computer coordination. Its approach uses what's called an affine set of axes to place computer graphics into videos using visible natural landmarks as reference points. "You could get the computer to precisely place a three-dimensional computer- generated character on top of a train," Vallino says. "When the train moves, the resulting motion of landmarks around the train is detected to keep the character correctly positioned in the image despite the motion. Augmented reality through such tracking is much easier to manage than the laborious frame-by-frame manipulation now used in Hollywood to make scenes like computerized Martians wandering through the White House."
Vallino and Kutulakos are joined in the augmented reality research by Chris Brown, a professor of computer science. The research is funded by the U.S. Department of Defense's Defense Advanced Research Projects Agency (DARPA) and the National Science Foundation.
EDITORS: For more information on augmented reality, including images, see Vallino's Web page at http://www.cs.rochester.edu /u/vallino/ .