Carnegie Mellon University
Skip navigation and jump directly to page content

The Perfect Blend

Videos Give 'See-Through' Illusion

See-Through Buildings

Researchers at Carnegie Mellon's Robotics Institute are working on technology that gives people the illusion of seeing through buildings, using a camera network.

When used with a windshield display in vehicles, for example, their work could help prevent accidents and fatalities by enabling drivers to see pedestrians, cyclists and oncoming traffic around corners.

"We provide users with the ability to see through walls by estimating the geometry of the camera network, and by appropriately blending images from the cameras viewing hidden part of the scene, with images from the point of view of the user," said Yaser Sheikh, assistant research professor at Carnegie Mellon's Robotics Institute, who has been working in collaboration with Professor Takeo Kanade, and graduate students Ankur Datta and Peter Barnum.
Rendering the final video turned out to be an interesting challenge, Sheikh said.

"On the one hand, we want to make the see-through seem as perceptually plausible as possible, but on the other hand, we had to maintain a sense of the artificial," he explained. "We couldn't have people running into walls."

In addition to helping drivers, they are also interested in extending this technology for search and rescue efforts during disasters like the earthquake in Haiti.

"Small, camera-equipped robots can be dispatched and the consolidated feeds that they send back can be used to locate victims and to plan efficient rescue strategies," Sheikh said.

At its core, this research is about consolidating several videos of the same scene into one perceptually coherent view. With the wide proliferation of cameras, there has been an explosion of video on the internet. Sheikh says this online repository is largely unorganized, despite the strong relationships that exist between the videos.

"The broad impact I hope our research will have is to organize videos in ways that enhance the experience of exploring the world through video."

The final production of the team's technology will require the application of wireless networking, sensor networks, computer vision, novel display systems and computer graphics.

Related Links: Watch Video  |  Yaser Sheikh's Website  |  Robotics Institute

Homepage Story Archives