Animated Photos
3D Object Manipulation in a Single Photograph using Stock 3D Models: Full Length Video
Reach into a photograph. Grab hold of a car. Toss it and watch it tumble. See the sides hidden from view. Make it fly, all with the click of a mouse.
Sound fun?
Yaser Sheikh, associate research professor of robotics, and a team of researchers at Carnegie Mellon University have made all of this possible, pushing the boundaries of photo editing into the third dimension.
Editors of photos routinely resize objects, or move them up, down or sideways, but Sheikh and his team are adding an extra dimension to photo editing by enabling editors to turn or flip objects any way they want, even exposing surfaces not visible in the original photograph.
This three-dimensional manipulation of objects in a single, two-dimensional photograph is possible because 3-D numerical models of many everyday objects — furniture, cookware, automobiles, clothes, appliances — are readily available online.
The research team led by Sheikh and Natasha Kholgade, a Ph.D. student in the Robotics Institute, found they could create realistic edits by fitting these models into the geometry of the photo and then applying colors, textures and lighting consistent with the photo.
Objects also can be animated; the researchers demonstrated that an origami bird held in a hand can be made to flap its wings and fly away, or a taxi cab shown in a street scene can levitate, flip over to reveal its undercarriage and zip off into the heavens.
Though the system is designed for use with digital imagery, it enables the same 3-D manipulation of objects in paintings and in historical photos.
"Instead of simply editing 'what we see' in the photograph, our goal is to manipulate 'what we know' about the scene behind the photograph," Kholgade said. "In the real world, we're used to handling objects — lifting them, turning them around or knocking them over. We've created an environment that gives you that same freedom when editing a photo."
Though a wide variety of stock models is available online, models are not available for every object in a photo. But that limitation is likely to subside, particularly as 3-D scanning and printing technologies become ubiquitous.
"The more pressing question will soon be, not whether a particular model exists online, but whether the user can find it," Sheikh said.
Kholgade presented the team's findings in August at the SIGGRAPH 2014 Conference on Computer Graphics and Interactive Techniques in Vancouver, Canada.
This research was sponsored, in part, by a Google Research Award. In addition to Kholgade and Sheikh, the team included Tomas Simon, a Ph.D. student in the Robotics Institute, and Alexei Efros, a former CMU faculty member now an associate professor of electrical engineering and computer science at the University of California, Berkeley.
Related Links: Read press release | SIGGRAPH 2014 | Robotics Institute