Powerful Collaborations

Driving Industry Advancements with Robotics and AI

At Three Mile Island, the basement of the damaged nuclear reactor was left untouched and unseen for five years until, in 1984, a robot entered the site to begin inspection and cleanup of the most serious commercial nuclear power plant accident in US history.

That robot was designed by Professor William “Red” Whittaker (MS ’75, PhD ’79), and his students. It was the first of many innovations to come from the Field Robotics Center, founded by Whittaker, now the Fredkin Research Professor at the CMU Robotics Institute, to focus on robots for work sites and natural terrain. Before long, a second of their robots was collecting core samples from the basement walls at Three Mile Island. That same year, the Terregator was built—the first autonomous outdoor navigation robot capable of successfully exploring rugged grounds.

“Civil Engineering jumped right in on the robotics work going on at the university and in the outside industry,” asserts Ben Motazed (BS ’78, MS ’80, PhD ’85), one of Whittaker’s students at the time. “No institution in the world was doing the kind of things we were doing.”

For Motazed, his time in the Field Robotics Center was life-changing. After his studies, he stayed at Carnegie Mellon for 10 years working in sensing and autonomous ground vehicles. From there, he applied these skills to do everything from engineer autonomous flight systems to help launch Google’s drone-based delivery service, Project Wing. Today, as an enterprise account manager at Intel, Motazed helps companies use drones to improve their operations in the oil and gas, construction, utilities, and agriculture industries.
 
While the Field Robotics Center has since moved from CEE to join the Robotics Institute, this spirit of innovation remains at the core of our department. Today’s CEE faculty and students build on this legacy, continuing to leverage technologies such as artificial intelligence and robotics developments that bring groundbreaking solutions to the complex infrastructure and environmental challenges of today and tomorrow.

Inspecting Deteriorating Infrastructure

Hundreds of feet above the ground or below water, bridge inspectors must attempt hazardous climbs and squeeze their way into tight dark spots. It’s difficult, time-consuming work that requires expensive, specialized equipment. It’s also an essential task for safe and high-performing infrastructure.

CEE Professor Burcu Akinci, along with members of the Carnegie Mellon Robotics Institute and researchers at Northeastern University, have been working on a safer, lower-cost, and more efficient way to inspect bridges with drones.

Already, some organizations capture infrastructure imagery and video with drones, but this footage provides only a limited view. Akinci and her collaborators have a different approach. Their drones collect and use data to rapidly construct interactive 3D models of bridges that inspectors can review. These drones become the inspector’s apprentice, automatically identifying and analyzing objects for defects. From the finished model, you can produce detailed inspection and assessment reports featuring objective measurements with defined degrees of certainty. You can also compare models year over year to review structural changes over time.

While the group is still refining algorithms and software, their prior work has received much attention and was even demonstrated to President Barack Obama in 2016. For Akinci, collaborating with the Robotics Institute is nothing new. “When I started at CMU in 2000, it was clear that we were missing a complete 3D understanding of what was happening in infrastructure and facilities,” she says. “I saw a need to develop better approaches to quality control, inspections, and condition assessments.”

With this realization, Akinci approached her colleagues in robotics and computer vision, who quickly jumped on board to assist with a research proposal for National Science Foundation funding. It was the start of a long line of projects that have led to today’s drone-based bridge inspections.

“They bring the vision expertise and I bring the civil engineering expertise in solving these problems. We need both,” she declares. “The Carnegie Mellon culture is extremely unique. It is not easy to identify another place where someone could so easily create these kinds of collaborations. It’s been close to 18 years and there’s not a single year we haven’t worked together.”

Designing Smart Space Habitats

Bridge inspection drones aren’t the only piece of flying technology on Akinci’s mind lately; she’s also participating in a project featuring technology designed to go far beyond Earth’s atmosphere. Using their knowledge of smart buildings on Earth, Akinci and fellow CEE Professor Mario Bergés are helping NASA design self-diagnosing and self-repairing systems for smart habitats in space.

As NASA plans lunar orbits and deep space missions, maintaining functioning systems in a space habitat could be a matter of life or death. Even in minor cases, repairs take away precious time from experiments and scientific exploration. “The astronauts have limited time and their missions have specific budgets and goals. Anything that deviates from what they’re tasked to do is costly,” explains Akinci.

Adding to that difficulty is that, in deep space, real-time communication with mission control on Earth is non-existent. Receiving messages takes too long. “You have to aim for real autonomy where the space habitat itself, either in conjunction with the occupants or on its own, can resolve issues that arise during the mission,” says Bergés. The goal is a system that can predict and fix potential malfunctions, or at least guide the astronauts in implementing solutions, before any flaw becomes critical.

As a researcher, Bergés focuses on extracting useful information from sensed measurements of infrastructure systems, including buildings. “Take the meter on your house,” he shares. “Through machine learning and statistical analysis, using only the electricity measurements captured by that meter, we can identify when each specific appliance turns on and off and even discover flaws in your appliances.”

Working with NASA, Bergés and Akinci will first see what they can uncover and predict using data from sensors inside the NASA Sustainability Base at Ames Research Center in California. Moving forward, they’ll study how to apply these predictions in smart space habitats and draw on the team’s knowledge of innovative facility management technologies to devise strategies to equip systems for self-repair.

“The constraints of space make the project more complicated but more interesting too,” says Bergés. “In this case, the entire building system is really a life-support system.”

Simplifying Drone Operations

Hae Young Noh is another CEE professor working on life-saving technology. In fires, earthquakes, and other emergencies, drones can help find survivors inside buildings. Yet controlling and localizing drones is complicated indoors. GPS isn’t reliable and the drones’ built-in cameras aren’t enough in low-visibility conditions.

You can add expensive sensors, but it’s not ideal. Maneuvering through halls and doors, with the potential for human collisions, indoor drones must remain small and lightweight. Extra sensors increase the drone’s payload and decrease battery life.

Noh and her Electrical & Computer Engineering (ECE) colleagues propose an alternative. When the drone flies, the noise of the spinning propellers is recorded. Combining that sound with building information, the researchers can localize the points of the drone’s four propellers to estimate both the drone’s location, and trajectory.

To help identify individual drones within a group, Noh’s team is also implementing low-cost, lightweight vibration sensors on each drone, so that they can match the movement captured on the drone’s camera with the unique vibration signature tracked by the sensor.

With these methods, Noh and her colleagues can reduce both drone-localization and identification times in situations where time is of the utmost importance.

Like other faculty, Noh is quick to note that many of her projects involve robotics or AI. For example, the same team has generated a novel approach to managing outdoor drones that wash skyscraper windows, another job dangerous for humans. Here, the group employs one large drone with GPS and high-resolution cameras to monitor and localize the smaller, lower-cost worker drones washing the windows.

“These projects are both partnerships with ECE Professor Pei Zhang, who is a drone and systems expert,” says Noh. “With my expertise in vibration, I focus on data analysis and probabilistic model updating. I also work on incorporating building information into drone navigation, especially for search and rescue.”

Noh’s other projects range from sensors that monitor the elderly’s movements as they walk or rise from bed—alerting caregivers of potential issues before falls happen—to sensors that can transform any tabletop or surface into a touch screen by tracking tapping and swiping movements. “The opportunities to incorporate AI and machine learning into our work are incredibly vast,” she says.

Introducing Future Engineers to Advanced Technologies

While many CEE students assist in our faculty’s boundary-pushing research, that’s far from students’ only opportunity to interact with such technology, and student research collaborations abound across departments.

What’s more, our faculty are bringing robotics technology into the curriculum earlier than ever, with our first-year Exploring CEE class featuring a project that demonstrates how engineers can use technology to understand the world around them.

For the project, students must first estimate by hand the area of a space on campus and the soil required to fill it. Later, they repeat the task using a drone’s camera and software, comparing the benefits and challenges of the two experiences.

“Our goal is for new students to understand how advanced technology can help them do their jobs better, to consider the civil and environmental engineering applications for sensors and robotics, and to also identify current technology limitations,” says course instructor Costa Samaras.

The possible uses, benefits, and costs of technology is a subject Samaras looks at often in his own research. In 2018 alone, his work has explored pressing questions on the environmental impacts of drone package delivery; economic, environmental, and travel implications of changes in parking choices due to driverless vehicles; and how automated vehicle technologies could reduce US fuel consumption.

“At CEE, it’s in our DNA to understand and work with robotics and advanced technology,” Samaras says. “It’s important that we encourage students from the beginning to use their judgment of when and how to incorporate these tools—and even question how these technologies could be improved. In doing so, we train civil engineers to think differently and find new ways to solve our infrastructure problems.”