The AI Tool That Could Make Manufacturing Faster and More Efficient — by Using Lego Bricks

A new AI-powered tool created by researchers at Carnegie Mellon University’s School of Computer Science(opens in new window) could change the way we manufacture and build things.  

BrickGPT(opens in new window) uses text prompts to help people — and even robots — bring ideas to life with Lego bricks. It takes a simple word, such as “guitar,” and creates a brick-by-brick guide for a person or a robot to build a physically stable model of that object. The tool is currently focused on building with Lego bricks, but turning a text prompt into something physically stable goes beyond play. 

“This research paves the way toward generative manufacturing, which is when people can use a generative model to design everyday objects they can build themselves,” said Jun-Yan Zhu(opens in new window), the Michael B. Donohue Assistant Professor of Computer Science and Robotics in CMU’s Robotics Institute(opens in new window) (RI). “They can build a chair, a sofa or a kids' toy. This is a new frontier, a new usage of these models beyond creating social media videos or photos. These brick toy pieces are a simple medium, and it’s a starting point.”

Researchers said this fusion of AI and robotics could speed up the process of designing and building new things.

“This could be a huge benefit to the manufacturing world,” said Changliu Liu(opens in new window), an associate professor in the RI. “It takes a long time to turn ideas into a physical design and prototype. But if you can integrate generative AI into the process, it can significantly improve efficiency and reduce the roadblocks to kicking off projects.” 

Currently, the BrickGPT demo can produce step-by-step guides for humans or robots to build 21 types of models out of Lego bricks, including a birdhouse, sofa and piano. If someone wanted to generate a sofa, they would type "sofa" into BrickGPT, which generates a 3D model. Then, an algorithm transforms the 3D model into brick structures and BrickGPT checks to ensure the structure is stable. A person or robotic arm can follow the steps and build the sofa. 

To train BrickGPT, researchers generated StableText2Brick, a dataset containing over 47,000 brick structures made from more than 28,000 unique 3D objects accompanied by detailed captions. Researchers took an existing dataset of 3D shapes, ShapeNetCore, and converted these shapes into a grid of small cubes, a process the study describes as voxelizing. They then trained an autoregressive large language model (LLM), which predicts future values based on previous ones. For example, in BrickGPT, the LLM predicts the next brick based on the previous one, ensuring that the structure is stable and won’t fall over. If there’s an error along the way, BrickGPT goes back and eliminates unstable points to guarantee the structure's stability. 

Along with Liu and Zhu, the SCS research team includes Ava Pun(opens in new window), a doctoral student in the Computer Science Department; Kangle Deng(opens in new window) and Ruixuan Liu(opens in new window), doctoral students in the RI; and Deva Ramanan(opens in new window), a professor in the RI. 

“If a structure is unstable, there’s a rollback process,” said Pun. “During that step, the model determines which bricks were wrong or unstable and we roll back to the point before that. We detect instability with our physics reasoning algorithm, which generates a stability score for each brick in the structure. If the score is high enough, that means that brick was stable.”

Researchers hope to scale up this model, enabling it to generate more than the current 21 objects. They also hope to expand the diversity of their library pieces to increase the accuracy and complexity of generated designs. 

The research was funded by CMU’s Manufacturing Futures Institute (opens in new window)with the support of facilities at Mill19(opens in new window). For more information, or to test the BrickGPT demo, visit the research website(opens in new window).