A team of researchers from MIT has developed an AI-driven system that allows users to simply say “Robot, make me a chair” and have a robot assemble the object from prefabricated parts. The system uses a vision-language model (VLM) to interpret user prompts and generate functional, component-based 3D designs that can be physically built by robots. The researchers, led by MIT graduate student Alex Kyaw, tested their approach through a user study and found that over 90% of participants preferred the designs produced by their system compared to those generated by alternative methods. The system works by first receiving a text prompt—like “make me a chair”—and an AI-generated image of a chair. The VLM then analyzes the image and determines how structural components and panel components should be arranged based on functionality and geometry. Unlike standard generative AI models that produce 3D meshes without clear component breakdowns, this system identifies which parts of the object should have panels—such as the seat and backrest—by reasoning over thousands of example objects it has seen during training. It assigns labels like “seat” or “backrest” to surfaces and maps those labels onto the 3D mesh, ensuring the design is suitable for robotic assembly. A key feature of the system is its human-in-the-loop design process. Users can refine the output by giving new prompts, such as “only use panels on the backrest, not the seat,” allowing them to guide the AI and maintain ownership over the final design. This interactive approach helps narrow down the vast design space and accommodates individual preferences. The team also evaluated the AI’s reasoning by asking it to explain its choices. The model demonstrated an understanding of functional aspects—like sitting and leaning—justifying why panels were placed where they were. This shows the system isn’t just randomly assigning parts but making decisions based on real-world use. Once the design is finalized, a robotic assembly system constructs the object using reusable prefabricated components that can be disassembled and reconfigured into different forms. This supports rapid prototyping and sustainable manufacturing, especially for complex structures like aerospace parts or architectural elements. Looking ahead, the researchers aim to expand the system’s capabilities to handle more complex materials—such as glass and metal—and incorporate moving parts like hinges and gears. Their ultimate goal is to make design and fabrication accessible to everyone, enabling people to turn everyday ideas into physical objects quickly and sustainably. “This is a first step toward a future where we can talk to robots and AI systems the same way we talk to each other to create things together,” Kyaw said. The work was presented at the Conference on Neural Information Processing Systems and involves collaborators from MIT’s departments of Electrical Engineering and Computer Science, Architecture, and Mechanical Engineering, as well as Google DeepMind and Autodesk Research.