InstructMesh Repairs AI 3D Models for Functional Printing

Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory, Google, and Northeastern University have unveiled InstructMesh, an AI-driven interface designed to bridge the gap between generative 3D modeling and functional fabrication. While current generative systems excel at visual aesthetics, they frequently produce structurally unsound or non-functional designs. InstructMesh addresses this limitation by allowing users to generate, inspect, and refine 3D models through natural language prompts before physical production, ensuring the final printed object aligns with both aesthetic vision and practical requirements. The platform integrates Microsoft’s TRELLIS system, which converts text and image inputs into 3D geometry, with OpenAI’s GPT-4 language model to combine visual generation with advanced reasoning. This hybrid architecture operates within the generative model’s latent space, enabling users to highlight specific regions for modification and describe desired changes using conversational language. The interface supplements textual commands with precision sliders, allowing nuanced adjustments to dimensions, extrusions, and structural reinforcements. By translating abstract prompts into geometric edits, InstructMesh significantly lowers the technical barrier to 3D design, eliminating the need for traditional computer-aided design expertise. Testing conducted by the research team demonstrates the platform’s effectiveness in real-world scenarios. When evaluated against popular pre-existing models, approximately 80 percent contained structural flaws. Novice users employing InstructMesh successfully identified and corrected these issues in roughly 90 percent of cases, as verified by expert reviewers. The tool has been used to fabricate a range of functional and customized items, including a structurally sound dragon-themed mug, a denim-textured knee brace, a multi-tentacle beverage dispenser, and a motorized bristle bot robot. User feedback confirms the platform’s intuitive workflow and its ability to produce prints that meet stated functional criteria. Looking ahead, the developers plan to expand the system by incorporating physics simulations to predict material stress and drop durability, alongside integration of the upcoming TRELLIS.2 architecture for finer geometric control. Lead author Faraz Faruqi, now at Google, also outlined potential applications within augmented reality platforms, where users could prompt context-aware designs based on their immediate surroundings. The research, led by Faruqi and MIT associate professor Stefanie Mueller, will be presented at the ACM Symposium on User Interface Software and Technology in November. The project received partial funding from Google and the MIT-HPI Collaborative Research Program.