AI Agents Generate Virtual Environments for Robot Training
Researchers from MIT's Computer Science and Artificial Intelligence Laboratory and the Toyota Research Institute have introduced SceneSmith, a framework that utilizes artificial intelligence agents to generate high-fidelity virtual environments for robot training. This system addresses the significant data bottleneck in robotics by creating diverse, physics-enabled simulation spaces from text prompts, thereby reducing reliance on labor-intensive physical teaching methods. SceneSmith employs a collaborative system of three autonomous agents driven by a multimodal vision-language model, specified as GPT-5.2. The architecture assigns distinct roles to a designer agent, which generates scene layouts and object configurations; a critic agent, which assesses visual realism and functional utility; and an orchestrator agent, which directs the iterative refinement cycle. This agentic workflow ensures that output environments satisfy rigorous quality criteria before being integrated into physics simulation software. Compared to baseline methods, SceneSmith produces environments with up to six times greater object density and includes manipulatable items with accurate physical attributes, such as mass, friction, and inertia. The framework has successfully generated over 1,300 scenes depicting complex indoor settings like restaurants and hotels. User evaluations indicated a preference for SceneSmith over previous systems more than 90 percent of the time, citing superior prompt adherence and visual fidelity. Functional testing validated the utility of these simulations: pretrained robot policies executed manipulation tasks effectively within generated environments that were unseen during training. Furthermore, the system's internal critic identified flawed robotic action plans with greater than 99 percent alignment with human judgment, providing a reliable mechanism for validating engineering strategies prior to real-world deployment. By leveraging internet-scale priors, SceneSmith infers spatial logic and structural relationships, allowing it to improvise creative arrangements and generate articulated objects that prior models often failed to render correctly. While the current generation process requires multiple hours per scene due to intensive agent scrutiny, the research team notes that enhanced computing power could significantly improve efficiency. Future iterations aim to incorporate deformable objects and expand asset generation capabilities. The findings were presented as a spotlight paper at the International Conference on Machine Learning.
