Researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and the Toyota Research Institute have developed a new AI-driven method called steerable scene generation to create diverse, realistic virtual environments for training robots. Unlike chatbots such as ChatGPT, which rely on vast amounts of text, robots need rich, physical simulations to learn how to interact with the real world. Traditional methods of creating these simulations—either through handcrafted digital scenes or physics-approximate AI-generated models—often lack realism or diversity. The new system tackles this challenge by using a diffusion model, a type of generative AI that creates images from random noise, and steers it toward generating lifelike 3D scenes such as kitchens, living rooms, and restaurants. Trained on over 44 million 3D rooms with common household objects, the model places existing 3D assets into new configurations and refines them to match real-world physics. It prevents common issues like object clipping—where items pass through each other—by ensuring physical plausibility. A key innovation is the use of Monte Carlo Tree Search (MCTS), a strategy previously used in AI systems like AlphaGo to evaluate sequences of decisions. In this case, MCTS treats scene generation as a step-by-step process, exploring multiple ways to build a scene and selecting the one that best meets a goal—such as maximizing realism or including more edible items. This approach allows the system to create scenes far more complex than those in its original training data. For example, it successfully added up to 34 items on a restaurant table, far exceeding the average of 17 seen in training. The method also supports reinforcement learning, where the AI learns to generate scenes that score highly based on user-defined goals. By rewarding specific outcomes—like arranging objects in a certain way—the model adapts to produce novel, task-relevant scenes. Users can also input natural language prompts, such as “a kitchen with four apples and a bowl on the table,” and the system accurately generates the scene, achieving 98% accuracy on pantry shelves and 86% on messy breakfast tables—outperforming existing tools by at least 10 percentage points. The system can also rearrange objects in existing scenes, such as placing apples on different plates or organizing books and games on a shelf, effectively “filling in the blank” while preserving the rest of the environment. According to lead researcher Nicholas Pfaff, the method’s strength lies in its ability to generate scenes that go beyond the original training distribution, creating diverse, realistic, and task-specific environments ideal for robot training. The team tested the system by simulating robots performing real-world tasks, such as placing cutlery in a holder or arranging bread on plates. The movements appeared smooth and physically plausible, suggesting the potential for training adaptable, real-world robots. While the current version uses a fixed library of 3D assets, the researchers aim to extend the system to generate entirely new objects and interactive elements—like cabinets that open or jars that can be twisted—making scenes even more dynamic. They also envision integrating vast internet data to further expand the range of possible training environments. Experts like Rick Cory of the Toyota Research Institute praise the framework as a promising step toward scalable, efficient robot training. The work, supported by Amazon and the Toyota Research Institute, was presented at the Conference on Robot Learning in September.