MIT FLEX Helps Robots Infer Intent Beyond Literal Commands

MIT engineers and collaborators from New York University have developed FLEX, a flexible feature-editing pipeline that helps robots infer unstated intent from natural language commands, improving motion planning without requiring users to specify every rule. The work will be presented at the annual Conference on Robot Learning and was led by Andreea Bobu, an MIT assistant professor of aeronautics and astronautics who heads the Collaborative Learning and Autonomy Research Lab. The team includes Nathan Dennler, a postdoc at MIT’s Computer Science and Artificial Intelligence Laboratory, Audrey Lee, and NYU researcher Ilia Sucholutsky. FLEX enables a robot to follow spoken or typed instructions literally while also reading between the lines to identify broader preferences. In one simulated example, a robot arm moves a coffee cup across a cluttered table and a user types “Don’t spill on the laptop!” Rather than only avoiding the laptop, the system reasons that liquids can damage electronic devices and updates the robot’s path to steer clear of a nearby phone as well. This generalization comes from a large language model, specifically GPT-4o, which draws on patterns learned from large text corpora to connect specific objects to wider categories such as electronics, fragile items, or flammable materials. The platform operates through three steps: Add, Generalize, and Mask. In the Add step, the robot records user-specified objects that should be avoided, such as a mug or laptop. In the Generalize step, the language model analyzes those items and infers a broader rule, such as avoiding fragile or electronic objects. In the Mask step, the robot’s computer vision system checks whether objects are still present and removes outdated entries from the avoidance list. The full process takes about five seconds. The researchers tested FLEX in simulations involving three tasks: serving coffee, moving a candle, and carrying a bowling ball. In each case, users typed natural commands such as “Don’t spill on my laptop,” “Stay away from the stack of paper,” and “Don’t drop on the mug.” The robot followed the literal instructions and also applied higher-level reasoning, avoiding electronics, flammable objects, and fragile things without explicit prompts. The researchers argue that existing natural-language robot systems often rely on users to tell machines exactly how to perform tasks, which can be impractical for household robots. By extracting common-sense assumptions from ordinary language, FLEX could make assistive and domestic robots easier to direct and more adaptable to individual homes and preferences.