Command Palette
Search for a command to run...
Mind2Act World : un banc d'essai diagnostique pour l'esprit incarné, l'action physique et la composition esprit-action en manipulation robotique
Mind2Act World : un banc d'essai diagnostique pour l'esprit incarné, l'action physique et la composition esprit-action en manipulation robotique
Résumé
Mind2Act World est un banc d'essai diagnostique pour l'esprit incarné, l'action physique et leur composition en manipulation robotique. Les tâches Mind testent le raisonnement fondé sur l'historique. Les tâches Act testent l'exécution physique précise et opportune, et les tâches Mind2Act testent leur composition en boucle fermée, où les résultats des actions guident les décisions ultérieures. Les 30 tâches proviennent de 5 domaines d'application du monde réel et couvrent 12 catégories de capacités. Chacune s'exécute à 3 niveaux de difficulté dans des scènes Clean et Random (180 configurations), avec 4 500 démonstrations expertes pour l'entraînement de politiques de bout en bout. Les agents et les politiques de bout en bout partagent les mêmes tâches, le même contrat d'information et le même évaluateur, et sont classés sur des classements distincts. Une boucle de diagnostic attribue ensuite les épisodes échoués d'un système à Mind, Act ou à leur composition et les résume dans une fiche de profil de modèle qui indique sa prochaine amélioration. En haut : les trois vues de Mind2Act World. Au milieu : les domaines d'application, les catégories de capacités, les tâches et la boucle de diagnostic. En bas : l'écosystème open source, les modèles évalués et les classements continuellement mis à jour.
One-sentence Summary
Researchers from THU and collaborating institutions introduce Mind2Act World, a diagnostic benchmark for robotic manipulation that evaluates embodied mind, physical action, and their closed-loop composition across 30 tasks from 5 real-world application domains and 12 capability categories, each at 3 difficulty levels in Clean and Random scenes (180 configurations) with 4,500 expert demonstrations, and that gives agents and end-to-end policies the same tasks, information contract, evaluator, separate leaderboards, and a diagnosis loop that attributes failed episodes to Mind, Act, or their composition and summarizes them in a model profile card.
Key Contributions
- Mind2Act World is introduced as a diagnostic benchmark for robotic manipulation, with 30 tasks organized into Mind, Act, and Mind2Act views across 5 application domains, 12 capability categories, 3 difficulty levels, and Clean/Random scenes, yielding 180 configurations and 4,500 expert demonstrations.
- The benchmark provides a shared evaluation protocol and diagnosis loop in which agents and end-to-end policies use the same tasks, information contract, and evaluator, and failed episodes are attributed to Mind, Act, or their composition to produce a model profile card for guiding improvement.
- A large-scale evaluation of three agents, four vision-language-action models, and two WAMs over 56,700 episodes shows limited transfer from single views to Mind2Act tasks: the best system drops from 52.7% on Mind and 39.0% on Act to 12.0% on Mind2Act, no end-to-end policy exceeds 3.6%, and eight tasks are completed by no system.
Introduction
Embodied intelligence requires robots to reason over task history, execute physical actions precisely, and compose both capabilities through environmental feedback. Existing benchmarks often evaluate reasoning and execution within the same tasks or separately, without testing whether physical outcomes effectively guide later decisions, so failures are hard to attribute to memory, execution, or closed-loop composition. The authors introduce Mind2Act World, a diagnostic benchmark with 30 manipulation tasks across five real-world domains, organized into Mind, Act, and Mind2Act views, each with difficulty levels and paired scene settings. They also provide a diagnosis harness that attributes failed episodes and summarizes systems in model profile cards.
Dataset
Dataset overview
- Mind2Act World is a simulation benchmark with 30 bimanual manipulation tasks on an ARX X5 platform.
- Tasks are organized into three views:
- Mind: history-dependent reasoning.
- Act: constrained physical execution.
- Mind2Act: composition of reasoning and execution.
- Each task has Easy, Medium, and Hard levels.
- Each level is evaluated in Clean and Random scene settings, giving 180 total configurations.
Composition and sources
- Tasks are built from real-world requirements across five application domains:
- Home and food service.
- Manufacturing and logistics.
- Instrument operation and dexterous tool use.
- Assembly and physical reasoning.
- Companion and edutainment.
- The authors use tabletop abstractions of practical requirements such as remembering orders, maintaining instrument readings, intercepting conveyor objects, and responding to appliance feedback.
Subset details
- Mind tasks: 10 tasks across five memory categories, with two tasks per category.
- Entity memory, feedback-rule learning, process memory, spatial memory, and integrated memory.
- These tasks require only routine physical execution and isolate reasoning over long-horizon dependencies.
- Act tasks: evaluate dynamic response, dynamic stability, and precise physical execution.
- Examples include tray carrying with loose balls, cyclic gate timing, instrument reading, stamp orientation, fabric folding, and domino spacing.
- Mind2Act tasks: evaluate composition across dependent operations.
- Mechanisms include cumulative progress, state-dependent prerequisites and repair, feedback-driven adjustment, sequence and plan execution, and predictive decision under time pressure.
- Difficulty levels keep the objective fixed while increasing memory load, precision, timing, workflow dependencies, or number of operations.
Data collection and filtering
- The authors provide expert demonstrations collected under one protocol for every task and difficulty level.
- Task scripts define operation sequences and end-effector targets.
- Inverse kinematics and motion planning convert targets into joint trajectories.
- Quality control checks physical completion against simulator states and events.
- Recording checks cover:
- Missing or corrupted frames.
- Invalid state or action values.
- Timestamp alignment.
- Duplicates.
- Demonstrations must satisfy task-specific qualification conditions stricter than task success requirements.
- Every retained demonstration passes the evaluator used at test time.
- Demonstration and evaluation seeds are disjoint.
Data schema and scale
- Each episode records:
- Task instruction.
- RGB images from one head camera and two wrist cameras at 640 x 480.
- Joint states and joint targets.
- Physics runs at 120 Hz, and recorded data are sampled at 30 Hz.
- The dataset contains:
- 50 episodes per task per difficulty level.
- 4,500 episodes total.
- 8.19 million frames per camera.
- 75.8 hours of recordings.
- Average episode durations:
- Act: 23.9 seconds.
- Mind: 58.5 seconds.
- Mind2Act: 99.6 seconds.
- Hard Mind2Act: 127.6 seconds.
Usage in the model
- The demonstrations are used to train end-to-end policies under a shared protocol.
- The benchmark supports architecture-agnostic evaluation with shared tasks, instances, and scoring rules.
- Clean and Random settings use the same task instances, so comparisons isolate visual and scene perturbation from task difficulty.
- The provided excerpt does not describe a formal training/validation/test split or mixture ratios.
- No image cropping strategy or additional metadata construction is described in this section beyond the recorded instruction, RGB images, joint states, and joint targets.
Method
The authors introduce the Mind2Act World Benchmark to evaluate whether failures in manipulation tasks arise from incorrect reasoning, imprecise execution, or a poor composition of the two. The benchmark comprises 30 simulation tasks on a bimanual ARX X5 platform, organized into three views that place different demands on a system.
As shown in the framework diagram:
The benchmark is structured around three distinct task views. Mind tasks focus on history-dependent reasoning, requiring the system to retain evidence and update task state throughout an episode when relevant cues disappear. Act tasks emphasize constrained physical execution, evaluating whether a specified operation produces the required physical effect within spatial and temporal constraints. Mind2Act tasks test the composition of these two capabilities, focusing on outcome-conditioned continuation where retained information determines the next operation and physical outcomes update the task state.
To diagnose specific failure modes, the authors design tasks with varying levels of difficulty. Each task features Easy, Medium, and Hard levels, where the objective remains fixed but memory load, reasoning demands, precision, timing, or workflow dependencies increase.
The difficulty progression across different tasks is illustrated below:
Furthermore, the authors analyze the difficult steps within Hard-level tasks to understand where failures typically occur. These intermediate operations often block subsequent stages, highlighting the importance of tracking stage progress alongside final success.
The intermediate steps and specific demands of Hard-level tasks are demonstrated in the following figure:
The data collection pipeline ensures high-quality demonstrations. Each episode has a setup defined at three levels: task, difficulty level, and episode parameters. Task scripts define the operation sequence and end-effector targets, which are converted into joint trajectories using inverse kinematics and motion planning. Quality control checks physical completion against simulator states and events, and demonstrations must satisfy task-specific qualification conditions to provide clear examples for learning.
For technical implementation, the benchmark is built in Isaac Sim. Task scenes combine shared robot and workspace assets with task-specific objects and mechanisms.
The task setups across different application domains are depicted here:
The evaluation protocol supports two action interfaces to ensure architecture-agnostic comparisons. End-to-end policies predict joint or end-effector actions, which are converted to joint targets. Agents, acting as multimodal language models, issue structured tool calls such as move_eef, wait_steps, and memory_resolve. The simulation advances at 120 Hz, while arm and gripper targets are updated at 30 Hz. Simulation pauses during model inference, and task deadlines use simulation time to measure responses independently of inference latency. Episodes end on completion, policy termination, or budget exhaustion, with partial-progress scores and event records used to distinguish incorrect decisions from failed execution.
Experiment
The Mind2Act World benchmark evaluates nine systems, including multimodal agents and fine-tuned end-to-end policies, across 30 bimanual ARX X5 simulation tasks organized into Mind, Act, and Mind2Act views with three difficulty levels and clean versus randomized scenes. The setup separates reasoning, physical execution, and their composition to identify whether failures come from memory, control, or integration. Overall, agents such as GPT-6 Astra retain task state and reason better, while end-to-end policies handle practiced execution and timing better, but no system reliably combines these abilities in dependent workflows. Failures concentrate in physical execution, especially grasping and sustained contact, and Mind2Act tasks typically fail early at single steps rather than at composition, with harder levels and appearance changes exposing weak memory and limited closed-loop correction.
Mind2Act World is the only compared benchmark that evaluates the same task demands both in isolation and in composition, while also attributing failures to reasoning, execution, or their composition and providing per-model profile cards. Most existing benchmarks entangle reasoning and execution within single tasks or measure them separately, limiting failure attribution. Its task demonstrations are also relatively long, especially in the Mind2Act view. Compared benchmarks generally lack composition testing and failure attribution, whereas Mind2Act World jointly addresses isolated and composed task demands. Training demonstrations in Mind2Act World are longer on average than those in several recent robot learning benchmarks, with the longest demonstrations in the Mind2Act view. Mind2Act World adds diagnosis beyond success and score through stage localization, attribution of failures, a dedicated harness, and profile cards, which most compared benchmarks do not report.
Mind2Act World spans five application domains mapped to real-world needs, including home and food service, manufacturing and logistics, instruments and tool use, assembly and physical reasoning, and companion and edutainment. The largest task groups are in instrument and tool use and home and food service, while assembly has the smallest group. Domain-level outcomes show wide gaps, and no domain is close to reliable deployment. Agentic performance is comparatively better in companion and edutainment, moderate in manufacturing and instrument tasks, and much lower in home and food service and assembly. End-to-end policies do best in manufacturing and logistics, where tasks repeat practiced motions such as conveyor pickup and gate passage. Four of the five domains contain especially difficult task pairs that no evaluated system completes, including cooking and kitchen loading, workpiece tracking and tray carrying, oriented stamping and whiteboard restoration, and domino chain and bridge building.
The evaluation design covers the same 180 task configurations for every system, but end-to-end policies are scored on ten times as many episodes per configuration as agents. Agents run on a shared subset of clean and random scenes, while end-to-end policies run on all clean and random instances. The primary result weights clean and random configurations equally across tasks and difficulty levels. End-to-end policies run 50 episodes per configuration, compared with 5 for agents, giving them a much larger evaluation sample per task and difficulty. Each task and difficulty level includes clean and random scenes, and the random setting adds table clutter, varied table materials, and varied lighting as an appearance stress test. The primary result combines clean and random configurations with equal weight, yielding 56,700 scored episodes across nine evaluated systems.
Agents and end-to-end policies are ranked separately, and the best model in each family succeeds in only a minority of episodes while most others trail far behind. Success is concentrated in single-capability Mind or Act tasks, and composed Mind2Act workflows remain much weaker for all systems. Since a scripted controller completes every task, the shortfall appears to come from model limitations rather than the tasks or harness. The leading agent and end-to-end policy succeed in only a minority of episodes overall, with most other systems far below either leader. A scripted controller with simulator state completes every task, suggesting the low success rates reflect model limitations rather than impossible tasks. Success is narrow: single-skill Mind and Act tasks account for the large majority of successes, and several tasks are completed by no system across all episodes. On the Mind view, the strongest agent leads by a wide margin and handles fixed remembered information, but no system solves tasks that require updating remembered positions and order after objects are moved. On the Act view, end-to-end policies perform best but still trail the strongest agent; precision and sustained-contact tasks remain difficult, with several such tasks unsolved. On the Mind2Act view, all systems perform worse than on Act, and the strongest agent drops sharply while end-to-end policies fall close to their already low Mind rates.
Across pooled evaluation settings, no evaluated system is close to reliable. Agent performance is moderate in edutainment, manufacturing, and instrument use but drops sharply in home and food service and assembly, while end-to-end policies rank domains differently and remain weak almost everywhere. Several tasks in most domains are never completed by any system. The strongest agent succeeds in roughly half of edutainment episodes and somewhat less in manufacturing and instrument use, then falls to under one in five for home and food service and to single digits for assembly. End-to-end policies are strongest in manufacturing and logistics, where repetitive motion helps, but they stay below 12% in every other domain. Four of the five application domains each contain multiple tasks that no evaluated system completes, including kitchen, tracking, stamping, and assembly-related tasks. Within home and food service, GPT-6 has uneven results across tasks and X-VLA achieves the best single-task success rate, but neither shows broad reliability.
Mind2Act World evaluates agents and end-to-end policies across 180 task configurations in five application domains, uniquely combining isolated and composed reasoning and execution demands with failure attribution through stage localization and profile cards. The evaluation design weights clean and random scenes equally, with agents scored on fewer episodes per configuration than end-to-end policies. Results show all systems remain far from reliable, as success concentrates in single-capability tasks while composed workflows and several domain-specific tasks stay unsolved, and a scripted controller completes every task, indicating model limitations rather than impossible tasks. Domain performance varies but no setting approaches dependable deployment.