One-sentence Summary

Researchers from Meta, Nanyang Technological University, University of Washington, Brown University, and Northwestern University introduce HumanCLAW, an evaluation framework that decouples vision-language model decision-making from low-level execution by having a harnessed VLM issue atomic skill commands converted into sub-second full-body motion, thereby isolating action intelligence, and a benchmark of 1,218 episodes reveals that the best model reaches only a 16.8% success rate due to lacking embodied self-awareness.

Key Contributions

• The paper introduces HumanCLAW, an evaluation framework that decouples a VLM’s action decision-making from low-level motor execution by translating atomic skill commands into sub-second whole-body motion with gravity and collisions, allowing physical consequences without balance or tracking errors.
• The paper builds HumanCLAW-Bench, a benchmark of 1,218 long-horizon, egocentric find-navigate-interact episodes across 41 indoor scenes, and tests nine state-of-the-art VLMs, with the best reaching only a 16.8% success rate.
• The paper’s experiments show that the core bottleneck is not visual recognition but a lack of embodied self-awareness: models lose track of their own body’s location, goal status, and collisions, which the paper attributes to training that treats the body as mere pixels without forming a proprioceptive, consequence-predicting internal model.

Introduction

Vision-language models (VLMs) show strong scene understanding, but it remains unclear whether they can translate that into real-time physical actions through a body—a capacity the authors call action intelligence. Prior evaluations either sidestep the motion problem by using abstract, pre-scripted actions, or entangle decision-making with embodiment-specific motor control, making it impossible to separate reasoning from execution. The authors introduce HumanCLAW, a framework that lets a frozen, off-the-shelf VLM control a full-body humanoid by selecting parameterized atomic skills (like walking a stride or turning an angle) in a closed-loop, physically simulated world. Each decision is realized as continuous motion and its physical consequences are applied, but motor failures are factored out, so errors can be attributed to the decision level. They then build HumanCLAW-Bench, a set of 1,218 egocentric find-navigate-sit episodes across 41 houses, and test nine frontier VLMs. The results show that while perception is largely intact, the models consistently fail at embodied self-awareness: knowing where the body is, whether it has arrived, and what its motions have just caused.

Dataset

The authors introduce HumanCLAW‑Bench, a long‑horizon embodied benchmark built on the HSSD dataset. It centers on a find‑navigate‑interact task where an agent must locate a specified object, reach it, and finally sit on it. The data is used exclusively for evaluation, with metrics stratified by difficulty.

Dataset composition and sources

• Source scenes: 41 validation houses from HSSD, a large, diverse set of human‑designed indoor homes spanning multiple floors.
• Episodes: 1,218 total episodes, each specifying an agent start pose and a target object category (chair, bed, couch, potted_plant, toilet, tv).
• Subsets:
  • Sit subset (597 episodes): targets bed, couch, toilet; episodes require the full find → navigate → sit progression.
  • Navigation subset (621 episodes): targets chair, potted_plant, tv; episodes terminate at successful navigation (no interaction required).

Processing and scene setup

• Physics: Plausibly movable objects (table‑top items, small furniture) are converted to dynamic so they are displaced by contact; large fixtures (beds, sofas, toilets, walls) remain static.
• Difficulty metadata: Each episode is scored on three geometric dimensions of the route to the target:
  • Distance: geodesic path length to the nearest same‑category goal.
  • Choice: branchiness, measured by the number of significant turns on a simplified path plus the number of rooms traversed.
  • Obstacle: average number of obstacles within 1 m of the route. These dimensions are computed by reconstructing the shortest navigable route with A* on a per‑floor 5 cm occupancy grid (walls and static furniture) and then binned into easy/medium/hard using fixed physical thresholds. The overall difficulty is the average of the three levels, yielding a roughly 23/46/31% split across tiers.

How the data is used

• Evaluation split: All 1,218 episodes are used for evaluation; no training split is drawn from these houses.
• Metrics: Find and navigation metrics are calculated over all episodes; interaction metrics are restricted to the 597 sit‑subset episodes.
• Difficulty‑stratified analysis: The tiered difficulty labels allow the authors to report performance separately for easy, medium, and hard episodes, isolating the effect of geometric challenge.

Method

The authors present HumanCLAW, a framework that enables a Vision-Language Model (VLM) to control a full-body humanoid in a physical environment through a closed-loop agentic process. The system operates at a sub-second rate, closing the loop between observation, decision, action, and feedback.

Refer to the framework diagram:

At each timestep $t$t, the agent receives a 2D egocentric RGB observation $o_t$ot​ and a compact textual history $h_t$ht​. A skill-decision module $\mathcal{H}$H, built around a harnessed VLM, maps these inputs along with the high-level instruction $p$p to a parameterized skill call:

where $s_t$st​ is an atomic skill from a fixed set and $c_t$ct​ is its continuous parameter. A decoupled motion generator $\mathcal{G}$G then realizes this call as a future motion chunk:

conditioned on the current body state $x_t$xt​. Finally, a locomotion-decoupled physical simulator $\mathcal{W}$W executes this chunk and updates the world state to return the next observation.

To enable the VLM to infer low-level motion skills from egocentric observations, the authors employ a contextual reasoning harness.

As shown in the figure below:

This harness structures the decision-making process through guided reasoning rather than direct motion prediction. It begins with contextual prompting, providing the VLM with agent identity, task instructions, available skills, and a history of past decisions to ensure action coherence. The VLM then performs ego-perception to externalize spatial understanding, describing visible objects, distances, and obstacles. The core of the harness is a high-to-mid-to-low reasoning scaffold. The model first interprets the long-horizon intent, derives a mid-level objective (e.g., approaching a target), and finally converts this into a low-level motion skill and its parameters in a structured JSON format. Before execution, a skill-specific spatial response verifier checks the proposal against safety rules to prevent unintended consequences like collisions, correcting the skill if necessary.

The motion generator $\mathcal{G}$G translates these symbolic skill calls into realistic full-body motion. The authors define a finite set of atomic skills (e.g., walk, sit, turn) with deterministic text interfaces to ensure zero-shot reliability. The backbone is a Motion Base DiT (Diffusion Transformer) trained for receding-horizon continuation. It observes the last five clean body states and generates the next fifteen frames (0.5s at 30 fps). The input tokens are formed by concatenating history and noisy future frames:

The model is trained using flow matching on AMASS data to predict the velocity field $\hat{v}_\theta$v^θ​, minimizing the loss $\| \hat { v } _ { \theta } - v ^ { \star } \| _ { 2 } ^ { 2 }$∥v^θ​−v⋆∥22​ on the future frames.

To align this skill-agnostic prior with specific commands, the authors attach plug-and-play ControlNet adapters.

As shown in the architecture diagram:

Each skill has a dedicated adapter while the base DiT remains frozen. At layer $\ell$ℓ, the frozen base block $B_\ell$Bℓ​ and trainable control block $C_\ell$Cℓ​ are combined via:

where $W_\ell$Wℓ​ is zero-initialized and $e_s$es​ is a skill-condition encoder (MLP or Fourier features depending on parameter type). This design allows new skills to be added by training a new adapter without affecting the base model or existing skills. The skill parameters are tied to physical outcomes read directly from motion geometry, such as final pelvis displacement for walking.

The resulting motion generation is both realistic and controllable.

As demonstrated in the visualizations:

The base DiT produces diverse but unconstrained trajectories, while attaching the skill ControlNet converges them to the intended behavior (e.g., a specific walking stride). Skills compose with natural transitions consistent with human dynamics, and continuous parameters allow fine-grained control, such as sitting at different target heights.

Finally, the system utilizes a locomotion-decoupled physical simulator to evaluate action intelligence. To prevent low-level motor control failures (like losing balance) from masking decision failures, the simulator employs a half-physics approach. The world follows rigid-body physics, including collision and gravity, but the humanoid is driven by kinematic velocities derived from the generated motion sequence:

This ensures the agent interacts physically with the environment (e.g., colliding with walls or pushing objects) while abstracting away balance control, ensuring that failures are attributable to action-level decisions.

Experiment

The benchmark evaluates vision-language models (VLMs) on a progressive find-navigate-interact task using a zero-shot controllable motion prior, which faithfully executes low-level skills and isolates high-level embodied reasoning. No current VLM solves the task; success collapses across stages, with the main bottlenecks being egocentric self-localization and body awareness rather than visual perception. Ablations reveal that a verifier, mid-level objectives, and a compact text history are essential for closing the loop and sustaining interaction, while longer visual history degrades performance. The strongest open model closely matches proprietary counterparts, and the core open challenge is to equip agents with a sense of their own body and spatial relation to the environment.

The controllable motion prior achieves achievement ratios close to 1.0 with very low variance across all locomotion skills, faithfully executing the commanded magnitude every time. In contrast, the text-to-motion baseline yields ratios that deviate substantially from 1.0, often doubling or halving the requested magnitude, and exhibits an order of magnitude larger variance, indicating it ignores the command and generates a generic motion instead. For walking, side-stepping, turning, stepping back, and sitting, the prior's ratios are between 0.97 and 1.00 with standard deviations ≤0.12, while the baseline produces ratios as extreme as 2.07 with standard deviations up to 1.28. Stair climbing is the only skill where the prior's ratio is noticeably below 1.0 (0.74–0.79), but it remains stable and calibratable; the baseline's stair ratios are erratic (0.82 up, 1.54 down) and variable.

No evaluated VLM reliably solves the full task; the best model succeeds in sitting on the target in only 16.8% of episodes. Success declines steeply from finding the target to navigating to it and further to interacting with it, with the ratio of navigation to finding success ranging from roughly 0.25 to 0.65 across models. Collision rates remain high for all models, and body misplacement accounts for the majority of interaction failures, underscoring a fundamental lack of embodied spatial awareness. The best model, Gemini-3.1, achieves a 64.9% FindSR but only 16.8% InteractSR, sitting on the target in fewer than one in five attempts. All models suffer from collision rates between 37.7% and 44.2% of steps, with Gemini-3.1 colliding on 39.5% of steps, indicating a pervasive inability to avoid contacts with the environment.

Ablation on HumanClawBench reveals that a verifier is decisive for interaction success, while extending text history beyond a compact recent window yields no benefit. Removing the verifier collapses navigation and interaction success rates, and text history saturates quickly: a minimal history recovers most performance while keeping input tokens low. The verifier transforms open-loop skill proposals into committed, closed-loop behavior. Without the verifier, NavSR drops from 27.0% to 2.0% and InteractSR from 18.9% to 0.0%, while FindSR declines only slightly (58.0% to 51.0%). Text history is necessary: with no history, NavSR falls to 11.0% and InteractSR to 0.0%, but extending history beyond 10 steps does not improve navigation and can even lower interaction success. Compact history (hist 10) uses 4.7K input tokens per step; hist 100 uses 12.9K tokens with no gain in NavSR (26.0% vs 27.0%) and lower InteractSR (11.3% vs 18.9%).

The table separates geometric from model-acknowledged success at each stage, revealing that target recognition is not the bottleneck: once the target is rendered in view, models almost always report it. In contrast, large gaps between geometry-only and full success emerge in navigation and interaction, where agents often come close to the target or briefly sit on it but fail to actively stop or commit the posture, pointing to deficits in egocentric self-localization and precise body placement. For the strongest models, the gap between geometry-only find success (GeoFindSR) and acknowledged find success (FindSR) is only 5–10 percentage points, showing that recognition is reliable once the target enters the ego view. Navigation success drops sharply when requiring an active stop: for GPT-5.5, GeoNavSR@20cm is 27.7% but NavSR@20cm is only 13.9%, indicating that many episodes meet the distance criterion yet the agent fails to stop.

Across all evaluated VLMs, collisions are most frequent with the legs and feet, followed by the arms and hands, while the head almost never collides. The smallest model, Qwen3.6-35B-A3B, achieves the lowest overall collision rate, and Claude-4.8 has the highest. The pattern reflects a systematic lack of body awareness, especially for lower limbs, consistent with the absence of proprioceptive input. Leg and foot collisions dominate across all models, ranging from 27.99% (Qwen3.6-35B-A3B) to 38.83% (GPT-5.5). Head collisions are negligible, never exceeding 6.93% (Claude-4.8) and falling below 3% for several models. Claude-4.8 records the highest collision rates for arms/hands (28.97%), torso (16.90%), and head (6.93%), while GPT-5.5 leads in leg/foot collisions (38.83%). Qwen3.6-35B-A3B, the smallest model, achieves the lowest overall collision percentage (34.58%) and the lowest leg/foot rate (27.99%).

The study evaluates embodied VLMs on a multi-stage task requiring spatial navigation and interaction, finding that no model reliably completes the full task; success drops sharply from target finding to navigation to interaction, with high collision rates and body misplacement revealing a fundamental lack of embodied spatial awareness. A verifier is critical for turning open-loop skill proposals into closed-loop behavior, while text history beyond a compact recent window provides no benefit. Controllable motion priors faithfully execute commanded magnitudes with low variance, unlike text-to-motion baselines that ignore commands and generate generic motions. Target recognition is not the bottleneck; failures stem from deficits in egocentric self-localization, precise body placement, and a systematic lack of lower-limb body awareness.