One-sentence Summary

Researchers at Sharpa Robotics, the SJTU-Sharpa Joint Lab for Embodied AI, SJTU, CUHK, and FDU introduce DexForesight, a framework that learns implicit spatial foresight by aligning policy queries with a frozen action-conditioned spatial world model trained offline on scene depth, kinematic hand geometry, and demonstration trajectories, avoiding explicit future prediction; at deployment only the policy is retained, operating on RGB observations, language, and robot state, improving DexJoCo simulation success from $49.4\%$49.4% to $60.5\%$60.5% and achieving $58.3\%$58.3% real-robot success versus $50.0\%$50.0% for the strongest baseline.

Key Contributions

• DexForesight aligns dexterous policy latents with predictive representations from a frozen action-conditioned spatial world model and uses the aligned features to condition action generation, enabling implicit spatial foresight without world-model execution, online depth reconstruction, or explicit future generation at deployment.
• The action-conditioned spatial world model uses offline scene depth, kinematic hand geometry, and approximately 370 hours of demonstration trajectories across 60 tasks and five categories, jointly predicting 3D hand motion and visual dynamics, including occluded finger motion.
• On 11 DexJoCo simulation tasks and three real-robot tasks, Close Box, Insert Battery, and Screw On Lid, DexForesight improves the base policy's success rate from 49.4% to 60.5% and achieves 58.3% real-robot success versus 50.0% for the strongest baseline.

Introduction

Controlling dexterous hands requires anticipating how fingers move relative to objects and scenes, including occluded parts that are invisible to cameras. World action models can learn this predictive spatial foresight, but applying them to dexterous manipulation is challenging because they often depend on online depth reconstruction or explicit future generation at deployment, and prediction targets based only on visible scene surfaces miss occluded finger motion. Prior spatially grounded policies also tend to improve scene geometry rather than model action-conditioned articulated hand dynamics. The authors introduce DexForesight, which decouples an action-conditioned spatial world model from the policy and transfers its predictive representations into policy latents. The world model is trained offline with scene depth, kinematic hand geometry, and future states to jointly predict 3D hand motion and visual changes. After freezing the world model, query alignment lets the policy reuse these features during action generation, so deployment requires only current RGB observations, language, and robot state, with no world-model execution, online depth reconstruction, or explicit future generation.

Dataset

The authors use a real-robot demonstration corpus and a smaller depth-annotated subset for world-model training.

Dataset composition and sources

• Approximately 370 hours of demonstrations across 60 real-robot tasks.
• Tasks are grouped into five categories:
  • Spatial arrangement
  • Assembly and contact
  • Rotation and articulation
  • Deformable objects
  • Tool use
• The data covers varied object placement changes, precise alignment, joint rotation, object shape changes, and sustained contact.
• A 100-hour subset is used for depth-injection training. It covers spatial arrangement, assembly and contact, and rotation and articulation, matching the three real-robot evaluation categories.

Depth processing

• For real-robot demonstrations, scene depth is estimated offline from recorded stereo RGB sequences using StereoAnyVideo.
• For simulation, depth is rendered directly from MuJoCo rollouts.
• OmniVGGT encodes RGB frames paired with corresponding depth maps into geometry tokens.
• Depth is required only for the current scene geometry, not at future timestamps.

Hand geometry and targets

• Hand geometry is represented by fixed points sampled on the hand’s URDF links.
• The points are transformed into the robot base frame using forward kinematics from recorded robot states.
• Each point keeps its link assignment and local coordinates across frames, preserving correspondence as the hand articulates.
• Current and historical hand point clouds are encoded with Point-BERT.
• Future hand-point displacements provide hand-flow prediction targets.
• Visual prediction targets use frozen DINOv3 features extracted from RGB.
• Future control conditioning uses K = 4 slots sampled from four-frame bins across a 16-frame future window.
• No explicit image cropping strategy is described.

How the data is used

• Training proceeds in three stages:
  1. Warmup: jointly optimizes the pretrained backbone and new prediction modules without depth input.
  2. Dynamics pretraining: learns action-conditioned interaction dynamics from RGB observations, language instructions, and histories of hand and robot states across the demonstration corpus.
  3. Depth injection: continues training on the 100-hour depth subset only, adding depth conditioning while keeping the same hand-motion and visual prediction objectives.
• No explicit train/validation/test split or mixture ratio is specified. The earlier stages use the full corpus, while the final stage uses the depth-estimated subset.

Method

The authors introduce DexForesight, a framework that decouples an action-conditioned spatial world model from the policy to ground dexterous manipulation in implicit spatial foresight. The overall architecture and query alignment process are illustrated below.

At any given time step $t$t, the policy receives multi-view RGB images, a language instruction, and the robot state, denoted as $o_t = (I_t, \ell, s_t)$ot​=(It​,ℓ,st​), and outputs an action chunk $A_t$At​. The spatial world model additionally ingests offline scene depth and a short history of hand geometry and robot state, denoted as $c_t$ct​. During training, it also receives a demonstrated control sequence $U_t$Ut​ aligned with future prediction timestamps. The two models produce query features as follows:

Here, $f_{\phi}$fϕ​ encompasses the world-model backbone, query reader, and action-conditioned query adapter, while the policy query reader $q_{\theta}$qθ​ attends to selected Vision-Language Model (VLM) hidden layers collected in $\mathcal{M}_{\theta}$Mθ​. Both outputs contain $N$N query tokens. The demonstrated controls $U_t$Ut​ serve as a training-only dynamics-conditioning variable to resolve which control-dependent future is modeled, and are never provided to the policy at deployment.

To learn spatial foresight, the spatial world model encodes scene observations and hand geometry within a backbone initialized from RynnBrain. Hand geometry is derived by sampling fixed points on the hand's URDF links and transforming them into the robot base frame via forward kinematics using recorded robot states. This preserves correspondence as the hand articulates and specifies positions of occluded surfaces. A Point-BERT encoder processes the current point cloud and a short history of hand motion, feeding these features into the backbone alongside RGB, language, and robot-state history.

For depth fusion, scene depth is estimated offline from stereo RGB sequences using StereoAnyVideo for real-robot demonstrations, or rendered directly from MuJoCo rollouts for simulation. An OmniVGGT encoder pairs RGB frames with their corresponding depth maps to produce geometry tokens $E_t$Et​. These tokens are injected into the backbone via cross-attention after specific layers. Following the cross-layer aggregation principle of Attention Residuals, a token-wise router combines the incoming hidden states with the outputs of all preceding injection layers, allowing the model to learn how much geometry to incorporate and how to combine features from different fusion layers.

Action-conditioned world queries are formed by a set of $N$N learnable queries attending to hidden states from specific backbone layers. To predict future states, the model uses $K=4$K=4 slots with offsets $\delta_k$δk​, sampling one offset from each four-frame bin across a 16-frame future window. The demonstrated control at each offset is encoded with its temporal offset, and the query adapter attends to these action tokens to produce $Z_t^{\mathrm{w}}$Ztw​. These queries condition two prediction experts: the Hand Flow Expert, which predicts displacements of corresponding hand points $F_{t,k} = P_{t+\delta_k} - P_t$Ft,k​=Pt+δk​​−Pt​, and the Visual Dynamics Expert, which predicts changes in frozen DINOv3 image features $\Delta V_{t,k} = \Phi(I_{t+\delta_k}) - \Phi(I_t)$ΔVt,k​=Φ(It+δk​​)−Φ(It​). Both experts predict noise added to their targets, supplemented by a Smooth L1 loss on the recovered clean targets. The joint training objective is:

where the auxiliary context loss reconstructs current features from the queries before action conditioning.

After training the world model, the authors freeze its parameters and use the action-conditioned queries $Z_t^{\mathrm{w}}$Ztw​ to supervise the policy queries $Z_t^{\pi}$Ztπ​. Learnable policy queries attend to selected hidden layers of the policy VLM, and a learned projection maps their outputs into the world-query feature space. These policy queries are projected to the action-context dimension and concatenated with the action expert's original context $C_t$Ct​ to generate actions. The alignment process matches corresponding query features and their pairwise similarities using a token loss and a relation loss. The final policy objective is:

At inference, the policy computes its queries once per observation and reuses the augmented context during action sampling, entirely removing the world model, depth encoder, and future experts.

The training pipeline leverages approximately 370 hours of demonstrations across 60 tasks, categorized into spatial arrangement, assembly and contact, rotation and articulation, deformable objects, and tool use. The data distribution and training stages are summarized below.

Training proceeds in three distinct stages. During the warmup stage, the authors jointly optimize the pretrained backbone and the newly introduced prediction modules to adapt backbone representations to the hand-motion and visual prediction objectives. Next, dynamics pretraining is performed on the full demonstration corpus, where the model learns action-conditioned interaction dynamics from RGB observations, language instructions, and histories of hand and robot states, without depth input. In the final depth-injection stage, depth conditioning is introduced and training continues on a 100-hour subset covering spatial arrangement, assembly and contact, and rotation and articulation. Geometry features are extracted by a frozen encoder and injected into the backbone through trainable fusion modules, while the backbone and prediction modules remain jointly optimized under consistent predictive supervision.

Experiment

The experiments evaluate DexForesight in simulation on 11 DexJoCo tasks and on a real robot for three challenging manipulation tasks, Close Box, Insert Battery, and Screw On Lid, comparing against VLA and world-action baselines. DexForesight outperforms these baselines in both settings while generating actions directly from policy queries with negligible inference overhead and without future prediction at deployment. The results also show that predictive 3D hand-motion information remains accessible in the deployed policy features, that transferring this representation is more effective than attaching direct prediction heads, and that offline depth, cross-layer depth fusion, and 3D hand-motion targets each contribute substantially to performance.

In simulated DexJoCo tasks, DexForesight achieves the highest overall success rate among the compared policies and world-action models, improving over its GR00T N1.7 backbone and the strongest baseline by roughly ten to eleven points. The gains are most pronounced on single-arm tasks, while bimanual performance is also the highest but closer to the best baseline. These results indicate that the approach retains useful predictive structure without future decoding at deployment. DexForesight leads overall simulation success, ahead of Fast-WAM and its GR00T N1.7 backbone. It achieves the highest success in both single-arm and bimanual task groups, with a larger advantage in single-arm tasks.

On real-robot held-out positions, DexForesight reaches the highest overall success rate among all compared methods, improving over the next-best WLA and its own GR00T backbone. It leads or ties on each manipulation task, with particular strength on Close and Insert. Its inference latency remains nearly identical to the QwenVLA policy baseline. DexForesight achieves the highest overall real-robot success, outperforming WLA and GR00T N1.7 while Fast-WAM and Cosmos3 trail substantially. It leads on Close and Insert and ties on Screw, showing the strongest task-level performance across the board. Failures concentrate on spatial challenges such as finger occlusion, partial object occlusion, and lid-box geometry. Inference time increases by only a few milliseconds relative to QwenVLA despite the added predictive training objective.

Frozen deployed policy latents from DexForesight retain more linearly accessible future hand-motion information than the compared action-only and zero-motion baselines. The advantage appears at short horizons and becomes larger as the prediction horizon increases. This indicates that predictive hand-motion structure remains available in policy queries even after the world model and future decoders are removed. DexForesight achieves lower linear-probe 3D displacement error than QwenVLA action-only and GR00T N1.7 at every tested horizon. The gap between DexForesight and the zero-motion prior widens substantially from the shortest to the longest prediction horizon. Future hand-motion information remains linearly readable from frozen DexForesight policy latents without requiring the world model or future decoders at deployment.

DexForesight achieves the highest overall simulated success among the compared variants, outperforming direct predictive additions to GR00T across single-arm and bimanual settings. The strongest direct-prediction control is the hand-flow head, but DexForesight still leads by a clear margin. Ablations show that removing offline depth, replacing cross-layer depth fusion, or replacing hand-motion supervision with future-action targets consistently reduces success. DexForesight reaches 60.5% overall success, exceeding the strongest direct-prediction baseline, the hand-flow head at 53.1%, by 7.4 points. Bimanual performance favors DexForesight: 41.6% versus 37.6% for the hand-flow head and 23.8% for the DINO future-residual head. Directly combining DINO and hand-flow heads yields 49.5% overall, below the hand-flow head alone, indicating that adding prediction heads does not match query alignment. Query-target ablations show lower success with pre-action queries, future-action targets, or shuffled conditioning actions compared to conditioning on demonstrated actions.

The evaluation covers simulated DexJoCo manipulation, real-robot held-out positions, frozen latent probing, and ablations. Simulation and real-robot results show that DexForesight improves overall success over its GR00T N1.7 backbone and other baselines, with the largest gains on single-arm tasks and on Close and Insert, while adding only negligible inference latency. Probing experiments indicate that deployed policy latents retain linearly accessible future hand-motion information without requiring the world model or future decoders. Ablations confirm that query alignment with demonstrated actions, offline depth, cross-layer depth fusion, and hand-motion supervision all contribute, whereas directly adding prediction heads is less effective.