One-sentence Summary

Researchers from Nanjing University of Science and Technology, National University of Singapore, Shenyang Institute of Automation, Chinese Academy of Sciences, and The Chinese University of Hong Kong propose HiFOR, a hierarchical foresight framework that jointly predicts short-term state evolution, semantic milestones, and final goal states through a Hierarchical Dependency Mask, and that combines structure-aware latent world modeling with action-guided factorized world retrieval to provide targeted guidance for action generation, as validated on two simulation benchmarks and six real-world robotic manipulation tasks.

Key Contributions

• The paper introduces HiFOR, a hierarchical foresight framework that jointly predicts short-term state evolution, semantic milestones, and final goal states through a Hierarchical Dependency Mask, linking local world dynamics with long-term task objectives.
• The paper develops a structure-aware latent world modeling mechanism that organizes predicted futures along horizon, view, and spatial dimensions, and an action-guided factorized world retrieval mechanism that retrieves action-relevant future information based on evolving action representations during denoising.
• Experiments on LIBERO, RoboTwin 2.0, and six real-world robotic manipulation tasks demonstrate HiFOR's effectiveness, and ablation studies validate the contributions of its key components.

Introduction

Vision-language-action (VLA) models have become central to robotic manipulation by unifying visual perception, language understanding, and action generation, but complex tasks increasingly require policies that can anticipate future environmental changes and task progression. Prior foresight-based methods often model either short-term dynamics or long-term goals in isolation, and they typically inject predicted future states, latents, images, or videos into action policies without selectively weighting their relevance to the current action. This limits coherent temporal reasoning and fails to account for how the importance of future information varies across prediction horizons, camera views, and spatial regions. The authors propose HiFOR, a hierarchical foresight framework that jointly predicts short-term state evolution, semantic milestones, and final goals, organizes these predictions into a structured latent world, and applies action-guided factorized retrieval during action denoising to adaptively select the most relevant future information for decision making.

Dataset

The authors construct a hierarchical planning dataset from existing successful expert demonstrations, without collecting additional robot trajectories. The goal is to replace uniform future-frame sampling with semantically meaningful milestone supervision for vision-language-action training.

• Sources and composition: existing successful expert demonstration trajectories; no new robot data collection is performed.
• Annotation schema: For each language instruction, a Planning Agent generates an ordered sequence of subgoals ${G_1, G_2, ..., G_K}$G1​,G2​,...,GK​. Each subgoal specifies a semantic stage, an event type, and a completion verification criterion.
• Timestamp localization: The method automatically finds completion timestamps for each subgoal in expert demonstrations. Candidate event boundaries are identified from gripper transitions, robot motion phases, and state information.
• Event handling: Stable-state events, such as grasping and placement, use the earliest frame where the target condition remains persistently satisfied. Dynamic events are localized using motion or state transition boundaries.
• Quality control: Temporal consistency is enforced as $t_1 < t_2 < ... < t_K$t1​<t2​<...<tK​. Confidence scores are assigned to milestones, and ambiguous or low-confidence cases are flagged for manual review. Annotations are stored as $\mathcal{M} = \{(G_k, t_k)\}_{k=1}^K$M={(Gk​,tk​)}k=1K​.
• Visual supervision: Multiview frames are encoded with a frozen DINOv2 ViT-L/14 model. Prefix tokens are removed, and patch features are adaptively pooled into a 4x4 spatial grid, producing $F_t^v = \phi_{\mathrm{DINO}}(I_t^v) \in \mathbb{R}^{16 \times 1024}$Ftv​=ϕDINO​(Itv​)∈R16×1024.
• Use in the model: The milestone annotations and DINOv2 visual features provide structured future supervision for hierarchical foresight prediction during VLA training.
• Reported scale and splits: The provided excerpt does not specify dataset size, training split ratios, or mixture weights.

Method

The authors formulate robotic manipulation with Vision-Language-Action (VLA) models as a language-conditioned sequential decision-making problem. At each timestep $t$t, the policy receives multi-view visual observations $\mathcal{O}_t = \{I_t^v\}_{v=1}^V$Ot​={Itv​}v=1V​, a language instruction $\pmb{L}$L, and the robot state $\pmb{s}_t$st​, predicting a future action chunk:

$$\pmb{A}_t = \{\pmb{a}_{t+h}\}_{h=0}^{H-1} = \pi_\theta(\mathcal{O}_t, \pmb{L}, \pmb{s}_t)$$At​={at+h​}h=0H−1​=πθ​(Ot​,L,st​)

where each action consists of relative end-effector translation, rotation, and gripper control.

The overall framework of the proposed method, HiFOR, is illustrated below.

Given the multimodal inputs, a VLA backbone encodes the current context to generate action representations. The authors then introduce Hierarchical Foresight Prediction to jointly predict short-term state evolution, intermediate semantic milestones, and final goal states. Building upon these predictions, Structure-Aware Latent World Modeling organizes the future latent representations along the horizon, view, and spatial dimensions. Finally, Action-Guided Factorized World Retrieval selectively retrieves action-relevant future information to guide the action expert.

To provide structured future supervision for VLA training without collecting additional trajectories, the authors construct a hierarchical planning dataset from existing expert demonstrations.

As shown in the figure above, a Planning Agent first generates a task-specific semantic template from the language instruction $\pmb{L}$L, decomposing it into an ordered sequence of subgoals $\mathcal{G} = \{G_1, G_2, \ldots, G_K\}$G={G1​,G2​,…,GK​}. Each subgoal specifies a semantic stage, event type, and completion criterion. Given this template, the system automatically localizes the completion timestamps of each subgoal in expert demonstrations by identifying candidate event boundaries using gripper transitions, robot motion phases, and state information. Temporal consistency is enforced, and confidence scores are assigned. For future visual supervision, multi-view semantic features are extracted from all trajectory frames using a frozen DINOv2 ViT-L/14 encoder. After removing prefix tokens, the patch features are adaptively pooled into a $4 \times 4$4×4 spatial grid:

$$\pmb{F}_t^v = \phi_{\text{DINO}}(\pmb{I}_t^v) \in \mathbb{R}^{16 \times 1024}$$Ftv​=ϕDINO​(Itv​)∈R16×1024

The Hierarchical Foresight Prediction module defines three complementary prediction horizons for each timestep $t < t_K$t<tK​: the short-term target $t_t^s = \min(t + \Delta, t_K)$tts​=min(t+Δ,tK​), the milestone target $t_t^m = \min_{k: t_k > t} t_k$ttm​=mink:tk​>t​tk​, and the goal target $t_t^g = t_K$ttg​=tK​. Corresponding visual supervision $\pmb{Z}_t^h \in \mathbb{R}^{V \times S \times D}$Zth​∈RV×S×D is constructed for each horizon $h \in \{s, m, g\}$h∈{s,m,g}. Short-term prediction additionally incorporates the future robot state sequence $\pmb{S}_t^s = (s_{t+1}, s_{t+2}, \ldots, s_{t_t^s})$Sts​=(st+1​,st+2​,…,stts​​) to constrain fine-grained motion dynamics.

To predict these targets, the current observations and instruction are encoded into visual tokens $\pmb{V}_t$Vt​ and language tokens $\pmb{E}_L$EL​. Three groups of learnable future queries, $\pmb{Q}^s, \pmb{Q}^m$Qs,Qm, and $\pmb{Q}^g$Qg, are introduced. A Hierarchical Dependency Mask (HDM) progressively expands attention visibility from short-term to goal queries, establishing cross-horizon dependencies:

$$\pmb{U}_t^h = \mathcal{H}_\theta \left(\pmb{Q}^h; [\pmb{V}_t; \pmb{E}_L; \pmb{Q}^{\le h}]\right), \quad h \in \{s, m, g\}$$Uth​=Hθ​(Qh;[Vt​;EL​;Q≤h]),h∈{s,m,g}

where $\pmb{Q}^{\le h}$Q≤h represents the concatenation of query groups up to horizon $h$h. The predicted features are projected into the DINOv2 feature space via visual prediction heads $\hat{\pmb{Z}}_t^h = \mathcal{P}_{\text{vis}}^h(\pmb{U}_t^h)$Z^th​=Pvish​(Uth​), and an additional state prediction head estimates $\hat{\pmb{S}}_t^s = \mathcal{P}_{\text{state}}(\pmb{U}_t^s)$S^ts​=Pstate​(Uts​). The foresight prediction loss is defined as:

$$\mathcal{L}_{\text{HFP}} = \lambda_m \mathcal{D}_{\text{v}}(\hat{\pmb{Z}}_t^h, \pmb{Z}_t^h) + \lambda_n \mathcal{D}_{\text{s}}(\hat{\pmb{S}}_t^s, \pmb{S}_t^s)$$LHFP​=λm​Dv​(Z^th​,Zth​)+λn​Ds​(S^ts​,Sts​)

To effectively utilize these multi-horizon predictions, the authors propose Structure-Aware Latent World Modeling. The predicted future features are organized into structured world tokens through feature projection and structural positional embeddings:

$$\pmb{w}_t^{h,v,p} = \mathcal{P}_{\text{w}}(\hat{\pmb{Z}}_t^h[v,p,:]) + \pmb{e}_h^{\text{hor}} + \pmb{e}_v^{\text{view}} + \pmb{e}_p^{\text{spa}}$$wth,v,p​=Pw​(Z^th​[v,p,:])+ehhor​+evview​+epspa​

These tokens form a unified structured latent world representation $\pmb{W}_t \in \mathbb{R}^{3 \times V \times S \times d_w}$Wt​∈R3×V×S×dw​, explicitly preserving horizon, view, and spatial dimensions. Horizon-level and view-level features are further derived through spatial aggregation to support different granularities.

Finally, Action-Guided Factorized World Retrieval (AWR) is introduced to selectively retrieve future information. Since the relevance of future information varies across denoising steps, dynamic retrieval queries $\pmb{q}_{t,\tau}^a = \mathcal{Q}(\pmb{H}_{t,\tau}^a)$qt,τa​=Q(Ht,τa​) are constructed from the action expert's hidden representations at each denoising step $\tau$τ. Independent linear projections generate queries for temporal horizons, camera views, and spatial locations. A unified factorized attention operation is then formulated across these dimensions:

$$a_{t,\tau}^\ell(i|c) = \text{Softmax}_i \left(\frac{(\pmb{q}_{t,\tau}^\ell)^\top \pmb{k}_t^\ell(i|c)}{\sqrt{d}}\right), \quad \ell \in \{H, V, S\}$$at,τℓ​(i∣c)=Softmaxi​(d​(qt,τℓ​)⊤ktℓ​(i∣c)​),ℓ∈{H,V,S}

The resulting attention weights are combined to aggregate the latent world values, yielding the action-relevant future representation $\pmb{f}_{t,\tau}$ft,τ​. As the action-conditioned queries evolve, the model dynamically adjusts its attention across prediction horizons, camera views, and spatial regions. The retrieved features are fused with action representations and injected into the action expert via residual connections.

The overall training objective jointly optimizes action generation and hierarchical foresight prediction through the flow-matching loss $\mathcal{L}_{\text{ACT}}$LACT​ and the foresight prediction loss $\mathcal{L}_{\text{HFP}}$LHFP​:

$$\mathcal{L} = \mathcal{L}_{\text{ACT}} + \mathcal{L}_{\text{HFP}}$$L=LACT​+LHFP​

Experiment

The evaluation spans simulation benchmarks on LIBERO and RoboTwin 2.0, real-world manipulation tasks with a single-arm robot, ablation studies, and qualitative attention visualization. Across these settings, HiFOR consistently outperforms the baseline, with particularly notable gains on long-horizon and complex manipulation tasks. Ablations show that hierarchical foresight prediction and action-guided retrieval provide complementary benefits, while attention analysis indicates the model adaptively shifts focus across temporal horizons and camera views depending on the task stage.

On the RoboTwin 2.0 Hard benchmark, HiFOR achieves the highest average success rate of 49.40% across 20 tasks, exceeding the pi0.5 baseline by 9.05 percentage points. Other reported methods are substantially lower, with Spatial Forcing averaging 29.40%, X-VLA 23.25%, and EventVLA 19.00%. Task-level gains are especially strong on Blocks Ranking RGB, Place Cans Plasticbox, Move Playingcard, and Stack Blocks Two. HiFOR attains the best overall average success rate of 49.40% across 20 RoboTwin 2.0 Hard tasks. It outperforms the pi0.5 baseline by 9.05 percentage points. Task-level improvements include Blocks Ranking RGB rising from 29% to 46% and Place Cans Plasticbox from 40% to 63%. HiFOR surpasses the baseline on Move Playingcard and Stack Blocks Two by 14% and 13%, respectively.

On LIBERO, HiFOR attains the highest overall average success rate at 98.3%, with leading results on the Spatial, Goal, and Long task suites. Long-horizon tasks show the largest relative gain, improving by 3.4 percentage points over the cited baseline. Prior vision-language-action methods such as OpenVLA-OFT and MemoryVLA are competitive, but HiFOR remains stronger overall and on long-horizon manipulation. HiFOR records the best overall LIBERO average and leads on Spatial, Goal, and Long suites while remaining competitive on Object tasks. Long-horizon performance differentiates HiFOR, reaching 95.8% on the Long suite compared with much lower scores from earlier methods such as OpenVLA and Diffusion Policy. The largest improvement over the cited baseline is on long-horizon tasks, with a gain of 3.4 percentage points. OpenVLA-OFT and MemoryVLA are among the stronger prior methods, but their overall averages trail HiFOR.

Ablations show that each core component adds successive gains in manipulation success, with the largest improvements on long-horizon LIBERO and RoboTwin 2.0. Short-term foresight provides an initial boost, milestone and goal predictions add further accuracy, and action-guided world retrieval contributes an additional gain. Semantic feature-based future prediction also outperforms direct RGB pixel prediction, especially on long-horizon tasks. Adding short-term foresight improves RoboTwin 2.0 success markedly over the baseline, and subsequent milestone, goal, and retrieval components each provide further gains. On LIBERO-Long, the full model improves success by several points over the baseline, with semantic feature-based prediction surpassing RGB-based prediction.

HiFOR is evaluated on RoboTwin 2.0 Hard and LIBERO benchmarks, where it achieves the highest average success rates of 49.40% and 98.3%, respectively. It outperforms the pi0.5 baseline by 9.05 percentage points on RoboTwin, with notable gains on tasks such as Blocks Ranking RGB and Place Cans Plasticbox, and shows the largest relative improvement on LIBERO long-horizon tasks. Ablation studies confirm that short-term foresight, milestone and goal prediction, and action-guided retrieval each contribute successive gains, while semantic feature-based future prediction outperforms direct RGB pixel prediction, especially for long-horizon manipulation.