HyperAIHyperAI

Command Palette

Search for a command to run...

HINT-WAM:世界行動モデルのための後知恵情報に基づく訓練

Phuong Tuan Dat Qi Li Xinchao Wang

概要

世界行動モデル(WAM)は、行動生成と将来の視覚的ダイナミクスの予測を結合することにより、ロボット操作において有望な汎化性能とデータ効率を示してきた。しかし、これらの能力を信頼性の高いタスク実行へと転換することは、既存の事後訓練手法にとって依然として課題である。オフラインの教師ありファインチューニングは専門家軌道に対して密な教師信号を提供するが、そのフローマッチング行動専門家はノイズが付加されたデモンストレーション行動のみで訓練され、自身のサンプラーが生成する中間状態では決して訓練されない。また、タスク固有の適応は事前訓練済みの能力を損なう可能性がある。オン方策強化学習は方策自身の挙動で訓練するが、スパースなタスク報酬では探索と功劳割当てが困難であり、密な報酬形成も直接的な行動目標ではなくスカラー評価しか提供しない。我々はHINT-WAMを導入する。これは、WAM自身が利用可能な特権情報である後知恵、すなわちデモンストレーションの将来映像を用いて、行動専門家を自身のサンプリング軌道上で教師付ける、初のオン方策自己蒸留フレームワークである。各デモンストレーション観測に対して、生徒は自身のノイズ除去軌道を通じて行動を生成し、この軌道上の各状態において、同じモデルがこの後知恵を追加条件として教師として機能し、条件付き行動分布を密な訓練目標として提供する。一方、生徒は展開時の入力のみを観測し、環境報酬やシミュレータとの相互作用を必要としない。この教師信号をフローマッチング行動専門家へ転移するために、HINT-WAMはエネルギースコアを用いて教師を生徒へ蒸留する。これは、教師の速度場を回帰するのではなく、生徒自身のサンプルから教師の行動分布に一致させる分布レベルの目的関数である。プロジェクトのウェブページはこのリンクから利用可能である。

One-sentence Summary

Researchers at the National University of Singapore introduce HINT-WAM, the first on-policy self-distillation framework for world action models, which supervises a flow-matching action expert on its own sampling trajectories by using hindsight as privileged information and distills the teacher into the student via an energy score to match the teacher’s action distribution from the student’s own samples rather than regressing its velocity field for reliable manipulation.

Key Contributions

  • HINT-WAM is introduced as the first on-policy self-distillation framework for world action models, with a hindsight teacher sharing the student backbone to provide dense action targets at states from the action expert's own sampling trajectory.
  • The framework uses future demonstration video as privileged hindsight information and distills the flow-matching action expert with an energy score, matching the teacher's action distribution from the student's own samples instead of regressing a velocity field and requiring no external teacher, rewards, or simulator interaction.
  • Experiments show HINT-WAM improves post-training performance over supervised fine-tuning and matched baselines on LIBERO-90 and RoboTwin 2.0, while reducing catastrophic forgetting of pretrained skills and composing with replay.

Introduction

World action models improve robot manipulation by coupling action generation with future visual prediction, and recent flow-matching action experts produce continuous action chunks. Reliable adaptation remains challenging because supervised fine-tuning suffers from exposure bias, since it trains only on demonstration actions rather than the model’s own sampled states, and cannot resolve ambiguous action continuations. On-policy reinforcement learning can address the state mismatch but often struggles with sparse rewards, while existing on-policy distillation methods assume discrete tokens, require a separate teacher, or regress a velocity field that averages over modes. The authors introduce HINT-WAM, an on-policy self-distillation framework in which the policy acts as its own teacher using hindsight future frames, supervises the flow-matching action expert on its own sampler states, and uses an energy-score objective to match the teacher’s action distribution without likelihood evaluation. This approach needs no external teacher, rewards, or simulator interaction, and improves post-training performance on LIBERO-90 and RoboTwin 2.0 while reducing catastrophic forgetting.

Method

The authors present HINT-WAM, a post-training framework that enables a pretrained World Action Model (WAM) to act as its own teacher by leveraging hindsight information. The overall pipeline is divided into three distinct stages: supervised fine-tuning, hindsight teacher training, and on-policy self-distillation.

In the first stage, the authors perform Supervised Fine-Tuning (SFT) to initialize the student model. The base architecture is Fast-WAM, which utilizes a Mixture-of-Transformers backbone with two experts sharing a single attention operation per layer. The video expert processes the current camera frame, while the lightweight action expert generates an action chunk a∈RH×D\mathbf{a} \in \mathbb{R}^{H \times D}a∈RH×D conditioned on the observation o=(I,ℓ,s)o = (I, \ell, s)o=(I,ℓ,s). The action expert operates as a flow-matching model, predicting a velocity vθ(aσ,σ,o)v_\theta(\mathbf{a}_\sigma, \sigma, o)vθ​(aσ​,σ,o) for a noisy action chunk aσ=(1−σ)a+σϵ\mathbf{a}_\sigma = (1 - \sigma)\mathbf{a} + \sigma\boldsymbol{\epsilon}aσ​=(1−σ)a+σϵ. The model is trained by minimizing the flow-matching loss:

LFM(θ)=E(o,a),σ,ϵ∥vθ(aσ,σ,o)−(ϵ−a)∥22.\mathcal{L}_{\mathrm{FM}}(\boldsymbol{\theta}) = \mathbb{E}_{(o, \mathbf{a}), \sigma, \boldsymbol{\epsilon}} \big \| v_{\boldsymbol{\theta}}(\mathbf{a}_{\sigma}, \sigma, o) - (\boldsymbol{\epsilon} - \mathbf{a}) \big \|_{2}^{2}.LFM​(θ)=E(o,a),σ,ϵ​​vθ​(aσ​,σ,o)−(ϵ−a)​22​.

This stage yields the student initialization θ0\theta_0θ0​.

Next, the authors construct the Hindsight Teacher in Stage A. To resolve target ambiguity without altering the base weights, they freeze the SFT model θ0\theta_0θ0​ and introduce two trainable modules: a hindsight encoder and gated cross-attention adapters. The hindsight encoder processes the future video latents of the demonstration. It drops the first latent frame to prevent information leakage, mean-pools the remaining frames over space, and projects the sequence. A set of learned queries then pools this sequence via multi-head attention to produce hindsight tokens ϕψ(z)\phi_\psi(\mathbf{z})ϕψ​(z). These tokens are injected into the action expert through gated cross-attention adapters attached after a subset of layers. The adapter computes:

h′=h+tanh⁡(αl)WupWoutAttn⁡(q,k,v),\mathbf{h}' = \mathbf{h} + \tanh(\alpha_l) W_{\mathrm{up}} W_{\mathrm{out}} \operatorname{Attn}(\mathbf{q}, \mathbf{k}, \mathbf{v}),h′=h+tanh(αl​)Wup​Wout​Attn(q,k,v),

where q\mathbf{q}q is derived from the action-token hidden states, and k,v\mathbf{k}, \mathbf{v}k,v are derived from the hindsight tokens. The entire backbone remains frozen, and only the encoder and adapter parameters ψ\psiψ are trained using the flow-matching loss augmented with hindsight.

Finally, Stage B focuses on On-policy Self-distillation to address exposure bias. The student θ\thetaθ, initialized from θ0\theta_0θ0​, is trained to match the frozen teacher T=(θ0,ψ)T = (\theta_0, \psi)T=(θ0​,ψ) on its own sampling trajectories. The student integrates its own probability-flow ODE without gradients to record the denoising states visited during inference. To supervise these states efficiently, the authors select the top-K points where the local Euler step error is highest and renormalize their weights to wˉk\bar{w}_kwˉk​.

To align the student with the teacher, the authors employ two distillation objectives. The first is velocity matching, which minimizes the difference between the student and teacher velocities at the selected states:

Lvel(θ)=∑k=1KwˉkE∥vθk−sg[vTk]∥22.\mathcal{L}_{\mathrm{vel}}(\theta) = \sum_{k=1}^{K} \bar{w}_k \mathbb{E} \big \| v_{\theta}^k - \mathrm{sg}[v_{T}^k] \big \|_{2}^{2}.Lvel​(θ)=k=1∑K​wˉk​E​vθk​−sg[vTk​]​22​.

Because the teacher behavior is multimodal from the student perspective, matching the conditional mean can be suboptimal. Therefore, the authors also utilize an energy score for distribution matching. The student draws MMM independent trajectories to obtain implied actions a^m,k\hat{\mathbf{a}}_{m,k}a^m,k​, while the teacher draws a single action aT\mathbf{a}_TaT​ using its ODE. The energy score loss is defined as:

LES(θ)=∑k=1Kwˉk[1M∑m=1M∥a^m,k−aT∥2−12M2∑m,m′=1M∥a^m,k−a^m′,k∥2].\mathcal{L}_{\mathrm{ES}}(\theta) = \sum_{k=1}^{K} \bar{w}_k \Big [ \frac{1}{M} \sum_{m=1}^{M} \left\| \hat{\mathbf{a}}_{m,k} - \mathbf{a}_T \right\|_2 - \frac{1}{2M^2} \sum_{m,m'=1}^{M} \left\| \hat{\mathbf{a}}_{m,k} - \hat{\mathbf{a}}_{m',k} \right\|_2 \Big ].LES​(θ)=k=1∑K​wˉk​[M1​m=1∑M​∥a^m,k​−aT​∥2​−2M21​m,m′=1∑M​∥a^m,k​−a^m′,k​∥2​].

This objective pulls the student samples toward the teacher sample while pushing them apart to prevent mode collapse. To prevent the student from blindly following teacher errors, a behavior-cloning anchor LBC(θ)=LFM(θ)\mathcal{L}_{\mathrm{BC}}(\theta) = \mathcal{L}_{\mathrm{FM}}(\theta)LBC​(θ)=LFM​(θ) is added. The final Stage-B objective balances the distillation and behavior-cloning terms:

LB(θ)=λdistLdist(θ)+λBCLBC(θ),\mathcal{L}_B(\theta) = \lambda_{\mathrm{dist}} \mathcal{L}_{\mathrm{dist}}(\theta) + \lambda_{\mathrm{BC}} \mathcal{L}_{\mathrm{BC}}(\theta),LB​(θ)=λdist​Ldist​(θ)+λBC​LBC​(θ),

where Ldist∈{Lvel,LES}\mathcal{L}_{\mathrm{dist}} \in \{\mathcal{L}_{\mathrm{vel}}, \mathcal{L}_{\mathrm{ES}}\}Ldist​∈{Lvel​,LES​}. At deployment, only the student model is utilized, ensuring the policy operates without requiring future observations.

Experiment

The experiments evaluate HINT-WAM on LIBERO-90 and RoboTwin 2.0, starting from a shared Fast-WAM checkpoint and comparing against SFT, full SFT, BC control, self-imitation, and shuffled-hindsight baselines. On unseen LIBERO-90 tasks, hindsight distillation improves over SFT and full SFT while remaining computationally cheaper than full SFT, and on RoboTwin it prevents the degradation caused by continued imitation. Continual learning results show that HINT-WAM forgets less than SFT and composes well with experience replay. Ablations indicate that the energy-score distribution matching, the behavior-cloning anchor, multiple student samples, and on-policy states each contribute, with the anchor and hindsight correction jointly producing the main improvement.

Fine-tuning the action expert improves LIBERO-90 success dramatically over the released Fast-WAM checkpoint, while RoboTwin results remain broadly high across methods. Ablations show that the strongest LIBERO-90 result comes from combining a behavior-cloning anchor with energy-score distribution matching, on-policy states, and multiple student samples. Individual ablations, including removing the anchor or using demonstrated states, fall back toward or below SFT. Fast-WAM shows a large LIBERO-90 gap relative to fine-tuned baselines, but RoboTwin success rates stay close across evaluated methods. Energy-score distribution matching outperforms velocity matching, MMD, and action-space MSE under matched ablation conditions. Combining the behavior-cloning anchor with distillation yields a strong positive interaction, whereas each component alone reduces success. Using the student's own on-policy states outperforms substituting demonstrated states noised like SFT, and more student samples improve success monotonically.

Post-training on LIBERO-90 without experience replay causes severe forgetting on the four original suites for every method, reducing average success from 96.55% to below 45%. HINT-WAM retains the most original-suite performance without replay, while full SFT forgets the most. With experience replay, the gap largely closes and HINT-WAM benefits further, outperforming SFT on both new and original tasks and nearly matching the released checkpoint. Without replay, all methods fall below 45% average success on the original suites after LIBERO-90 training. HINT-WAM preserves the most pretrained skill without replay, while full SFT loses the most. Experience replay largely recovers original-suite performance, and HINT-WAM with replay exceeds SFT on both new and original tasks. The largest replay gain for HINT-WAM appears on the long-horizon suite.

The evaluation covers fine-tuning on LIBERO-90 and RoboTwin, ablations on distillation components, and post-training retention on four original suites. Fine-tuning the action expert greatly improves LIBERO-90 over the released Fast-WAM checkpoint, and the best setup combines a behavior-cloning anchor with energy-score distribution matching, on-policy states, and multiple student samples, while RoboTwin success stays broadly high across methods. Without experience replay, LIBERO-90 post-training causes severe forgetting on original suites for all methods; replay largely restores performance, with HINT-WAM retaining the most pretrained skill and exceeding SFT on new and original tasks.


AIでAIを構築

アイデアからローンチまで — 無料のAIコーディング支援、すぐに使える環境、最高のGPU価格でAI開発を加速。

AI コーディング補助
すぐに使える GPU
最適な料金体系

HyperAI Newsletters

最新情報を購読する
北京時間 毎週月曜日の午前9時 に、その週の最新情報をメールでお届けします
メール配信サービスは MailChimp によって提供されています