HyperAIHyperAI

Command Palette

Search for a command to run...

HINT-WAM: 월드 액션 모델을 위한 사후 정보 기반 훈련

Phuong Tuan Dat Qi Li Xinchao Wang

초록

월드 액션 모델(WAM)은 행동 생성과 미래 시각 동역학 예측을 결합함으로써 로봇 조작에서 유망한 일반화 성능과 데이터 효율성을 보여 왔다. 그러나 이러한 능력을 신뢰할 수 있는 작업 수행으로 전환하는 것은 기존의 사후 훈련 방법들에게 여전히 어려운 과제로 남아 있다. 오프라인 지도 미세 조정은 전문가 궤적에 대해 조밀한 지도를 제공하지만, 그 플로우 매칭 행동 전문가는 오직 노이즈가 추가된 시연 행동에 대해서만 훈련되며 자신의 샘플러가 생성한 중간 상태에 대해서는 결코 훈련되지 않는다. 또한 작업 특화 적응은 사전 훈련된 능력을 저해할 수 있다. 온폴리시 강화 학습은 정책 자신의 행동에 대해 훈련하지만, 희소한 작업 보상은 탐색과 크레딧 할당을 어렵게 만들며, 조밀한 보상 형성은 직접적인 행동 목표가 아닌 스칼라 평가만을 제공한다. 우리는 HINT-WAM을 소개한다. 이는 WAM 자체가 특권 정보로 활용할 수 있는 사후 정보, 즉 시연의 미래 비디오를 사용하여 행동 전문가가 자신의 샘플링 궤적에 대해 지도받도록 하는 최초의 온폴리시 자기 증류 프레임워크이다. 각 시연 관측에 대해 학생은 자신의 디노이징 궤적을 통해 행동을 생성하고, 이 궤적 상의 각 상태에서 동일한 모델이 사후 정보에 추가로 조건화된 교사 역할을 하여 조건부 행동 분포를 조밀한 훈련 목표로 제공한다. 반면 학생은 배포 시점의 입력만을 관측하며 환경 보상이나 시뮬레이터 상호작용이 필요하지 않다. 이러한 지도를 플로우 매칭 행동 전문가에게 전달하기 위해, HINT-WAM은 에너지 점수를 사용하여 교사를 학생으로 증류한다. 이는 교사의 속도장을 회귀하는 대신 학생 자신의 샘플로부터 교사의 행동 분포를 일치시키는 분포 수준의 목적 함수이다. 프로젝트 웹사이트는 이 링크에서 확인할 수 있다.

One-sentence Summary

Researchers at the National University of Singapore introduce HINT-WAM, the first on-policy self-distillation framework for world action models, which supervises a flow-matching action expert on its own sampling trajectories by using hindsight as privileged information and distills the teacher into the student via an energy score to match the teacher’s action distribution from the student’s own samples rather than regressing its velocity field for reliable manipulation.

Key Contributions

  • HINT-WAM is introduced as the first on-policy self-distillation framework for world action models, with a hindsight teacher sharing the student backbone to provide dense action targets at states from the action expert's own sampling trajectory.
  • The framework uses future demonstration video as privileged hindsight information and distills the flow-matching action expert with an energy score, matching the teacher's action distribution from the student's own samples instead of regressing a velocity field and requiring no external teacher, rewards, or simulator interaction.
  • Experiments show HINT-WAM improves post-training performance over supervised fine-tuning and matched baselines on LIBERO-90 and RoboTwin 2.0, while reducing catastrophic forgetting of pretrained skills and composing with replay.

Introduction

World action models improve robot manipulation by coupling action generation with future visual prediction, and recent flow-matching action experts produce continuous action chunks. Reliable adaptation remains challenging because supervised fine-tuning suffers from exposure bias, since it trains only on demonstration actions rather than the model’s own sampled states, and cannot resolve ambiguous action continuations. On-policy reinforcement learning can address the state mismatch but often struggles with sparse rewards, while existing on-policy distillation methods assume discrete tokens, require a separate teacher, or regress a velocity field that averages over modes. The authors introduce HINT-WAM, an on-policy self-distillation framework in which the policy acts as its own teacher using hindsight future frames, supervises the flow-matching action expert on its own sampler states, and uses an energy-score objective to match the teacher’s action distribution without likelihood evaluation. This approach needs no external teacher, rewards, or simulator interaction, and improves post-training performance on LIBERO-90 and RoboTwin 2.0 while reducing catastrophic forgetting.

Method

The authors present HINT-WAM, a post-training framework that enables a pretrained World Action Model (WAM) to act as its own teacher by leveraging hindsight information. The overall pipeline is divided into three distinct stages: supervised fine-tuning, hindsight teacher training, and on-policy self-distillation.

In the first stage, the authors perform Supervised Fine-Tuning (SFT) to initialize the student model. The base architecture is Fast-WAM, which utilizes a Mixture-of-Transformers backbone with two experts sharing a single attention operation per layer. The video expert processes the current camera frame, while the lightweight action expert generates an action chunk a∈RH×D\mathbf{a} \in \mathbb{R}^{H \times D}a∈RH×D conditioned on the observation o=(I,ℓ,s)o = (I, \ell, s)o=(I,ℓ,s). The action expert operates as a flow-matching model, predicting a velocity vθ(aσ,σ,o)v_\theta(\mathbf{a}_\sigma, \sigma, o)vθ​(aσ​,σ,o) for a noisy action chunk aσ=(1−σ)a+σϵ\mathbf{a}_\sigma = (1 - \sigma)\mathbf{a} + \sigma\boldsymbol{\epsilon}aσ​=(1−σ)a+σϵ. The model is trained by minimizing the flow-matching loss:

LFM(θ)=E(o,a),σ,ϵ∥vθ(aσ,σ,o)−(ϵ−a)∥22.\mathcal{L}_{\mathrm{FM}}(\boldsymbol{\theta}) = \mathbb{E}_{(o, \mathbf{a}), \sigma, \boldsymbol{\epsilon}} \big \| v_{\boldsymbol{\theta}}(\mathbf{a}_{\sigma}, \sigma, o) - (\boldsymbol{\epsilon} - \mathbf{a}) \big \|_{2}^{2}.LFM​(θ)=E(o,a),σ,ϵ​​vθ​(aσ​,σ,o)−(ϵ−a)​22​.

This stage yields the student initialization θ0\theta_0θ0​.

Next, the authors construct the Hindsight Teacher in Stage A. To resolve target ambiguity without altering the base weights, they freeze the SFT model θ0\theta_0θ0​ and introduce two trainable modules: a hindsight encoder and gated cross-attention adapters. The hindsight encoder processes the future video latents of the demonstration. It drops the first latent frame to prevent information leakage, mean-pools the remaining frames over space, and projects the sequence. A set of learned queries then pools this sequence via multi-head attention to produce hindsight tokens ϕψ(z)\phi_\psi(\mathbf{z})ϕψ​(z). These tokens are injected into the action expert through gated cross-attention adapters attached after a subset of layers. The adapter computes:

h′=h+tanh⁡(αl)WupWoutAttn⁡(q,k,v),\mathbf{h}' = \mathbf{h} + \tanh(\alpha_l) W_{\mathrm{up}} W_{\mathrm{out}} \operatorname{Attn}(\mathbf{q}, \mathbf{k}, \mathbf{v}),h′=h+tanh(αl​)Wup​Wout​Attn(q,k,v),

where q\mathbf{q}q is derived from the action-token hidden states, and k,v\mathbf{k}, \mathbf{v}k,v are derived from the hindsight tokens. The entire backbone remains frozen, and only the encoder and adapter parameters ψ\psiψ are trained using the flow-matching loss augmented with hindsight.

Finally, Stage B focuses on On-policy Self-distillation to address exposure bias. The student θ\thetaθ, initialized from θ0\theta_0θ0​, is trained to match the frozen teacher T=(θ0,ψ)T = (\theta_0, \psi)T=(θ0​,ψ) on its own sampling trajectories. The student integrates its own probability-flow ODE without gradients to record the denoising states visited during inference. To supervise these states efficiently, the authors select the top-K points where the local Euler step error is highest and renormalize their weights to wˉk\bar{w}_kwˉk​.

To align the student with the teacher, the authors employ two distillation objectives. The first is velocity matching, which minimizes the difference between the student and teacher velocities at the selected states:

Lvel(θ)=∑k=1KwˉkE∥vθk−sg[vTk]∥22.\mathcal{L}_{\mathrm{vel}}(\theta) = \sum_{k=1}^{K} \bar{w}_k \mathbb{E} \big \| v_{\theta}^k - \mathrm{sg}[v_{T}^k] \big \|_{2}^{2}.Lvel​(θ)=k=1∑K​wˉk​E​vθk​−sg[vTk​]​22​.

Because the teacher behavior is multimodal from the student perspective, matching the conditional mean can be suboptimal. Therefore, the authors also utilize an energy score for distribution matching. The student draws MMM independent trajectories to obtain implied actions a^m,k\hat{\mathbf{a}}_{m,k}a^m,k​, while the teacher draws a single action aT\mathbf{a}_TaT​ using its ODE. The energy score loss is defined as:

LES(θ)=∑k=1Kwˉk[1M∑m=1M∥a^m,k−aT∥2−12M2∑m,m′=1M∥a^m,k−a^m′,k∥2].\mathcal{L}_{\mathrm{ES}}(\theta) = \sum_{k=1}^{K} \bar{w}_k \Big [ \frac{1}{M} \sum_{m=1}^{M} \left\| \hat{\mathbf{a}}_{m,k} - \mathbf{a}_T \right\|_2 - \frac{1}{2M^2} \sum_{m,m'=1}^{M} \left\| \hat{\mathbf{a}}_{m,k} - \hat{\mathbf{a}}_{m',k} \right\|_2 \Big ].LES​(θ)=k=1∑K​wˉk​[M1​m=1∑M​∥a^m,k​−aT​∥2​−2M21​m,m′=1∑M​∥a^m,k​−a^m′,k​∥2​].

This objective pulls the student samples toward the teacher sample while pushing them apart to prevent mode collapse. To prevent the student from blindly following teacher errors, a behavior-cloning anchor LBC(θ)=LFM(θ)\mathcal{L}_{\mathrm{BC}}(\theta) = \mathcal{L}_{\mathrm{FM}}(\theta)LBC​(θ)=LFM​(θ) is added. The final Stage-B objective balances the distillation and behavior-cloning terms:

LB(θ)=λdistLdist(θ)+λBCLBC(θ),\mathcal{L}_B(\theta) = \lambda_{\mathrm{dist}} \mathcal{L}_{\mathrm{dist}}(\theta) + \lambda_{\mathrm{BC}} \mathcal{L}_{\mathrm{BC}}(\theta),LB​(θ)=λdist​Ldist​(θ)+λBC​LBC​(θ),

where Ldist∈{Lvel,LES}\mathcal{L}_{\mathrm{dist}} \in \{\mathcal{L}_{\mathrm{vel}}, \mathcal{L}_{\mathrm{ES}}\}Ldist​∈{Lvel​,LES​}. At deployment, only the student model is utilized, ensuring the policy operates without requiring future observations.

Experiment

The experiments evaluate HINT-WAM on LIBERO-90 and RoboTwin 2.0, starting from a shared Fast-WAM checkpoint and comparing against SFT, full SFT, BC control, self-imitation, and shuffled-hindsight baselines. On unseen LIBERO-90 tasks, hindsight distillation improves over SFT and full SFT while remaining computationally cheaper than full SFT, and on RoboTwin it prevents the degradation caused by continued imitation. Continual learning results show that HINT-WAM forgets less than SFT and composes well with experience replay. Ablations indicate that the energy-score distribution matching, the behavior-cloning anchor, multiple student samples, and on-policy states each contribute, with the anchor and hindsight correction jointly producing the main improvement.

Fine-tuning the action expert improves LIBERO-90 success dramatically over the released Fast-WAM checkpoint, while RoboTwin results remain broadly high across methods. Ablations show that the strongest LIBERO-90 result comes from combining a behavior-cloning anchor with energy-score distribution matching, on-policy states, and multiple student samples. Individual ablations, including removing the anchor or using demonstrated states, fall back toward or below SFT. Fast-WAM shows a large LIBERO-90 gap relative to fine-tuned baselines, but RoboTwin success rates stay close across evaluated methods. Energy-score distribution matching outperforms velocity matching, MMD, and action-space MSE under matched ablation conditions. Combining the behavior-cloning anchor with distillation yields a strong positive interaction, whereas each component alone reduces success. Using the student's own on-policy states outperforms substituting demonstrated states noised like SFT, and more student samples improve success monotonically.

Post-training on LIBERO-90 without experience replay causes severe forgetting on the four original suites for every method, reducing average success from 96.55% to below 45%. HINT-WAM retains the most original-suite performance without replay, while full SFT forgets the most. With experience replay, the gap largely closes and HINT-WAM benefits further, outperforming SFT on both new and original tasks and nearly matching the released checkpoint. Without replay, all methods fall below 45% average success on the original suites after LIBERO-90 training. HINT-WAM preserves the most pretrained skill without replay, while full SFT loses the most. Experience replay largely recovers original-suite performance, and HINT-WAM with replay exceeds SFT on both new and original tasks. The largest replay gain for HINT-WAM appears on the long-horizon suite.

The evaluation covers fine-tuning on LIBERO-90 and RoboTwin, ablations on distillation components, and post-training retention on four original suites. Fine-tuning the action expert greatly improves LIBERO-90 over the released Fast-WAM checkpoint, and the best setup combines a behavior-cloning anchor with energy-score distribution matching, on-policy states, and multiple student samples, while RoboTwin success stays broadly high across methods. Without experience replay, LIBERO-90 post-training causes severe forgetting on original suites for all methods; replay largely restores performance, with HINT-WAM retaining the most pretrained skill and exceeding SFT on new and original tasks.


AI로 AI 구축

아이디어에서 출시까지 — 무료 AI 코코딩, 즉시 사용 가능한 환경, 최적의 GPU 가격으로 AI 개발을 가속화하세요.

AI 협업 코딩
바로 사용 가능한 GPU
최적의 가격

HyperAI Newsletters

최신 정보 구독하기
한국 시간 매주 월요일 오전 9시 에 이번 주의 최신 업데이트를 메일로 발송합니다
이메일 서비스 제공: MailChimp