HyperAIHyperAI

Command Palette

Search for a command to run...

HINT-WAM: Hindsight-informiertes Training für Weltaktionsmodelle

Phuong Tuan Dat Qi Li Xinchao Wang

Zusammenfassung

Weltaktionsmodelle (WAMs) haben vielversprechende Generalisierung und Dateneffizienz in der Robotermanipulation gezeigt, indem sie Aktionsgenerierung mit Vorhersagen zukünftiger visueller Dynamik koppeln. Die Übertragung dieser Fähigkeiten in zuverlässige Aufgabenausführung bleibt jedoch eine Herausforderung für bestehende Post-Training-Methoden. Offline-überwachtes Feintuning bietet dichte Supervision auf Expertentrajektorien, aber sein Flow-Matching-Aktionsexperte wird nur auf verrauschten Demonstrationsaktionen trainiert und niemals auf den Zwischenzuständen, die sein eigener Sampler erzeugt; aufgabenspezifische Anpassung kann zudem vortrainierte Fähigkeiten beeinträchtigen. On-Policy-Verstärkungslernen trainiert auf dem eigenen Verhalten der Policy, doch spärliche Aufgabenbelohnungen erschweren Exploration und Kreditzuweisung, während dichtes Reward-Shaping weiterhin skalare Bewertungen statt direkter Aktionsziele liefert. Wir stellen HINT-WAM vor, das erste On-Policy-Selbstdestillationsframework, das den Aktionsexperten auf seinen eigenen Sampling-Trajektorien supervidiert, indem es Hindsight, das zukünftige Video der Demonstration, als privilegierte Information nutzt, die dem WAM selbst zur Verfügung steht. Für jede Demonstrationsbeobachtung generiert der Student seine Aktion durch seine eigene Denoising-Trajektorie, und an den Zuständen entlang dieser Trajektorie fungiert dasselbe Modell als Lehrer, der zusätzlich auf dieses Hindsight konditioniert ist und eine bedingte Aktionsverteilung als dichtes Trainingsziel bereitstellt, während der Student nur Deployment-Zeit-Eingaben sieht und keine Umgebungsbelohnungen oder Simulatorinteraktion benötigt. Um diese Supervision auf einen Flow-Matching-Aktionsexperten zu übertragen, destilliert HINT-WAM den Lehrer in den Studenten mit einem Energy Score, einem verteilungsbasierten Ziel, das die Aktionsverteilung des Lehrers aus den eigenen Samples des Studenten angleicht, statt sein Geschwindigkeitsfeld zu regressieren. Die Projektwebseite ist unter diesem Link verfügbar.

One-sentence Summary

Researchers at the National University of Singapore introduce HINT-WAM, the first on-policy self-distillation framework for world action models, which supervises a flow-matching action expert on its own sampling trajectories by using hindsight as privileged information and distills the teacher into the student via an energy score to match the teacher’s action distribution from the student’s own samples rather than regressing its velocity field for reliable manipulation.

Key Contributions

  • HINT-WAM is introduced as the first on-policy self-distillation framework for world action models, with a hindsight teacher sharing the student backbone to provide dense action targets at states from the action expert's own sampling trajectory.
  • The framework uses future demonstration video as privileged hindsight information and distills the flow-matching action expert with an energy score, matching the teacher's action distribution from the student's own samples instead of regressing a velocity field and requiring no external teacher, rewards, or simulator interaction.
  • Experiments show HINT-WAM improves post-training performance over supervised fine-tuning and matched baselines on LIBERO-90 and RoboTwin 2.0, while reducing catastrophic forgetting of pretrained skills and composing with replay.

Introduction

World action models improve robot manipulation by coupling action generation with future visual prediction, and recent flow-matching action experts produce continuous action chunks. Reliable adaptation remains challenging because supervised fine-tuning suffers from exposure bias, since it trains only on demonstration actions rather than the model’s own sampled states, and cannot resolve ambiguous action continuations. On-policy reinforcement learning can address the state mismatch but often struggles with sparse rewards, while existing on-policy distillation methods assume discrete tokens, require a separate teacher, or regress a velocity field that averages over modes. The authors introduce HINT-WAM, an on-policy self-distillation framework in which the policy acts as its own teacher using hindsight future frames, supervises the flow-matching action expert on its own sampler states, and uses an energy-score objective to match the teacher’s action distribution without likelihood evaluation. This approach needs no external teacher, rewards, or simulator interaction, and improves post-training performance on LIBERO-90 and RoboTwin 2.0 while reducing catastrophic forgetting.

Method

The authors present HINT-WAM, a post-training framework that enables a pretrained World Action Model (WAM) to act as its own teacher by leveraging hindsight information. The overall pipeline is divided into three distinct stages: supervised fine-tuning, hindsight teacher training, and on-policy self-distillation.

In the first stage, the authors perform Supervised Fine-Tuning (SFT) to initialize the student model. The base architecture is Fast-WAM, which utilizes a Mixture-of-Transformers backbone with two experts sharing a single attention operation per layer. The video expert processes the current camera frame, while the lightweight action expert generates an action chunk a∈RH×D\mathbf{a} \in \mathbb{R}^{H \times D}a∈RH×D conditioned on the observation o=(I,ℓ,s)o = (I, \ell, s)o=(I,ℓ,s). The action expert operates as a flow-matching model, predicting a velocity vθ(aσ,σ,o)v_\theta(\mathbf{a}_\sigma, \sigma, o)vθ​(aσ​,σ,o) for a noisy action chunk aσ=(1−σ)a+σϵ\mathbf{a}_\sigma = (1 - \sigma)\mathbf{a} + \sigma\boldsymbol{\epsilon}aσ​=(1−σ)a+σϵ. The model is trained by minimizing the flow-matching loss:

LFM(θ)=E(o,a),σ,ϵ∥vθ(aσ,σ,o)−(ϵ−a)∥22.\mathcal{L}_{\mathrm{FM}}(\boldsymbol{\theta}) = \mathbb{E}_{(o, \mathbf{a}), \sigma, \boldsymbol{\epsilon}} \big \| v_{\boldsymbol{\theta}}(\mathbf{a}_{\sigma}, \sigma, o) - (\boldsymbol{\epsilon} - \mathbf{a}) \big \|_{2}^{2}.LFM​(θ)=E(o,a),σ,ϵ​​vθ​(aσ​,σ,o)−(ϵ−a)​22​.

This stage yields the student initialization θ0\theta_0θ0​.

Next, the authors construct the Hindsight Teacher in Stage A. To resolve target ambiguity without altering the base weights, they freeze the SFT model θ0\theta_0θ0​ and introduce two trainable modules: a hindsight encoder and gated cross-attention adapters. The hindsight encoder processes the future video latents of the demonstration. It drops the first latent frame to prevent information leakage, mean-pools the remaining frames over space, and projects the sequence. A set of learned queries then pools this sequence via multi-head attention to produce hindsight tokens ϕψ(z)\phi_\psi(\mathbf{z})ϕψ​(z). These tokens are injected into the action expert through gated cross-attention adapters attached after a subset of layers. The adapter computes:

h′=h+tanh⁡(αl)WupWoutAttn⁡(q,k,v),\mathbf{h}' = \mathbf{h} + \tanh(\alpha_l) W_{\mathrm{up}} W_{\mathrm{out}} \operatorname{Attn}(\mathbf{q}, \mathbf{k}, \mathbf{v}),h′=h+tanh(αl​)Wup​Wout​Attn(q,k,v),

where q\mathbf{q}q is derived from the action-token hidden states, and k,v\mathbf{k}, \mathbf{v}k,v are derived from the hindsight tokens. The entire backbone remains frozen, and only the encoder and adapter parameters ψ\psiψ are trained using the flow-matching loss augmented with hindsight.

Finally, Stage B focuses on On-policy Self-distillation to address exposure bias. The student θ\thetaθ, initialized from θ0\theta_0θ0​, is trained to match the frozen teacher T=(θ0,ψ)T = (\theta_0, \psi)T=(θ0​,ψ) on its own sampling trajectories. The student integrates its own probability-flow ODE without gradients to record the denoising states visited during inference. To supervise these states efficiently, the authors select the top-K points where the local Euler step error is highest and renormalize their weights to wˉk\bar{w}_kwˉk​.

To align the student with the teacher, the authors employ two distillation objectives. The first is velocity matching, which minimizes the difference between the student and teacher velocities at the selected states:

Lvel(θ)=∑k=1KwˉkE∥vθk−sg[vTk]∥22.\mathcal{L}_{\mathrm{vel}}(\theta) = \sum_{k=1}^{K} \bar{w}_k \mathbb{E} \big \| v_{\theta}^k - \mathrm{sg}[v_{T}^k] \big \|_{2}^{2}.Lvel​(θ)=k=1∑K​wˉk​E​vθk​−sg[vTk​]​22​.

Because the teacher behavior is multimodal from the student perspective, matching the conditional mean can be suboptimal. Therefore, the authors also utilize an energy score for distribution matching. The student draws MMM independent trajectories to obtain implied actions a^m,k\hat{\mathbf{a}}_{m,k}a^m,k​, while the teacher draws a single action aT\mathbf{a}_TaT​ using its ODE. The energy score loss is defined as:

LES(θ)=∑k=1Kwˉk[1M∑m=1M∥a^m,k−aT∥2−12M2∑m,m′=1M∥a^m,k−a^m′,k∥2].\mathcal{L}_{\mathrm{ES}}(\theta) = \sum_{k=1}^{K} \bar{w}_k \Big [ \frac{1}{M} \sum_{m=1}^{M} \left\| \hat{\mathbf{a}}_{m,k} - \mathbf{a}_T \right\|_2 - \frac{1}{2M^2} \sum_{m,m'=1}^{M} \left\| \hat{\mathbf{a}}_{m,k} - \hat{\mathbf{a}}_{m',k} \right\|_2 \Big ].LES​(θ)=k=1∑K​wˉk​[M1​m=1∑M​∥a^m,k​−aT​∥2​−2M21​m,m′=1∑M​∥a^m,k​−a^m′,k​∥2​].

This objective pulls the student samples toward the teacher sample while pushing them apart to prevent mode collapse. To prevent the student from blindly following teacher errors, a behavior-cloning anchor LBC(θ)=LFM(θ)\mathcal{L}_{\mathrm{BC}}(\theta) = \mathcal{L}_{\mathrm{FM}}(\theta)LBC​(θ)=LFM​(θ) is added. The final Stage-B objective balances the distillation and behavior-cloning terms:

LB(θ)=λdistLdist(θ)+λBCLBC(θ),\mathcal{L}_B(\theta) = \lambda_{\mathrm{dist}} \mathcal{L}_{\mathrm{dist}}(\theta) + \lambda_{\mathrm{BC}} \mathcal{L}_{\mathrm{BC}}(\theta),LB​(θ)=λdist​Ldist​(θ)+λBC​LBC​(θ),

where Ldist∈{Lvel,LES}\mathcal{L}_{\mathrm{dist}} \in \{\mathcal{L}_{\mathrm{vel}}, \mathcal{L}_{\mathrm{ES}}\}Ldist​∈{Lvel​,LES​}. At deployment, only the student model is utilized, ensuring the policy operates without requiring future observations.

Experiment

The experiments evaluate HINT-WAM on LIBERO-90 and RoboTwin 2.0, starting from a shared Fast-WAM checkpoint and comparing against SFT, full SFT, BC control, self-imitation, and shuffled-hindsight baselines. On unseen LIBERO-90 tasks, hindsight distillation improves over SFT and full SFT while remaining computationally cheaper than full SFT, and on RoboTwin it prevents the degradation caused by continued imitation. Continual learning results show that HINT-WAM forgets less than SFT and composes well with experience replay. Ablations indicate that the energy-score distribution matching, the behavior-cloning anchor, multiple student samples, and on-policy states each contribute, with the anchor and hindsight correction jointly producing the main improvement.

Fine-tuning the action expert improves LIBERO-90 success dramatically over the released Fast-WAM checkpoint, while RoboTwin results remain broadly high across methods. Ablations show that the strongest LIBERO-90 result comes from combining a behavior-cloning anchor with energy-score distribution matching, on-policy states, and multiple student samples. Individual ablations, including removing the anchor or using demonstrated states, fall back toward or below SFT. Fast-WAM shows a large LIBERO-90 gap relative to fine-tuned baselines, but RoboTwin success rates stay close across evaluated methods. Energy-score distribution matching outperforms velocity matching, MMD, and action-space MSE under matched ablation conditions. Combining the behavior-cloning anchor with distillation yields a strong positive interaction, whereas each component alone reduces success. Using the student's own on-policy states outperforms substituting demonstrated states noised like SFT, and more student samples improve success monotonically.

Post-training on LIBERO-90 without experience replay causes severe forgetting on the four original suites for every method, reducing average success from 96.55% to below 45%. HINT-WAM retains the most original-suite performance without replay, while full SFT forgets the most. With experience replay, the gap largely closes and HINT-WAM benefits further, outperforming SFT on both new and original tasks and nearly matching the released checkpoint. Without replay, all methods fall below 45% average success on the original suites after LIBERO-90 training. HINT-WAM preserves the most pretrained skill without replay, while full SFT loses the most. Experience replay largely recovers original-suite performance, and HINT-WAM with replay exceeds SFT on both new and original tasks. The largest replay gain for HINT-WAM appears on the long-horizon suite.

The evaluation covers fine-tuning on LIBERO-90 and RoboTwin, ablations on distillation components, and post-training retention on four original suites. Fine-tuning the action expert greatly improves LIBERO-90 over the released Fast-WAM checkpoint, and the best setup combines a behavior-cloning anchor with energy-score distribution matching, on-policy states, and multiple student samples, while RoboTwin success stays broadly high across methods. Without experience replay, LIBERO-90 post-training causes severe forgetting on original suites for all methods; replay largely restores performance, with HINT-WAM retaining the most pretrained skill and exceeding SFT on new and original tasks.


KI mit KI entwickeln

Von der Idee bis zum Launch – beschleunigen Sie Ihre KI-Entwicklung mit kostenlosem KI-Co-Coding, sofort einsatzbereiter Umgebung und bestem GPU-Preis.

KI-gestütztes kollaboratives Programmieren
Sofort einsatzbereite GPUs
Die besten Preise

HyperAI Newsletters

Abonnieren Sie unsere neuesten Updates
Wir werden die neuesten Updates der Woche in Ihren Posteingang liefern um neun Uhr jeden Montagmorgen
Unterstützt von MailChimp