One-sentence Summary

Researchers from NVIDIA, Stanford University, and The University of Texas at Austin propose RoboTTT, a robot foundation model that scales visuomotor context to $8K$8K timesteps without growing inference latency by integrating Test-Time Training to update fast weights with gradient descent during both training and inference, thereby compressing histories into weight space for long-context conditioning and unlocking one-shot incontext imitation from human video demonstrations, on-the-fly policy improvement, and robust completion of multi-stage, long-horizon tasks, achieving an $87\%$87% improvement over single-step context baselines and a further $62\%$62% gain when scaling pretraining context from $1K$1K to 8K timesteps, establishing context length as a new scaling axis for robot policies.

Key Contributions

• RoboTTT integrates Test-Time Training with fast weights into robot foundation models, compressing histories into weight space, to scale visuomotor context to 8K timesteps without increasing inference latency. The training recipe combines sequence action forcing with truncated backpropagation through time to make long-context training tractable.
• At this context length, RoboTTT unlocks new capabilities: one-shot in-context imitation from human video demonstrations, on-the-fly policy improvement, robustness to perturbations, and stronger multi-stage long-horizon performance, improving by 87% over a single-step context baseline on real-robot manipulation tasks.
• Scaling pretraining context length yields steady closed-loop performance gains for the first time: RoboTTT trained with 8K-timestep context outperforms the same model pretrained with 1K timesteps by 62%, fully completes a five-minute ten-stage assembly task that no baseline finishes, and establishes context length as a new scaling axis for robot foundation models.

Introduction

Most robot foundation models rely on single-step or short visuomotor histories, limiting their capacity for long-horizon tasks, in-context imitation from human videos, on-the-fly policy improvement, and robust closed-loop control. Scaling context length in existing Transformer-based policies causes inference latency to grow with history, while recurrent architectures often fall short in capturing complex long-range dependencies. To overcome these barriers, the authors introduce RoboTTT, a policy that integrates Test-Time Training using fast weights updated via gradient descent during both training and inference. This design compresses long observation-action streams into a compact recurrent state, scaling visuomotor context to 8,000 timesteps without increasing inference cost and unlocking new robot capabilities.

Method

The authors introduce RoboTTT, a framework that integrates Test-Time Training (TTT) layers into robot foundation models to enable learning over long-context trajectories. The architecture builds upon a Vision-Language Model (VLM) backbone and a Diffusion Transformer (DiT) action head.

Model Architecture RoboTTT incorporates TTT layers after the self- and cross-attention layers within the DiT action head. This design ensures that attention mechanisms process single-step information, while TTT layers handle information flow across the time dimension. The input to the DiT is a robot trajectory spanning $T$T timesteps, comprising learned register tokens $R_t$Rt​, vision-language tokens $\Phi_t$Φt​, encoded proprioception tokens $q_t$qt​, and noised action tokens $\tilde{A}_t$A~t​. To maintain computational efficiency, the VL tokens do not pass directly through the TTT layers; instead, a small number of register tokens carry this information across time.

To preserve the capabilities of the pretrained VLA model, the authors employ a learned tanh gating mechanism. For each DiT layer, a parameter $\boldsymbol{\alpha} \in \mathbb{R}^d$α∈Rd is initialized near zero, gating the TTT contribution as follows: $O = \tanh(\alpha) \odot O_{\text{TTT}} + O_{\text{attn}}$O=tanh(α)⊙OTTT​+Oattn​ where $O_{\text{TTT}}$OTTT​ is the TTT layer output and $O_{\text{attn}}$Oattn​ is the attention layer output. This allows the model to gradually adjust the influence of the TTT layers without overwhelming the pretrained computation.

As shown in the figure below, the training process involves running TTT over sequences to update fast weights, while inference starts from a learned initialization $W_0$W0​ and propagates fast weights forward based on observations.

Sequence Training and Optimization The training recipe focuses on learning both a suitable fast weight initialization $W_0$W0​ and its update dynamics. The TTT projection matrices and $W_0$W0​ are learned as part of the model parameters. Given a training sequence, the model runs TTT in an inner loop, computing the outer task loss at every timestep and optimizing the full model on the averaged loss. This meta-learning approach tailors fast weight updates specifically to robot trajectories.

The authors utilize a flow-matching objective for actions, where the DiT head learns to denoise action chunks. To stabilize training on long sequences, they introduce sequence action forcing, sampling the noise level $\tau_t$τt​ independently for each action chunk in the sequence. This prevents the entire sequence from becoming uniformly easy or hard to learn. The sequence loss is defined as: $\mathcal{L}_{\text{fm}}(\xi; W_0) = \frac{1}{T} \sum_{t=1}^T \mathbb{E}_{\tau_t, \epsilon} \left[ \| v_\theta(\Phi_t, A_t^{\tau_t}, q_t; W_{t-1}) - (A_t - \epsilon) \|^2 \right]$Lfm​(ξ;W0​)=T1​∑t=1T​Eτt​,ϵ​[∥vθ​(Φt​,Atτt​​,qt​;Wt−1​)−(At​−ϵ)∥2]

Training on long sequences with full backpropagation is memory-intensive. To address this, the authors adopt Truncated Backpropagation Through Time (TBPTT). The input sequence is divided into segments where gradients flow only within each segment. Crucially, fast weights are carried over across segment boundaries, allowing TTT to continue over the entire sequence while keeping GPU memory dependent on segment length rather than total sequence length.

As illustrated in the figure below, gradients are truncated at segment boundaries, but the fast weights carry over with detached gradients to maintain the temporal context.

Effective Learning from Context RoboTTT decouples fast weight updates from slow weight updates by masking the flow-matching loss on selected timesteps. These masked timesteps serve as pure context, updating fast weights without providing an imitation target. This mechanism enables learning from heterogeneous contexts, such as human video demonstrations or suboptimal robot rollouts.

A key application of this is DAgger Distillation. In a standard DAgger setting, a trajectory contains interleaved robot actions and human corrections. Standard fine-tuning discards suboptimal robot actions. RoboTTT, however, uses both in asymmetric roles. During sequence training, the fast weights are updated on the full interaction history, including suboptimal robot actions, but the flow-matching loss is masked to compute only on human corrections. This asymmetry distills the failure-to-correction mapping into the fast weights, teaching the policy to produce corrections in response to failures online.

The figure below visualizes this process, showing how robot actions serve as context for fast weight updates while human corrections provide the learning signal.

Experiment

The experiments evaluate RoboTTT on three dexterous, long-horizon bimanual assembly tasks against single-step, history-augmented, and recurrent baselines. RoboTTT consistently outperforms baselines by tracking task progress to avoid stage aliasing, recovering strategically from errors, and achieving fine-grained precision, while scaling pretraining context length yields steady gains through its gradient-based fast-weight updates. Long-context conditioning further unlocks one-shot imitation from human videos and robust within-episode recovery from perturbations, and a DAgger distillation approach lets the model improve by learning from its own suboptimal actions as context.

RoboTTT consistently achieves the most fully successful trials across all three assembly tasks. It is the only method to complete the demanding Gear Bot task, with 2 out of 10 trials fully successful, while no baseline records any full success. On the other tasks, RoboTTT leads by a wide margin, more than doubling the successes of the best baseline on Circuit and tripling them on Pup Go Car. RoboTTT is the only method with any fully successful trials on Gear Bot, succeeding in 2 of 10 attempts. RoboTTT achieves 13 fully successful trials on Circuit, compared to only 8 for the best baseline.

On the Circuit task with one-shot imitation from a human video of an unseen configuration, RoboTTT reaches a 65% task completion score and completes 6 of 10 trials fully, while the GDN baseline scores only 33% and never succeeds completely. The large gap shows RoboTTT generalizes far better to new assembly configurations when conditioned on a single demonstration. RoboTTT achieves more than double the completion score of GDN (65% vs 33%) and fully succeeds in 6 out of 10 trials, whereas GDN fails in every trial. The evaluation conditions on a single in-context human video of an unseen Circuit configuration, testing one-shot generalization to new component layouts and assembly orders. GDN's zero successful rollouts indicate that its recurrent state update, without test-time gradient descent, is insufficient for robust imitation in this setting.

RoboTTT shows the strongest robustness to external perturbations, successfully recovering in 15 of 20 roof and 18 of 20 tire trials. The single-step GR00T N1.7 baseline recovers in about half of trials, while appending history frames drastically reduces recoveries to just 3 roof and 5 tire successes. GDN's recurrent state improves over the history variant but remains behind RoboTTT on roof perturbations. RoboTTT recovers from roof perturbations more often than all baselines, and matches GDN on tire perturbations. Including past observations as context severely harms recovery, with success rates falling below 25% for both perturbation types.

RoboTTT is evaluated on three assembly tasks, a one-shot generalization benchmark from a human demonstration, and perturbation recovery tests. It is the only method to complete the challenging Gear Bot assembly and achieves substantially more fully successful trials than all baselines on Circuit and Pup Go Car. In the one-shot setting with an unseen Circuit configuration, RoboTTT reaches a 65% completion score and fully succeeds in most trials, while the best baseline fails completely. Under external perturbations, RoboTTT demonstrates the strongest recovery by successfully handling the majority of roof and tire displacements, whereas baselines, particularly those using past observations as context, frequently fail to recover.