One-sentence Summary

Researchers from Nanjing University of Science and Technology, Zhejiang University, and National University of Singapore introduce LaMem-VLA, a latent-memory-native framework that reconstructs historical experience into short-term and long-term latent memory tokens and weaves them into multimodal reasoning via coordinated curator, seeker, condenser, and weaver components, thereby enabling memory to directly guide action generation in long-horizon robotic manipulation tasks, as demonstrated on SimplerEnv and LIBERO.

Key Contributions

• LaMem-VLA introduces a latent-memory-native framework that reconstructs historical experience into latent memory tokens and interweaves them directly with VLA reasoning through four coordinated components: a curator, seeker, condenser, and weaver.
• The framework maintains complementary short-term visual and long-term semantic memory vaults; retrieved evidence is distilled into compact latent memory tokens and inserted into the continuous embedding sequence alongside the current observation and instruction, allowing past context to shape action generation in the same latent space.
• Experiments on SimplerEnv and LIBERO demonstrate that LaMem-VLA outperforms existing approaches, validating context-native latent memory for long-horizon, memory-dependent manipulation tasks.

Introduction

Vision-language-action (VLA) models map visual observations and language instructions to robot actions, but they typically assume a Markovian setting: actions are predicted from the current observation without temporal context. This creates a short-horizon bias that leaves them struggling with long-horizon manipulation tasks, where awareness of task progress and past state changes is essential. Prior methods try to incorporate history by extending the input window, compressing past observations, or retrieving tokens from an external memory bank, but they either incur growing computational costs, lose fine-grained detail, or keep memory as an auxiliary policy-side signal that never enters the VLA’s internal reasoning process. The authors address these limitations with LaMem-VLA, a native latent memory framework that organizes robotic history into complementary short-term visual and long-term semantic memory vaults. It retrieves task-relevant evidence, condenses it into compact latent tokens compatible with the model’s embedding space, and weaves those tokens directly into the action-generation sequence, making memory an integral part of the VLA’s perception, instruction understanding, and action reasoning.

Method

The authors formulate the robotic manipulation task in Vision-Language-Action (VLA) models as a language-conditioned Markovian decision-making problem. At each timestep $t$t, the VLA policy $\Pi_\theta$Πθ​ takes a natural language instruction $I$I and the current visual observation $o_t$ot​ as input to predict a chunk of future actions:

$\boldsymbol{a}_{t:t+H-1} = \boldsymbol{\Pi}_\theta(\boldsymbol{o}_t, \boldsymbol{I})$at:t+H−1​=Πθ​(ot​,I)

where $H$H denotes the action horizon, and each action $a_t \in \mathbb{R}^7$at​∈R7 is a 7-DoF end-effector control vector comprising 3-DoF relative translation, 3-DoF relative rotation, and a 1-DoF gripper state.

While effective for short-horizon behaviors, this Markovian paradigm can induce a temporal short-horizon bias in long-horizon manipulation tasks. To address this, the authors introduce LaMem-VLA, an end-to-end native latent memory framework that directly weaves dual-scale historical experience into VLA reasoning to refine action generation.

As shown in the figure below:

LaMem-VLA closes the loop between latent memory reconstruction and action reasoning through four coordinated modules. At each timestep $t$t, the vision-language backbone embeds the current visual observation $o_t$ot​ and instruction $I$I into visual tokens $X_t$Xt​ and instruction tokens $I$I. Learnable action queries are appended to the token sequence to obtain manipulation-relevant latent action representations.

The latent memory curator dynamically establishes and updates two complementary memory vaults. The short-term memory vault $\mathcal{M}^{short}$Mshort stores visual tokens that preserve recent perceptual evidence. Each short-term memory unit is represented as a key-value pair $m_s^i = (k_s, v_s)$msi​=(ks​,vs​), where the key $k_s$ks​ provides a concise retrieval summary and the value $v_s$vs​ stores the latent content. Specifically, a learnable SE-bottleneck compression module $\mathcal{C}_s$Cs​ distills the current visual tokens $X_t$Xt​ into compact short-term memory tokens:

$\boldsymbol{v}_s = \mathcal{C}_s(\boldsymbol{X}_t) \in \mathbb{R}^{N_s \times C}, \quad \boldsymbol{k}_s = \text{MeanPool}(\boldsymbol{v}_s) \in \mathbb{R}^C$vs​=Cs​(Xt​)∈RNs​×C,ks​=MeanPool(vs​)∈RC

The long-term memory vault $\mathcal{M}^{long}$Mlong stores action hidden states that track task progress and action continuity across longer horizons. When the number of stored units exceeds the vault capacity, the curator applies a compression strategy by computing the cosine similarity between temporally adjacent keys, selecting the most redundant pair, and consolidating them by averaging both key and value tokens.

The latent memory seeker retrieves evidence from these vaults according to the current multimodal cognition context. Given the visual and instruction tokens, the VLA backbone produces a context-aware query $Q_t^{con}$Qtcon​. The seeker appends learnable query slots $Q^{init} \in \mathbb{R}^{K_q \times C}$Qinit∈RKq​×C and updates them using a lightweight transformer-based query builder $\mathcal{B}$B:

$\boldsymbol{Q}_t = \mathcal{B}([\boldsymbol{Q}_t^{\text{con}}; \boldsymbol{Q}^{\text{init}}])[-K_q:] \in \mathbb{R}^{K_q \times C}$Qt​=B([Qtcon​;Qinit])[−Kq​:]∈RKq​×C

The mean-pooled query $q_t = \text{MeanPool}(Q_t)$qt​=MeanPool(Qt​) serves as the global retrieval vector. The seeker uses $q_t$qt​ to retrieve the Top-$K$K context-relevant units from both the short-term and long-term vaults via cosine similarity, yielding retrieved sets $Z^{short}$Zshort and $Z^{long}$Zlong.

The latent memory condenser then reconstructs this retrieved evidence into fixed-length latent memory tokens. It introduces learnable short-term visual memory slots $T_s \in \mathbb{R}^{L_s \times C}$Ts​∈RLs​×C and long-term semantic memory slots $T_l \in \mathbb{R}^{L_l \times C}$Tl​∈RLl​×C, updating them with lightweight memory formers $\mathcal{F}_v$Fv​ and $\mathcal{F}_c$Fc​ conditioned on the context query tokens and retrieved evidence:

$\boldsymbol{M}^{\text{short}} = \mathcal{F}_v([\boldsymbol{Q}_t; \boldsymbol{Z}^{\text{short}}; \boldsymbol{T}_s])[-L_s:], \quad \boldsymbol{M}^{\text{long}} = \mathcal{F}_c([\boldsymbol{Q}_t; \boldsymbol{Z}^{\text{long}}; \boldsymbol{T}_l])[-L_l:]$Mshort=Fv​([Qt​;Zshort;Ts​])[−Ls​:],Mlong=Fc​([Qt​;Zlong;Tl​])[−Ll​:]

This produces query-conditioned latent memory tokens in the same embedding space used by VLA reasoning.

Finally, the latent memory weaver injects these synthesized memory tokens into the VLA reasoning sequence before action queries are resolved. The memory-augmented input sequence $S_t$St​ is constructed as:

$\boldsymbol{S}_t = [\boldsymbol{M}^{\text{short}} + \mathbf{1}_{L_s}\boldsymbol{b}_s^\top; \boldsymbol{M}^{\text{long}} + \mathbf{1}_{L_l}\boldsymbol{b}_l^\top; \boldsymbol{X}_t; \boldsymbol{I}; \boldsymbol{Q}^{\text{action}}]$St​=[Mshort+1Ls​​bs⊤​;Mlong+1Ll​​bl⊤​;Xt​;I;Qaction]

where $b_s$bs​ and $b_l$bl​ are learnable source embeddings. Because these memory tokens are part of the model input sequence, they participate in self-attention with the current observation, language instruction, and action queries, forming memory-grounded action tokens $Z^{action}$Zaction.

These memory-grounded action tokens are fed into a diffusion-based action expert to generate a continuous action chunk. The diffusion expert $\epsilon_\theta$ϵθ​ is trained with a mean squared error loss to predict the injected noise under the action and memory conditions:

$\mathcal{L}_{\text{action}} = \mathbb{E}_{n, \epsilon} \left[ \left\| \epsilon - \epsilon_\theta(\boldsymbol{a}_{t:t+H-1}^n, n, \boldsymbol{Z}^{\text{action}}) \right\|_2^2 \right]$Laction​=En,ϵ​[​ϵ−ϵθ​(at:t+H−1n​,n,Zaction)​22​]

During inference, DDIM sampling iteratively denoises the action chunk to produce history-aware continuous 7-DoF control actions.

Experiment

LaMem-VLA is evaluated on real-to-sim manipulation (SimplerEnv-Bridge) and diverse simulated task suites (LIBERO) to assess how dual-scale latent memory improves vision-language-action policies. The experiments demonstrate that injecting compressed short-term visual tokens and long-term semantic tokens directly into the reasoning sequence provides effective historical context, resolving temporally ambiguous states and supporting reliable action prediction across both short-horizon and long-horizon settings. Ablations confirm the complementary roles of the two memory streams and show that latent-native integration outperforms external memory conditioning, while moderate retrieval budgets and token counts balance performance and efficiency.

LaMem-VLA achieves the highest average success rate of 73.9% on the SimplerEnv-Bridge benchmark, outperforming all compared methods. The model surpasses the next best baseline by 16.6 percentage points and maintains robust performance across tasks, including those where prior models failed. This improvement is attributed to integrating compressed latent memory tokens into the action prediction sequence. LaMem-VLA's average success rate is 16.6 points higher than CogACT and also exceeds the recent state-of-the-art VLA π0, demonstrating the benefit of latent memory tokens for historical context. On the difficult Stack Cube task, LaMem-VLA reaches 41.7% success, whereas several earlier methods (RT-1-X, OpenVLA) scored 0%, and it attains near-perfect 95.8% on Put Eggplant in Basket.

LaMem-VLA achieves the highest success rates on all five LIBERO benchmark suites, reaching an overall average of 97.6%. It outperforms memory-augmented and standard vision-language-action baselines without requiring the additional proprioceptive and wrist-camera inputs that starred methods use. The largest gains appear on long-horizon tasks, where it surpasses the previous best memory-augmented model by over a point on both Long-10 and Long-90. LaMem-VLA reaches a 97.6% average success rate across all five suites, exceeding the strongest reported memory-augmented method by 1.1 points and a strong baseline like CogACT by 4.4 points. On the long-horizon suites, LaMem-VLA attains 95.8% on Long-10 and 97.0% on Long-90, improving over MemoryVLA by 2.4 and 1.4 points respectively, while not using extra proprioceptive or wrist-camera inputs.

Ablating the dual-scale latent memory shows that the full model with both short-term and long-term memory achieves the highest success rates on both benchmarks. Removing both memory streams causes the largest performance drop, while keeping only one stream yields intermediate results, confirming that the two memory vaults contribute complementary information. The full LaMem-VLA model reaches 73.9% on SimplerEnv and 97.0% on LIBERO-90, outperforming every ablated variant. Removing both short-term and long-term memory leads to the steepest decline, falling to 57.3% on SimplerEnv and 92.1% on LIBERO-90. Retaining only short-term memory (65.6%, 95.4%) or only long-term memory (64.6%, 94.8%) gives similar moderate performance, showing that neither vault alone is sufficient.

An ablation on memory integration shows that incorporating compressed latent memory tokens directly into the VLA input sequence yields the highest success rates, with the full LaMem-VLA model reaching 73.9% on SimplerEnv and 97.0% on LIBERO-90. Removing memory entirely drops performance sharply, while policy-side conditioning or raw retrieval conditioning provide intermediate gains but fall short of latent-native integration. The memory-free baseline achieves only 57.3% on SimplerEnv and 92.1% on LIBERO-90, the lowest across all conditions. Using memory as an external policy-side condition improves performance to 71.9% on SimplerEnv and 94.8% on LIBERO-90. Directly conditioning on raw retrieved evidence yields 69.8% on SimplerEnv and 95.1% on LIBERO-90, indicating that uncompressed tokens introduce redundancy. LaMem-VLA prepends compressed memory tokens to the VLA input, achieving the best results (73.9% on SimplerEnv and 97.0% on LIBERO-90).

The number of retrieved memory units K strongly affects manipulation performance, with success rates rising as K grows from 2 to 8 and then dipping at K=12. The optimal setting at K=8 provides sufficient historical evidence without overloading the memory condenser, achieving peak success on both benchmarks. Increasing K from 2 to 4 yields clear gains on both SimplerEnv (66.7% to 70.8%) and LIBERO-90 (94.4% to 95.9%). Performance peaks at K=8 (73.9% on SimplerEnv, 97.0% on LIBERO-90) and declines at K=12, showing a moderate retrieval budget works best.

LaMem-VLA integrates compressed latent memory tokens directly into its action prediction sequence, substantially improving visual-language-action manipulation performance. It sets new state-of-the-art success rates on SimplerEnv-Bridge and across all LIBERO suites, particularly on long-horizon tasks, without relying on extra proprioceptive or wrist-camera inputs. Ablation studies confirm that combining short-term and long-term latent memories is crucial, that conditioning on these compressed tokens within the input sequence works best, and that a retrieval budget of 8 memory units yields the strongest results.