One-sentence Summary

Researchers from MemTensor (Shanghai) Technology Co., Ltd., Renmin University of China, and National University of Singapore et al. propose Metis, the first memory foundation model, which endows foundation models with a native persistent memory state and autonomous memory procedures, enabling gradient-free online memory updates via a forward pass with frozen inference weights, and demonstrating native memory capabilities.

Key Contributions

• A formal definition of native memory is introduced, consisting of a persistent and evolving memory state within the backbone and native memory procedures that autonomously store and utilize information through model computation.
• Metis is proposed as the first prototype, equipping a foundation model with a native memory state via local and hyper memory blocks, enabling gradient-free online memory updates with a single forward pass while learned weights remain frozen at inference.
• Large-scale memory-specific training data and a mid-training framework with memory reconstruction, memory operation, and regularization objectives are constructed to acquire native memory procedures, and extensive experiments demonstrate Metis’s native memory capabilities with detailed behavioral analysis.

Introduction

As foundation models power increasingly capable AI agents, memory is essential for retaining past information across interactions. Prior memory approaches typically rely on external modules, such as retrieval-augmented generation, which are decoupled from the backbone model, difficult to optimize end-to-end, and add inference latency. The authors introduce memory foundation models that internalize memory as a native mechanism, converting external memory into dynamic internal states and computations. They present Metis, a prototype that uses novel memory blocks and a dedicated mid-training framework to learn autonomous memory operations, removing the need for separate memory engineering.

Dataset

The authors build a mid-training dataset for memory procedures by synthesizing examples from 27 public benchmarks. The dataset comprises two complementary subsets, each with distinct composition and processing.

Dataset Composition and Sources

• Primary data: 357,137 samples, ~406 million tokens, drawn from 27 benchmarks across fiction, science, news, and logical reasoning domains.
• Auxiliary data: 609,443 samples, constructed from the same source facts plus normal dialogue pools.
• Both subsets rely on a common synthesis pipeline that extracts structured seeds from the benchmarks and rewrites them into interaction sequences.

Primary Data Subset (Core Memory Operations)

• Covers four memory operations: remember, forget, update, and reflect.
• Each sample is a temporally ordered sequence of turns, with state-consistent answers that depend on earlier memory states.
• Varies along two extra dimensions:
  • Salience of instruction: explicit memory commands vs. implicit narrative descriptions.
  • Noise level: clean sequences vs. sequences with distractor turns inserted between the reference and the query.
• Filtering: A language model verifier discards samples that fail consistency checks (answer must reflect the intended memory state), orthogonality checks (distractors must not leak the core fact), and shortcut checks (query must not be answerable without the reference). Semantic diversity is also monitored to avoid template collapse.
• Token distribution is dominated by distractor-heavy remember samples, enforcing long-range memory.

Auxiliary Data Subset (Robustness and Generalization)

• Four subtypes, each targeting specific failure modes:
  • Multi-entity binding (76,153 samples): stores two confusable facts and queries both, training the model to bind values correctly.
  • Selective forgetting (76,153 samples): revokes one fact while keeping another, preventing collateral forgetting.
  • Post-memory dialogue (357,137 samples): appends an ordinary conversation turn after a memory query, so the model does not leak stored values.
  • Memory-irrelevant dialogue (100,000 samples): asks a question that does not need memory even when a memory state exists, teaching the model when to ignore memory.
• Construction:
  • Paired-fact synthesis: for each source fact, a language model generates a confusable counterpart using transformations (change subject/relation, imitate value format, or stay semantically adjacent). A verifier discards contradictions, restatements, or dependencies.
  • Normal dialogue pool: assembled from everyday assistant dialogues, open-domain conversations, and entity-related dialogues that are answerable without memory, filtered for memory cues, real-time facts, and unsafe content.
  • Subtype formulation: the above ingredients are composed into the four interaction patterns.

How the Paper Uses the Data

• The primary data provides explicit supervision for training native memory procedures (remember, forget, update, reflect) through optimization.
• The auxiliary data is used to improve generalization in complex scenarios: multi-fact reasoning, mixed dialogues, and pollution resistance.
• No explicit training split or mixture ratio is reported; both subsets are assumed to be combined during mid-training.
• The synthesis pipeline itself is the main processing: seed extraction from source benchmarks, static rewriting into explicit/implicit styles with distractor insertion, and automatic quality verification.

Method

The authors propose Metis, a prototype architecture for memory foundation models that internalizes memory into the backbone computation. Unlike external memory systems that rely on explicit storage and retrieval procedures outside the model, Metis maintains a native memory state and executes memory procedures autonomously alongside the forward computation.

As shown in the figure below, the Metis architecture integrates Metis blocks into a standard causal language model. Each Metis block consists of two core components: a local memory block and a hyper memory block. The local memory block is responsible for maintaining the dense representation of prior information. It defines a dense memory network $M^{(l)} \in \mathbb{R}^{d_k \times d_v}$M(l)∈Rdk​×dv​ and a query-key normalization vector $S^{(l)} \in \mathbb{R}^{d_k}$S(l)∈Rdk​ at the $l$l-th layer. These dynamic parameters are updated across different interaction steps and are initialized to zero. The hyper memory block constructs the parametric function spaces for native memory procedures. It contains static parameters optimized during mid-training, including a learnable importance vector $\tilde{w}_{agg}^{(l)}$w~agg(l)​ for adaptive aggregation, and projection matrices $\tilde{W}_K^{(l)}$W~K(l)​, $\tilde{W}_V^{(l)}$W~V(l)​, and $\tilde{W}_Q^{(l)}$W~Q(l)​ for mapping hidden states into memory keys, values, and queries.

The native memory procedure in Metis is divided into storage and utilization phases, both tightly coupled with the forward computation. In the native memory storage procedure, the hyper memory block updates the local memory block based on intermediate activations. The model first pre-normalizes the hidden states and scores each token using the importance vector to obtain an importance distribution. A subset of positions is selected based on a threshold, and the corresponding hidden states are gathered. These selected states are then projected into memory keys $\tilde{K}_t^{(l)}$K~t(l)​ and values $\tilde{V}_t^{(l)}$V~t(l)​. The dense memory network and the normalization vector are updated using a discounted accumulation strategy, effectively compressing historical information into a fixed-size matrix.

In the native memory utilization procedure, the local memory block incorporates the current memory state into the forward computation. The authors define a memory attention mechanism that computes the interaction between the memory query states and the stored memory network, normalized by the query-key normalization vector. This memory attention is then integrated into the main attention branch. A balancing parameter controls the fusion of the original causal self-attention and the memory attention, allowing the model to leverage stored information from previous steps without the need for explicit context prefilling.

To empower Metis with these native memory capabilities, the authors design a multi-objective optimization strategy during mid-training. The training objectives include memory reconstruction, memory operation, and regularization. The memory reconstruction objective drives the model to store and recover reference passages with minimal loss, establishing a foundation for lossless storage. The memory operation objective teaches the model to perform semantic operations such as remembering, forgetting, updating, and reflecting based on input instructions, utilizing both explicit commands and implicit narratives. Finally, the regularization objective mitigates interference and memory pollution by training on complex scenarios, such as multi-entity binding and selective forgetting, ensuring that the model binds values to correct keys and avoids leaking stored information into unrelated responses. The training process employs a task-weighted sampler with linear annealing to gradually shift the focus from storage-oriented data to more challenging long-range and regularization tasks.

Experiment

Metis is evaluated on memory operation and memory-based question-answering tasks, comparing a native memory state (no context replay) against full-context, partial-context, and parametric memory baselines. The experiments demonstrate that Metis can store and retrieve information effectively without replaying the original evidence, achieving the best average performance on both memory operations and QA, and generalizing to out-of-distribution benchmarks. Ablation studies reveal that adaptive aggregation, auxiliary training data, and separate memory queries are critical for robust memory formation, while capacity studies show that long sequences and many update steps gradually degrade retention. The approach largely preserves the backbone’s general capabilities when the memory is empty, but stored irrelevant information can interfere with later tasks, and low-rank decomposition of memory states indicates that most useful information concentrates in a low-dimensional subspace.

The primary data covers four memory operations—remember, forget, update, and reflect—each using a distinct multi-turn skeleton where the final query answer is consistent with earlier memory steps. Samples are synthesized from 27 public benchmarks spanning diverse domains, providing explicit supervision for learning native memory procedures through optimization. Each memory operation follows a fixed interaction pattern: remember states a fact then queries it; update modifies a fact before querying; forget revokes a fact before querying; reflect introduces multiple single-hop facts and queries their multi-hop composition. The source benchmarks include fiction, science, news, and logical reasoning datasets, which supply verified facts and reduce hallucination while enriching context and traceability.

The primary dataset contains 357k samples across remember, forget, update, and reflect operations, with a total of 406.1 million tokens. The remember operation dominates the token budget, accounting for 89% of all tokens, while having fewer samples than forget, indicating much longer interaction sequences. Explicit instructions are the most frequent style overall, but the implicit and distract styles are well represented, especially in the remember and reflect operations. The remember operation consumes 89% of total tokens, despite contributing only 16% of all samples. Distract-style samples, where memory instructions are embedded in irrelevant conversation, constitute half of the entire dataset. Forget and update operations are heavily skewed toward explicit memory commands, whereas remember and reflect are balanced between explicit and implicit styles.

The auxiliary data is organized into four subtypes that address interference and memory pollution by composing facts and dialogues into complex interaction patterns. Multi-entity binding and selective forgetting train the model to handle multiple confusable facts without cross-talk, while post-memory and memory-irrelevant dialogues prevent stored values from leaking into unrelated conversation turns. Multi-entity binding stores two similar facts then queries both, forcing the model to bind each value to its corresponding fact. Selective forgetting revokes one fact while keeping another, teaching the model to forget selectively without collateral loss.

The auxiliary dataset contains 609,443 quality-verified samples distributed across four subtypes that target interference and memory pollution. Memory pollution scenarios dominate the collection, with post-memory dialogue alone accounting for more than half of all samples. Multi-fact scenarios are equally split between multi-entity binding and selective forgetting but together represent only about a quarter of the total. Post-memory dialogue is the largest subtype, contributing over half of the auxiliary samples. Memory pollution scenarios outnumber multi-fact scenarios by roughly three to one.

Full-context Qwen3.5 models achieve strong overall scores on memory operation tasks, but removing context causes a sharp performance decline. Partial context preserves substantially more information on the Metis test set than on MemOps (Gold), where incomplete histories fail to reliably support operations. Under no-context conditions, Metis attains the best average results on both benchmarks, recovering part of the lost performance. Full-context Qwen3.5 models average above 84 on MemOps (Gold) and above 73 on the Metis test set, while partial context causes averages to drop to 22–30 and 60–65, respectively. On the Metis test set, partial context retains remember and reflect scores above 60, but on MemOps (Gold) all operation scores fall below 40, showing that incomplete histories are especially unreliable for that benchmark. Metis achieves the best average performance under no-context setting, outperforming other methods, and increasing model size to 27B brings clear gains in remembering, updating, reflection, and overall scores. Forgetting consistently scores lower than other operations on MemOps (Gold) across all context settings, and it remains the most difficult operation even for Metis-27B.

The evaluation setup involves multi-turn memory operations (remember, forget, update, reflect) trained on synthetic data from diverse benchmarks, with auxiliary data that addresses interference and memory pollution. Experiments validate that full conversation context is critical for reliable memory performance, as partial context causes a sharp decline, particularly on the MemOps (Gold) test set. The Metis model recovers the most performance under no-context conditions, while forgetting consistently proves to be the most difficult operation across all settings.