One-sentence Summary

The MemTensor Team introduces MemT-1.0, a memory-native foundation model that integrates Gated DeltaNet-2 with stable token-level memory updates and Mixture-of-Storage scaling and is trained on 34B tokens; the released MemT-1.0-Nano and MemT-1.0-Flash outperform parametric memory baselines on average, remain competitive with retrieval-augmented methods, achieve approximately $1.86\times$1.86× end-to-end inference speedup and $66\times$66× reduction in persistent storage relative to full context KV cache at a history length of 128K tokens, and improve task success on agentic tasks by internalizing retrieved procedural experience.

Key Contributions

• Introduces MemT-1.0, a memory-native foundation model architecture that integrates persistent memory into model computation via Gated DeltaNet-2 with a stable token-level memory update mechanism and a Mixture-of-Storage mechanism for scaling native memory capacity.
• Develops a deliberate backbone preparation and a two-stage training pipeline of continued pre-training and post-training on 34B tokens, releasing the MemT-1.0-Nano and MemT-1.0-Flash model family.
• Reports that MemT-1.0 models outperform parametric memory baselines on average and remain competitive with retrieval-augmented methods; MemT-1.0-Nano achieves about 1.86× end-to-end inference speedup and 66× storage reduction at 128K tokens, and improves SWE-bench Verified and Multilingual success by 7.6 and 8.3 percentage points over the backbone without experience.

Introduction

Foundation models are increasingly internalizing capabilities such as reasoning and multimodality, and memory is a natural next target because it allows models to accumulate experience without carrying the original context. Previous memory-native work, including Metis, showed that memory can become a native model capability, but its token selection approach left stability and capacity scaling as open challenges. The authors introduce MemT-1.0, which retains dynamic local memory and trainable hyper memory but replaces token selection with stable token-level updates based on Gated DeltaNet-2 and adds Mixture-of-Storage to scale native memory capacity. They train Nano and Flash model variants through a two-stage pipeline with 34B tokens, evaluate them on 11 memory benchmarks, and demonstrate improved historical information use while largely preserving general capabilities.

Dataset

Dataset description

• Backbone preparation mixture. The authors post-train DeepSeek-V4-Flash with data from three tracks: general capabilities, agentic coding, and agentic working. Data construction follows three steps: capability profiling, attribution by intervention, and capability-conditioned synthesis. Reliably solved tasks supply replay data. Occasionally solved tasks supply the backbone’s own attempts that pass programmatic verifiers. Failures caused by missing conventions or procedures are learned from guided sampling or style-matched demonstrations from stronger models. Failures that persist under intervention are excluded, and knowledge gaps are left mainly to retrieval or native memory. The final mixture contains 126.5K trajectories, 2.4B tokens total, and 1.1B loss-bearing tokens. By count, it is 56.4% general, 18.0% agentic coding, and 25.3% agentic working; agentic trajectories provide about 90% of loss-bearing tokens.

• MemT-1.0 training corpus. The authors construct a 34B-token corpus for training MemT-1.0. It includes 20B tokens of continued pre-training data from large-scale general-purpose language modeling corpora, emphasizing long-form and sequential text. The remaining 14B tokens are post-training data in three categories: memory-native language modeling data, native memory storage and utilization data, and memory task data. Storage and utilization tasks include summarization, continuation, and compression. Memory task data strengthens selective retention and long-range dependency resolution. Part of the corpus is based on anonymized internal MemOS interaction data. All post-training categories use a unified context-query-answer format: context is first stored through native memory, then the model receives the query and generates the target answer conditioned on the memory state.

• Experience-guided agentic coding data. The authors build an experience bank from 41,057 successful coding trajectories released by Scale-SWE. They use trajectory annotation, procedural policy induction, policy clustering, experience generation, and quality auditing. After filtering, the bank contains 1,040 reusable coding experiences. Each experience records triggering conditions, recommended execution steps, applicability constraints, and verification procedures. For training samples, they generate 18,820 original coding trajectories with Qwen3.6-35B-A3B using the AweAgent harness in the Scale-SWE environment. They identify breakpoints in unsuccessful trajectories, inject retrieved experience subsets, and evaluate 199,500 breakpoint-experience variants. They retain 45,481 successful continuations, and after further quality filtering obtain 42,403 training samples. Samples are partitioned by task to prevent leakage between training and validation. Each sample is represented as the original trajectory prefix up to the breakpoint, the injected experience subset, and the successful continuation.

• How the data is used. The backbone preparation mixture is used to post-train DeepSeek-V4-Flash before native memory is attached; the backbone is then frozen. The 34B-token corpus supports MemT-1.0 training. The experience-conditioned coding samples are used for application-specific post-training of MemT-1.0-Nano: the experience is written into native memory, the original prefix is replayed with memory reading disabled, and after the breakpoint the model predicts the successful continuation. Experience text and the trajectory prefix are excluded from the loss. The final setup uses knowledge distillation with full-suffix supervision.

• Processing notes. No explicit cropping strategy is described. The main formatting choices are the unified context-query-answer structure for MemT-1.0 post-training and the prefix/experience/continuation structure for experience-conditioned agentic training.

Method

MemT-1.0 is formulated under a multi-step scenario with discrete steps. At each step, the autoregressive generation process is conditioned on the input and previously generated tokens, while the stateful model parameters are updated to incorporate new information. The authors build MemT-1.0 upon general causal language models, augmenting selected standard Transformer blocks with native memory blocks to maintain memory from previous information. As shown in the figure below:

The architecture integrates a Local Memory Block and a Hyper Memory Block. The local memory block stores online information as a dynamic memory state, initialized to zero at the start of new trajectories. The hyper memory block contains trainable parameters, including memory projections, gating projections such as erase, write, and forget gates, and memory output gating. These parameters are optimized during training and remain fixed during online interaction.

The native memory operates through two parallel procedures: storage and utilization. The memory storage procedure updates the dynamic memory states using a stable token-level update mechanism based on GDN-2. It computes memory keys and values, along with erase, write, and forget gates, to perform token-level updates that prevent divergence. The memory utilization procedure retrieves information from the accumulated memory states. It calculates memory queries and reads out information via matrix readout, which is then gated by an output gate using a SoftCapSiLU activation before being concatenated and added to the original attention output.

To jointly scale the online memory capacity and trainable memory parameters, the authors introduce Mixture-of-Storage (MoS). As shown in the figure below:

MoS expands each native memory block into multiple parallel memory experts. Unlike conventional sparse Mixture-of-Experts, MoS adopts dense routing over all memory experts. It introduces two dense routers for memory utilization and storage, which share a trainable expert address matrix. The utilization router computes routing probabilities to weight the memory attention outputs from each expert, while the storage router modulates the erase gate, write gate, and log-decay for each expert's state update.

The training of MemT-1.0 involves backbone preparation, data construction, and a multi-stage training pipeline. The authors first post-train the DeepSeek-V4-Flash backbone to improve reliability on tasks it solves occasionally, using a three-step pipeline of capability profiling, attribution by intervention, and capability-conditioned synthesis. As shown in the figure below:

This backbone preparation significantly improves performance across multiple benchmarks before native memory is attached. The prepared backbone is then frozen to serve as the host model and the full-context distillation teacher.

For data construction, the authors compile a large-scale training corpus of 34B tokens, comprising continued pre-training data and post-training data. The post-training data is organized into memory-native language modeling data, data for native memory storage and utilization, and memory task data, all formulated in a unified context-query-answer format.

The core training objective aligns the model capability with the backbone given full context through an off-policy distillation framework. The frozen backbone serves as the teacher, providing next-token distributions over the full historical context. The MemT-1.0 student receives the current input and the incoming local memory state representing the preceding history. The training loss combines a forward KL divergence from a truncated teacher distribution and a next-token cross-entropy loss.

The training proceeds in two stages. First, Continued Pre-Training establishes basic language modeling capabilities with native memory. The input sequence is divided into consecutive segments, and the student performs causal language modeling within the current segment while relying on native memory to access information from preceding segments, using cross-entropy loss. Second, Post-Training adapts the model to native memory-based response generation. Historical context is processed through the native storage procedure to produce memory states, and the model is trained using KL-based distillation to approximate the full-context teacher outputs, focusing optimization on the response tokens.

To enable experience-guided agentic execution, the authors develop a framework where MemT-1.0 stores and reuses procedural experience during long-horizon task execution. As shown in the figure below:

The framework retrieves relevant experiences from an experience bank and stores them either in the agent textual context or in MemT-1.0-Nano native memory. In the native memory setting, the experience text is processed through a forward pass with memory writing enabled, updating the memory state without modifying the original task context. Subsequent agent actions then access the stored experience through memory attention, allowing the experience to influence action generation without explicitly appearing in the message history.

Experiment

The experiments examine MemT-1.0 through scaling, benchmark, efficiency, storage, ablation, and agentic application perspectives. Scaling studies on MemT-1.0-Nano show that continued memory-related post-training improves memory capability at fixed capacity, while adding memory experts provides further gains, especially later in training. Benchmark evaluations across 11 memory tasks and 6 general capability benchmarks indicate that MemT models lead parametric memory baselines but still trail strong textual memory systems, while general capabilities remain broadly preserved under noise. Ablations attribute substantial gains to token-level updates and post-training, and agentic coding plus long-history efficiency results show that native memory can internalize procedural experience, improve task consistency, and keep storage constant with better long-history inference efficiency.

Windowed execution of one MoS memory block substantially lowers peak memory relative to whole-pack execution at a 32K-token pack length. A one-step window achieves the largest memory reduction but adds a modest time overhead, while a two-step window retains most of the memory savings with a smaller time increase. Switching from whole-pack to windowed execution reduces peak memory by roughly 80-87% for the tested window sizes. A 2048-token one-step window gives the greatest memory reduction but incurs a larger relative time increase than a 4096-token two-step window. The two-step window still reduces peak memory substantially while keeping the added forward-plus-backward time under 2%.

MemT-1.0-Flash and MemT-1.0-Nano are the top two parametric memory methods in the evaluation, with a modest aggregate gap driven mainly by memory operations. Flash is stronger on memory operations, personalized understanding, conversational recall, and implicit learning, while Nano is stronger on PerMemSafe and Memora. Textual memory systems that receive explicit historical evidence at query time still achieve the highest performance, especially on LongMemEval-S and BEAM-1M. MemT-1.0-Flash and MemT-1.0-Nano rank first and second among parametric memory methods, ahead of the strongest parametric baseline by clear margins. The two MemT variants show complementary strengths: Flash leads on memory operations, personalized understanding, conversational recall, and implicit learning, while Nano leads on PerMemSafe and Memora. Textual memory systems remain stronger overall, particularly on LongMemEval-S and BEAM-1M, reflecting the benefit of explicit historical evidence at query time.

Under the noise setting, MemT-1.0-Flash matches its DeepSeek-V4-Flash backbone on average, with a notable gain on BigCodeBench-Hard and modest gains on IFEval and LongBench v2, while showing small declines on MMLU-Pro, GPQA Diamond, and LiveCodeBench v6. MemT-1.0-Nano shows a slight average decrease around one point, with most benchmarks staying close to its backbone and gains on IFEval and BigCodeBench-Hard. The pattern of improvement on one coding benchmark and decline on another indicates that noise effects vary within the same capability domain. MemT-1.0-Flash preserves average general capability relative to its DeepSeek-V4-Flash backbone under the noise setting. The largest MemT-1.0-Flash improvement occurs on BigCodeBench-Hard, while MMLU-Pro shows the largest decline. MemT-1.0-Nano shows only a small average decrease, with gains on IFEval and BigCodeBench-Hard and its largest decline on LiveCodeBench v6. Effects are not uniform within coding benchmarks, as BigCodeBench-Hard improves while LiveCodeBench v6 declines.

Ablations show that token-level updates are important for MemT-1.0-Nano, as switching to turn-level updates causes the largest average drop among architecture variants. Replacing the stable update with GDN or GDN-2 only slightly reduces overall performance, suggesting the stable mechanism gives a modest edge. Training ablations indicate post-training is the main driver of memory-specific improvements, while continued pre-training adds an initialization benefit. The full model achieves the highest overall average among the listed architecture variants, with GDN-based replacements close behind. Token-level updating is a key factor; the turn-level variant shows the largest regression in average score. Replacing the stable update with GDN or GDN-2 reduces the overall average only modestly relative to the full model. Post-training improves memory-specific performance by roughly ten points over the continued pre-training checkpoint, while continued pre-training also supports the final pipeline.

Native memory consistently improves average coding success and repeated-run reliability compared with the no-experience baseline across SWE-bench Verified and Multilingual. Its performance is comparable to textual experience, with small advantages in some settings. Gains are stronger for pass^3 than for pass@3, indicating more consistent solving across repeated runs. Native memory improves average success and pass^3 over the no-experience baseline on both SWE-bench Verified and Multilingual. In the Verified Before Execution setting, native memory reaches a 69.33% average success rate, slightly above textual experience and the no-experience baseline. Repeated-run reliability improves substantially: Verified pass^3 rises from 54.00% to 61.60%, while Multilingual pass^3 rises from 45.33% to 53.67%. Pass@3 remains relatively flat or slightly lower, suggesting the benefit is consistency rather than increasing the chance of solving a task at least once. In the Harness-Triggered setting, native memory achieves the highest pass^3 and average success among the listed verified-benchmark methods.

These experiments evaluate memory efficiency, parametric memory quality, noise robustness, ablation choices, and repeated software-engineering runs. Windowed execution of the MoS memory block greatly reduces peak memory compared with whole-pack execution, and a two-step window offers a favorable tradeoff between memory savings and time overhead. MemT-1.0-Flash and MemT-1.0-Nano lead parametric memory methods with complementary strengths, though textual memory systems remain strongest, especially when historical evidence is available. Under noise, MemT variants generally preserve backbone capability; ablations show token-level updates and post-training are key contributors, and native memory improves coding success and repeated-run consistency over a no-experience baseline.