One-sentence Summary

DeepSeek-AI introduces DeepSeek-V4.1-Flash, a $552\text{B-parameter}$552B-parameter multimodal Mixture-of-Experts model supporting one-million-token contexts, whose Causal Encoder-Decoder architecture activates $16\text{B}$16B parameters per token during decode but only $8\text{B}$8B during prefill, and which combines cross-layer KV cache reuse in Compressed Sparse Attention 2 with FP4 KV caching to cut the global HBM KV cache footprint to $890\text{ bytes}$890 bytes per token (about $\frac{1}{4}$41​ of DeepSeek-V4-Flash) and uses SWA Bounded Replay to reduce the persistent SSD/host KV footprint to roughly $\frac{1}{8}$81​, delivering substantially better performance for text-based and multimodal agentic workloads.

Key Contributions

• Introduces DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts model with 552B backbone parameters and one-million-token context support, whose Causal Encoder-Decoder (CED) architecture activates 8B parameters per token during prefill versus 16B during decode, improving cost efficiency for input-heavy agentic workloads.
• Combines cross-layer KV cache reuse in Compressed Sparse Attention 2 (CSA2) with FP4 KV caching to reduce the global KV cache footprint (always in HBM) to 890 bytes per token, about one quarter of DeepSeek-V4-Flash’s footprint, and introduces SWA Bounded Replay to cut the persistent KV cache footprint (on SSD or host memory) to roughly one eighth of the baseline.
• Pretrained on a 45T-token multimodal corpus and post-trained across diverse text and multimodal agentic tasks, DeepSeek-V4.1-Flash delivers substantially better overall performance than DeepSeek-V4-Flash and matches or exceeds larger open-source models such as GLM-5.3 and Kimi-K3 on key benchmarks despite a smaller parameter footprint.

Introduction

Long-horizon AI agents require processing extremely long contexts, which places heavy demands on the memory and storage systems that hold key-value (KV) caches. Prior sparse attention methods reduced compute costs, but persistent storage and data movement of these caches became the main bottleneck, limiting throughput and driving up deployment costs. The authors introduce DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts model designed to aggressively compress KV caches through joint optimization of architecture, cache precision, and deployment strategy. Using a Causal Encoder-Decoder design and a new Compressed Sparse Attention 2 mechanism, the model shares global KV caches across layers and uses FP4 precision to cut runtime cache storage to roughly one quarter of its predecessor, while a technique called SWA Bounded Replay reduces persistent disk storage to about one eighth. This enables the model to support million-token contexts with low inference latency and cost, achieving performance comparable to frontier models while being significantly more efficient to deploy.

Dataset

The authors describe a two-stage data strategy for pre-training and post-training, with each stage curated through its own dedicated pipeline.

Pre-training Data Construction

• Text Data Curation: The authors focus on the holistic interactions between diverse corpora to maximize information gain. They use a systematically constructed pipeline guided by a scaling ladder over model parameters and training data. To improve quality, they filter out model-generated content with limited information gain, such as outputs from less capable models and low-quality machine-translated text, treating these as implicit duplication. The corpus is expanded with recent open-source code from new repositories, commits, libraries, and frameworks to cover more programming languages and reflect current software engineering scenarios.
• Multimodal Data Curation: The multimodal dataset comprises three types: image-text pairs, interleaved image-text data, and domain-specific data. The authors prioritize cleaning raw web data over large-scale synthesis. They re-bootstrapped their crawler from Common Crawl to improve coverage of multimodal sources. For image-text pairs, they extract images with associated alt text, apply an image-text relevance threshold, and deduplicate based on image semantics. Interleaved data is built from webpages and PDFs through progressively expensive stages: first applying heuristic and statistical filtering, deduplication, and quality models; then assembling surviving documents into interleaved sequences with image-aware filtering and deduplication; finally using SmolVLM for strict quality scoring. Filtered-out documents are partially recycled into image-text pairs. Domain-specific datasets are added to improve fine-grained visual perception, OCR, and long-tail knowledge, along with image-code pairs and computer-use trajectories for agentic understanding.
• Data Integration and Deduplication: The final training corpus is the union of text-only and multimodal processed data. For overlapping samples, the authors replace text-only versions with multimodal counterparts and use the larger epoch count. After substitution, the corpus uses a 7:1 token ratio of text-only to multimodal data. Ultra-long documents are pre-split before mixing for uniform token distribution, and a best-fit packing algorithm achieves a padding rate of at most 10⁻⁴.

Post-training Data and Environment Construction

• The post-training pipeline focuses entirely on what the model is trained on rather than optimization algorithms. It synthesizes diverse, verifiable training tasks with reference solutions and reward signals, procedurally constructs interactive agent environments, and applies rigorous filtering, deduplication, and difficulty calibration.
• Task Formalization: Each task is a triplet (problem, environment, verification system), evaluated on difficulty and correctness. These signals are used to iteratively train the model to construct better tasks. RL task lifecycles are monitored, and trajectories from new RL runs provide fresh evidence for quality re-auditing.
• General Agent Environments: Built from interaction data voluntarily returned by internal employees and external partners using the latest model. The authors construct mocked tools that reproduce real-world interfaces, API schemas, and behavioral constraints. Negative feedback and failure cases are collected at scale and incorporated to generate single-turn and multi-turn agent environments that replay failures and enable targeted RL.
• Coding Agent Environments: Built from two sources: coding-agent sessions filtered for complex tasks or poor performance, deduplicated by trajectory, and public GitHub repositories meeting a star-count threshold. Environment construction involves multiple specialized agents: one determines buildability and verification feasibility, selects task starting points, and designs evaluation points; another sets up dependencies, test code, and task descriptions in an isolated container, removing solution leaks; multiple agents attempt the task, and a quality-inspection agent reviews the environment and trajectories for issues. A repair agent fixes errors and adjusts evaluation points before re-verification.
• Usage: Both general and coding agent environments feed into a unified training system, producing RL training data that is correct, discriminative, and controllable in length and difficulty, driving iterative model improvement under continuous quality monitoring.

Method

DeepSeek-V4.1-Flash is a multimodal mixture-of-experts (MoE) Transformer that processes images and text to generate text autoregressively. The language backbone consists of 40 causal Transformer layers, structured as a 20-layer causal encoder followed by a 20-layer decoder. The multimodal input pathway utilizes a custom vision encoder, DeepSeek-ViT, which processes images at varying resolutions using 2D-RoPE and a 3x3 pixel-unshuffle operation to reduce the visual token count before an MLP projector maps them to the language backbone hidden dimension. The model incorporates 552B backbone parameters and 196B Engram parameters, activating 8B parameters per token during prefill and 16B during decode.

To address the computational costs of long-context inference, the authors introduce the Causal Encoder-Decoder (CED) architecture and Compressed Sparse Attention 2 (CSA2). CED treats the bottom 20 layers as a causal encoder. For the decoder layers ($l > L/2$l>L/2), global key-value (KV) entries are projected directly from the hidden state of the final encoder layer ($H_{L/2}$HL/2​) using layer-dependent projection weights:

This design effectively halves the prefill computation. Sliding window attention (SWA) is maintained layer-wise across all layers to preserve local context.

CSA2 jointly exploits entry size, sequence dimension, and layer dimension to control KV cache storage and attention computation. Each CSA2 layer is statically assigned one of three operating modes: Full, Reindex, or Reuse.

In Full Mode, the layer computes its own main KV, indexer Q, and indexer K, running the indexer to produce fresh Top-K indices. Reindex Mode reuses the main KV and indexer K from the most recent Full Mode layer, allowing the sparse selection to change across layers while sharing the cache. Reuse Mode reuses both the main KV and the Top-K indices from a preceding layer, performing attention without computing new indexer queries or evaluating index scores. This cross-layer reuse significantly reduces cache storage and indexer computation.

To further reduce the computational bottleneck of indexing in extremely long contexts, the authors introduce the Hierarchical Sparse Indexer, which operates exclusively in the decoder.

The first layer assigned to Full Mode scores all causally visible main KV positions and performs blockwise candidate selection. It collects the positions covered by the selected blocks into a shared candidate pool. Subsequent layers in Reindex Mode score only the candidate positions within this pool rather than the full context, changing the per-query cost of deeper indexers from linear in context length to constant.

The authors implement several optimizations to enhance efficiency. Single-Pass mHC shifts the input-mixing coefficients by one block, allowing the residual update, input mixing, and coefficient prediction to be fused into a single kernel, halving the activation memory traffic. Engram adds sparsely accessed conditional memory to decouple memorization from computation, utilizing n-gram orders and multi-head hashing with FP8 precision. DSpark serves as a speculative decoding module, combining semi-autoregressive drafting with confidence-scheduled verification, and is trained in a dedicated stage after backbone pre-training. Additionally, the main KV cache is compressed to FP4 using quantization-aware training during post-training, nearly halving the storage footprint without requiring native matrix-multiplication support for the format.

The training process leverages head-wise Muon for Query and Key weights, providing different preconditioners for different attention heads to handle heterogeneity. To reduce optimizer-state memory footprint for the large Engram embedding tables, the authors employ a momentum-based update followed by Sinkhorn balancing, which equalizes the row-wise and column-wise RMS of the update matrix.

The post-training pipeline follows a standard paradigm of supervised fine-tuning followed by reinforcement learning (RL) and on-policy distillation. The authors focus on large-scale automated pipelines for data synthesis and environment construction. They formalize tasks as triplets of problem, environment, and verification system, and use difficulty and correctness as reward signals to iteratively train the model to construct better tasks for general and coding agents.

To improve model performance in complicated scenarios, the authors employ large-scale asynchronous RL in synthesized tasks. They decouple agent rollout execution into an agent sandbox and a worker container, managed by the DeepSeek Elastic Compute (DSec) platform, which scales to millions of concurrent sandbox instances using sharding and relaxed consistency scheduling.

Performance continues to improve as compute scales with cumulative RL steps, whether within a single scaffold, across variants of the same scaffold, or across heterogeneous scaffolds. To extend effective RL compute beyond a single run, the authors use model merging to reinitialize successive RL runs, combining improvements acquired along different optimization paths. Furthermore, a scalar effort level is introduced as an explicit conditioning signal during RL, allowing the model to dynamically adjust its reasoning strength and token budget at deployment time. The asynchronous post-training infrastructure maintains high rollout concurrency through sample-level dispatch and mitigates length bias and off-policy effects via per-dataset concurrency limits and loss masking for stale tokens.

Experiment

The evaluation spans pretraining, post-training, and infrastructure experiments. Pretraining benchmarks across world knowledge, reasoning, coding, long context, and multimodal tasks show that DeepSeek-V4.1-Flash, despite activating fewer parameters and using less KV cache than DeepSeek-V4-Pro, performs on par or better across most dimensions. Post-training evaluations focus on reasoning and agentic capabilities, with DeepSeek-V4.1-Flash significantly outperforming its predecessor on code, cyber security, general, and visual agent tasks, matching or exceeding top open-source and proprietary models. A reasoning effort control allows smooth accuracy-latency trade-offs, with most gains concentrated in lower budgets. Robustness tests across various agent scaffolds show performance transfers well, and preliminary multi-agent collaboration experiments demonstrate consistent advantages over single-agent baselines under time deadlines.

DeepSeek-V4.1-Flash-Base matches or exceeds the performance of the larger DeepSeek-V4-Pro-Base on most world knowledge benchmarks while activating far fewer parameters and using a smaller KV cache. The newer model also shows a notable efficiency advantage over the older Flash variant, despite both being MoE architectures. DeepSeek-V4.1-Flash-Base activates 8B/16B parameters versus 49B for DeepSeek-V4-Pro-Base, yet it achieves higher scores on MMLU-Pro. On MMLU-Pro, DeepSeek-V4.1-Flash-Base leads with 74.1, surpassing DeepSeek-V4-Pro-Base’s 73.5. DeepSeek-V4.1-Flash-Base ties or slightly lags on AGIEval and C-Eval, but the gaps are within the 0.3 threshold considered equal. DeepSeek-V4.1-Flash-Base outperforms DeepSeek-V4-Flash-Base on MMLU-Pro and MultiLoKo, indicating improved world knowledge despite similar or lower activation.

The table defines how public API reasoning-effort tiers map to scalar effort values, with the maximum tier set to 100, the high tier to 75, and the low tier to 50. This mapping provides a straightforward interface for users to control the degree of reasoning performed by the model. The underlying effort level is used as a conditioning signal during reinforcement learning, affecting both single-turn and multi-turn tasks. The maximum API tier corresponds to an effort value of 100, representing the highest level of reasoning thoroughness. The high and low tiers map to effort values of 75 and 50, respectively, offering a clear step-down in reasoning intensity. The effort level is explicitly fed into the model via a system prompt, allowing users to adjust cost-quality trade-offs.

DeepSeek-V4.1-Flash shows consistent gains over its predecessor and matches or surpasses top-tier open and closed models across reasoning and agentic benchmarks. Its most striking improvements appear in agentic tasks, where it overtakes leading proprietary systems on several metrics. In visual agent tasks, it leads open-source models but still trails the best closed-source alternatives. Codeforces rating improves from 3289 to 3471 over DeepSeek-V4-Flash but no model values are provided for comparison. Matches the best open-source score on MathArena Apex (65.6%) and edges past DeepSeek-V4-Pro (65.3%). Agentic gains are large: Deep-SWE v1.1 jumps from 54.4% to 74.2%, surpassing Opus-5 and GPT-5.6 Sol. Terminal-Bench 2.1 reach 90.6%, ahead of Opus-5 (89.1%) and GLM-5.3 (88.2%). On cyber-security tasks, it sets a new state of the art among open-source models. Visual agent performance beats the leading open-source Kimi-K3 but remains below top closed-source systems.

Across eight agent scaffolds, performance on DeepSWE v1.1 and Terminal-Bench v2.1 remains high, with the top results achieved by mini-SWE and DeepSeek Harness in Standard mode on DeepSWE and by DeepSeek Harness Minimal and mini-SWE on Terminal-Bench. The variation across scaffolds is moderate, indicating robust transfer of agentic capabilities across different harness designs. mini-SWE achieves the highest DeepSWE resolution rate at 74.2%, closely followed by DeepSeek Harness Minimal at 72.6%. On Terminal-Bench, DeepSeek Harness Minimal and mini-SWE tie for the best Pass@1 at around 90%, while Claude Code trails at 88.0%. Performance differences across scaffolds are within roughly 9 points on DeepSWE and 7 points on Terminal-Bench, suggesting limited sensitivity to scaffold choice.

DeepSeek-V4.1-Flash-Base delivers comparable or superior world knowledge performance to the larger DeepSeek-V4-Pro-Base while using far fewer activated parameters and a smaller KV cache, with notable gains on MMLU-Pro and near-parity elsewhere. The full Flash model shows strong improvements over its predecessor, especially in agentic tasks, where it surpasses leading proprietary systems on software engineering and terminal benchmarks, though it still trails closed models in visual agent tasks. Across multiple agent scaffolds, performance remains consistently high with moderate variation, indicating robust generalization of agentic capabilities.