One-sentence Summary

DeepSeek-AI introduces DeepSeek-V4-Pro ($1.6\text{T}$1.6T parameters, $49\text{B}$49B activated) and DeepSeek-V4-Flash ($284\text{B}$284B parameters, $13\text{B}$13B activated), Mixture-of-Experts models featuring a hybrid attention architecture with Compressed Sparse Attention and Heavily Compressed Attention, manifold-constrained hyper-connections, and the Muon optimizer, which support million-token contexts while requiring only $27\%$27% of the single-token inference FLOPs and $10\%$10% of the KV cache of DeepSeek-V3.2, thereby redefining the state-of-the-art for open models.

Key Contributions

• DeepSeek-V4 series introduces a hybrid attention architecture combining Compressed Sparse Attention and Heavily Compressed Attention, reducing single-token inference FLOPs to 27% and KV cache to 10% of DeepSeek-V3.2 at one-million-token contexts.
• DeepSeek-V4-Pro-Max redefines the state-of-the-art for open models by substantially outperforming prior open-source models on knowledge benchmarks and achieving reasoning performance close to frontier proprietary models.
• The architecture incorporates Manifold-Constrained Hyper-Connections to enhance residual connections and the Muon optimizer for faster convergence, with both models pre-trained on more than 32T tokens.

Introduction

The growing adoption of test-time scaling in large language models has driven strong performance improvements, but the quadratic complexity of standard attention creates a severe bottleneck for the ultra-long contexts needed in agentic workflows, cross-document analysis, and other long-horizon tasks. Prior architectures, including earlier DeepSeek versions, struggled to maintain efficiency as context lengths stretched toward millions of tokens, limiting both practical deployment and further scaling gains.

The authors introduce the DeepSeek-V4 series, a preview of two models (DeepSeek-V4-Pro with 49B activated parameters and DeepSeek-V4-Flash with 13B activated) designed to break this efficiency barrier. Their core contribution is a hybrid attention mechanism that combines Compressed Sparse Attention and Heavily Compressed Attention, which aggressively compresses KV caches along the sequence dimension while preserving modeling fidelity. Together with Manifold-Constrained Hyper-Connections, the Muon optimizer, and extensive infrastructure optimizations (fused MoE kernels, FP4 quantization-aware training, heterogeneous KV cache storage), the architecture natively supports one-million-token contexts while using as little as 10 percent of the inference FLOPs and 7 percent of the KV cache size of its predecessor.

Dataset

The authors build the DeepSeek-V4 pre-training corpus on top of the data used for DeepSeek-V3, with a focus on higher diversity, better quality, and longer effective contexts. The final corpus exceeds 32 trillion tokens and spans multiple high-quality categories.

Key details of the dataset construction:

• Web data: Additional filtering removes batched auto-generated and templated content to reduce the risk of model collapse.
• Math and programming: These remain core components. Coding capabilities are further strengthened by introducing agentic data during the mid-training phase.
• Multilingual data: A larger multilingual corpus is built to improve long-tail knowledge coverage across different cultures.
• Long documents: Special emphasis is placed on scientific papers, technical reports, and other materials with unique academic value.
• Tokenization: The tokenizer inherits the 128K vocabulary size from DeepSeek-V3, with a few special tokens added for context construction. The token-splitting and Fill-in-Middle (FIM) strategies are also retained.
• Packing and masking: Documents from different sources are packed into appropriate sequences to minimize sample truncation. Unlike DeepSeek-V3, the pre-training uses sample-level attention masking.

The preprocessed data is used directly for model pre-training, with the mixture of sources and the packing strategy designed to support longer context learning and stable training across domains.

Method

The DeepSeek-V4 series retains the foundational Transformer architecture and Multi-Token Prediction (MTP) modules established in previous iterations while introducing significant architectural upgrades to enhance long-context efficiency and training stability. The overall framework integrates a hybrid attention mechanism, strengthened residual connections, and optimized Mixture-of-Experts (MoE) components.

The authors leverage the DeepSeekMoE paradigm for the Feed-Forward Networks (FFNs), utilizing fine-grained routed experts alongside shared experts. A key modification involves changing the activation function for computing affinity scores from Sigmoid to $\text{Sqrt}(\text{Softplus}(\cdot))$Sqrt(Softplus(⋅)). To ensure load balancing without auxiliary loss, an auxiliary-loss-free strategy is employed, augmented by a sequence-wise balance loss. Furthermore, the initial Transformer blocks utilize Hash routing, where target experts are determined by a predefined hash function based on the input token ID.

To strengthen the conventional residual connections between adjacent Transformer blocks, the model incorporates Manifold-Constrained Hyper-Connections (mHC). Standard Hyper-Connections (HC) expand the residual stream width by a factor of $n_{\text{hc}}$nhc​, decoupling residual width from the actual hidden size. The update rule is formulated as:

where $\mathcal{F}_l$Fl​ denotes the layer function. The core innovation of mHC is constraining the residual mapping matrix $B_l$Bl​ to the manifold of doubly stochastic matrices (the Birkhoff polytope) $\mathcal{M}$M. This constraint ensures that the spectral norm $\|B_l\|_2$∥Bl​∥2​ is bounded by 1, making the residual transformation non-expansive and enhancing numerical stability during forward and backward passes. The parameters are dynamically generated and projected onto the manifold using the Sinkhorn-Knopp algorithm.

To address the computational bottleneck of attention in ultra-long contexts, the authors design a hybrid attention architecture combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA).

CSA compresses the Key-Value (KV) cache and applies sparse attention. It first compresses every $m$m tokens into a single entry using learnable compression weights and positional biases. A "Lightning Indexer" then selects the top-$k$k compressed KV entries for core attention. The core attention operates in a Multi-Query Attention (MQA) manner with shared keys and values, followed by a grouped output projection to manage computational costs.

HCA employs a more aggressive compression strategy, consolidating the KV cache of every $m'$m′ tokens (where $m' \gg m$m′≫m) into a single entry without overlapping compression. Unlike CSA, HCA does not employ sparse selection; instead, it performs dense attention on the heavily compressed entries. Both mechanisms include a supplementary sliding window attention branch to preserve causality and model local dependencies effectively.

The efficiency of this hybrid approach is substantial. By utilizing mixed precision storage (BF16 for RoPE dimensions and FP8 for others) and FP4 precision for the lightning indexer, the KV cache size is drastically reduced.

To support the massive scale of the MoE layers, the authors propose a fine-grained Expert Parallelism (EP) scheme. In standard EP, communication latency often bottlenecks performance. The authors observe that in DeepSeek-V4, computation time exceeds communication time within a single MoE layer. They fuse communication and computation into a single pipelined kernel. Experts are split and scheduled into "waves." As soon as communication for a wave completes, computation begins immediately, allowing token transfer for the next wave and result sending for the current wave to proceed concurrently.

For optimization, the model employs the Muon optimizer for the majority of modules, utilizing hybrid Newton-Schulz iterations for orthogonalization. This involves a two-stage process: rapid convergence followed by precise stabilization of singular values.

In the post-training phase, the authors replace the mixed Reinforcement Learning stage with On-Policy Distillation (OPD). This method merges capabilities from multiple domain-specific teacher models into a unified student model. The objective function minimizes the KL divergence between the student and teacher distributions:

To ensure stable gradient estimation, the framework adopts full-vocabulary logit distillation rather than token-level estimates, caching teacher hidden states to reconstruct logits on the fly and managing memory through efficient teacher scheduling.

Experiment

The evaluation setup spans pre-training and post-training stages, assessing base models on world knowledge, language understanding, coding, math, and long-context benchmarks, while post-training evaluations cover knowledge, reasoning, agentic tasks, and real-world use cases including Chinese writing, search, and professional productivity. DeepSeek-V4-Flash-Base outperforms the larger DeepSeek-V3.2-Base across most benchmarks, and DeepSeek-V4-Pro-Base achieves near-universal dominance, establishing new performance highs across all categories. Post-training, DeepSeek-V4-Pro-Max sets a new state-of-the-art among open-source models on knowledge and reasoning tasks, matches leading closed models on several metrics, and demonstrates strong agentic and long-context capabilities, though it still trails proprietary models on certain knowledge benchmarks and complex instruction-following scenarios. Real-world task evaluations confirm DeepSeek-V4-Pro's advantages in Chinese writing, search-augmented QA, white-collar professional tasks, and code agent scenarios, with human evaluations and internal metrics highlighting practical gains over prior models and competitive standing against frontier systems.

DeepSeek-V4-Flash-Base surpasses the larger DeepSeek-V3.2-Base on most benchmarks despite using substantially fewer activated and total parameters, especially in world knowledge and long-context tasks. DeepSeek-V4-Pro-Base achieves a further decisive capability leap, establishing near-universal dominance over both predecessors with dramatic gains in knowledge-intensive evaluations and broad improvements across reasoning, coding, and long-context understanding. DeepSeek-V4-Flash-Base outperforms DeepSeek-V3.2-Base across a wide array of benchmarks while using only 13B activated parameters compared to 37B. DeepSeek-V4-Flash-Base shows particularly strong advantages in world knowledge tasks like AGIEval and MMLU-Redux. DeepSeek-V4-Pro-Base achieves the highest scores on nearly all evaluations, with especially large gains on knowledge-intensive benchmarks such as MMLU-Pro. On MMLU-Pro, DeepSeek-V4-Pro-Base reaches 73.5, substantially exceeding both DeepSeek-V3.2-Base and DeepSeek-V4-Flash-Base. DeepSeek-V4-Pro-Base delivers comprehensive improvements over its predecessors across knowledge, reasoning, coding, and long-context categories.

DeepSeek-V4 models offer three reasoning modes that trade off speed against depth by applying distinct length penalties and context windows during RL training. The Non-think mode provides fast, intuitive responses for routine tasks, while Think High and Think Max progressively allocate more reasoning tokens for complex problem-solving and exploring capability boundaries. Human evaluations show the Think Max variant excels at task completion and content quality by anticipating user intent and producing coherent long-form output, though it can overlook specific formatting constraints. Three reasoning modes (Non-think, Think High, Think Max) are produced by varying length penalties and context windows during RL training, resulting in different output token lengths. Non-think mode targets fast, intuitive responses for routine daily tasks and low-risk decisions, while Think High applies conscious logical analysis for complex problem-solving. Think Max pushes reasoning to its fullest extent using a special system prompt and is designed to explore the boundaries of model reasoning capability. Human evaluations found the Think Max variant outperforms a competing model on Chinese white-collar tasks, with primary strengths in task completion and content quality. The model proactively provides supplementary insights and self-verification steps, and delivers in-depth coherent narratives rather than overly simplistic bullet points. Occasional weaknesses include overlooking specific formatting constraints, struggling to condense long texts into succinct summaries, and room for improvement in visual presentation design.

The 'Think Max' mode injects a system-prompt instruction that demands exhaustive reasoning, requiring the model to decompose problems completely, stress-test logic against edge cases, and document every intermediate step and rejected alternative. This instruction is used during specialist training with distinct length penalties and context windows to shape high-effort reasoning behavior. Human evaluations show that models trained with such reasoning effort excel at task completion and content quality by providing supplementary insights and self-verification, though they may occasionally over-think or overlook specific formatting constraints. The injected instruction explicitly forbids shortcuts and mandates thorough problem decomposition to resolve root causes. Reasoning must be stress-tested against all potential paths, edge cases, and adversarial scenarios. The model is required to document every intermediate step, considered alternative, and rejected hypothesis in its deliberation. Human evaluations indicate this exhaustive reasoning mode leads to strengths in anticipating implicit user intents and providing self-verification steps. A noted trade-off of extensive reasoning is occasional over-thinking and reduced proficiency in condensing long inputs into succinct summaries.

A set of special tokens enables structured auxiliary tasks such as web search decisions, title generation, query formulation, authority classification, domain identification, and URL fetching within the assistant pipeline. These tokens define clear formatting boundaries that separate user prompts from assistant actions, supporting controlled multi-step workflows. A dedicated token determines whether a user prompt needs web search or can be answered directly, separating the prompt from the assistant decision with a thinking marker. Title generation occurs after the first assistant response, using a special token placed after the end-of-sentence marker to produce a concise conversation title. Search query formulation is handled by a distinct token that follows the user prompt, enabling explicit query generation. Separate tokens classify the user prompt's demand for source authoritativeness and identify its domain, each placed immediately after the user prompt. A token structure extracts URLs from the user prompt and determines whether each URL should be fetched and read, using an extracted URL marker followed by a read decision token.

DeepSeek-V4-Pro-Max establishes a new state-of-the-art among open-source models on knowledge and reasoning benchmarks. It opens a wide lead over prior open models on factual accuracy and achieves top scores on code reasoning, even surpassing closed models. However, it still trails Gemini-3.1-Pro on several knowledge tasks. On SimpleQA-Verified, DeepSeek-V4-Pro-Max outperforms all open-source baselines by roughly 20 percentage points, though it remains behind Gemini-3.1-Pro. On LiveCodeBench, DeepSeek-V4-Pro-Max attains the highest score, surpassing both open and closed models including Gemini-3.1-Pro.

The evaluation compares base model variants and reasoning modes, showing that DeepSeek-V4-Flash-Base outperforms the larger DeepSeek-V3.2-Base on most benchmarks despite using far fewer activated parameters, while DeepSeek-V4-Pro-Base achieves a decisive leap with dramatic gains in knowledge-intensive evaluations and broad improvements across reasoning, coding, and long-context tasks. Three reasoning modes, Non-think, Think High, and Think Max, are produced by varying length penalties and context windows during RL training, with Think Max using an exhaustive reasoning instruction that mandates complete problem decomposition and stress-testing, leading to superior task completion and content quality at the cost of occasional over-thinking. A set of special tokens structures auxiliary tasks like web search decisions, query formulation, and authority classification to support controlled multi-step workflows. In final benchmarks, DeepSeek-V4-Pro-Max establishes a new state-of-the-art among open-source models on knowledge and reasoning, opening a wide lead on factual accuracy and achieving top code reasoning scores, though it still trails Gemini-3.1-Pro on several knowledge tasks.