One-sentence Summary

Kimi Team introduces and open-sources Kimi K3, a $2.8\,\text{T}$2.8T Mixture-of-Experts model with $104\,\text{B}$104B activated parameters, native vision, and a $1\,\text{M}$1M-token context window, which leverages Kimi Delta Attention, Attention Residuals, and Stable LatentMoE for a $\sim 2.5\times$∼2.5× scaling efficiency gain over K2, and achieves frontier-level performance via reinforcement learning across general, agentic, and coding domains, though it trails Claude Fable 5 and GPT-5.6 Sol.

Key Contributions

• Kimi K3 is a 2.8T-parameter Mixture-of-Experts model with native vision and a 1M-token context window, incorporating Kimi Delta Attention, Attention Residuals, and Stable LatentMoE (896 experts, 16 active per token) to deliver an approximately 2.5× improvement in overall scaling efficiency over Kimi K2.
• The model undergoes reinforcement learning across long-horizon coding, agentic, and reasoning tasks at multiple effort levels, and domain- and effort-specialized policies are consolidated into a unified model via multi-teacher on-policy distillation, enabling compositional generalization and robust long-horizon execution.
• The full model weights are released openly, and the paper details infrastructure co-designs such as fused KDA kernels and balanced expert-parallel training (MoonEP) for 2.8T-parameter pre-training, along with a co-located RL system featuring resumable sandboxes for million-token agentic trajectories.

Introduction

The authors leverage the dual scaling axes of large language models: the traditional axis of training larger models on more data, and the emerging axis of scaling test-time computation through reasoning and agentic behaviors. Recent open-source models have advanced quickly on test-time scaling but have remained stalled at roughly 1-trillion-parameter scales, creating a risk that further progress will converge while the gap to proprietary frontier systems widens. To address this, the authors introduce Kimi K3, a 2.8-trillion-parameter Mixture-of-Experts model with native multimodal vision, 104 billion activated parameters, and a 1-million-token context window. Architectural innovations (Kimi Delta Attention, Attention Residuals, and Stable LatentMoE) together with refined data and recipes deliver a 2.5× improvement in scaling efficiency over its predecessor, and post-training via reinforcement learning across coding, agentic, reasoning, knowledge, and vision domains consolidates multi-effort test-time scaling into a single unified model. The result is an open frontier model that pushes both scaling axes simultaneously into the megaparameter and megacontext regime.

Dataset

The authors assemble a multi-source dataset for both pre-training and post-training, with extensive filtering, rephrasing, and synthetic augmentation to cover text, vision, long-context, and specialized task domains.

Pre-training data

• Text corpus: Four domains — Web Text, Code, Mathematics, and Knowledge. Each is cleaned with rule‑based heuristics, classifier quality scoring, and deduplication. Domain‑specific sampling rates are set via ablation studies on smaller models. Knowledge and mathematics data are further rephrased: diverse style/perspective prompts, chunk‑wise autoregressive generation, and fidelity verification against original documents.
• Vision corpus: Builds on the taxonomy from prior Kimi models, combining open‑source collections with in‑house pipelines for filtering, synthesis, and dedup. Covers captioned images, interleaved image‑text documents, OCR, perception, video, and visual coding. Coordinate supervision is given in both absolute and normalized formats for resolution‑robust localization. Additionally, a large‑scale programmatic multimodal set couples code snippets with rendered visuals — SVG, 3D assets, webpages, game screens, and CAD schematics.

Long‑context data

• Natural long documents and videos: Filtered through a dedicated pipeline — exact and fuzzy deduplication, perceptual hashing over video frames, classifier‑based quality filtering, and structural validation. Because truly long, coherent content is rare, these are upsampled to avoid being overwhelmed by short sequences.
• Synthetic long‑context sequences: Multimodal documents and sub‑tasks are carefully permuted and concatenated so that solving an embedded task requires attending to information scattered across the entire 1M‑token context. This forces the attention mechanism to learn genuine long‑range patterns.

Post‑training RL task datasets

• Knowledge‑graph‑guided tasks: An agentic, self‑evolving hierarchical knowledge graph is built through web‑scale exploration. Agents recursively expand coarse seed concepts, deduplicating against existing nodes. Sampled nodes at various granularities — combined with ancestor context — generate web queries; retrieved real‑world materials are then assembled by a synthesis agent into diverse training tasks.
• Kernel optimization tasks: A large suite ranging from single‑operator kernels to fused mega‑kernels, sourced from high‑quality GitHub repositories (e.g., Flash Linear Attention). Covers CUDA, Triton, CuTe DSL, Gluon, ThunderKittens, TileLang, and multiple GPU architectures/numerical formats (BF16, FP8, FP4). Rewards are based on correctness and performance, with a hacking‑detection system to penalize reward‑gaming strategies.
• Personal assistant tasks: Realistic mock implementations of apps (Gmail, Notion, Slack, Canvas) enable reproducible, long‑horizon workflows. Scenarios inspired by HR, legal, and finance involve persistent environments with dozens of interdependent events over simulated days, requiring thousands of tool calls. Initial workspaces are built by agents that search the web for reference materials.
• Web development tasks: Expert‑curated tasks range from one‑line scene descriptions to full‑stack applications. Span websites, interactive games, 3D/WebGL, data visualisation, SVGs, and more. Run in containerized sandboxes; rewards combine deterministic functional checks (build success, structural/pixel similarity) and model‑based judging of source code and output artifacts.

How the data is used

• The text and vision corpora are mixed for pre‑training, with sampling proportions optimized on small models.
• Long‑context data is introduced during the cooldown phase through a progressive curriculum (context window grows from 256K to 1M tokens), with upsampling and synthetic sequences ensuring the model learns long‑range dependencies.
• Post‑training task sets drive reinforcement learning; each provides automated or LLM‑based reward signals to train agentic and specialised capabilities.

Method

The Kimi K3 architecture is designed to scale information flow along three complementary dimensions: sequence length, network depth, and model width. Along the sequence dimension, Hybrid Attention combines three Kimi Delta Attention (KDA) layers with one Gated Multi-head Latent Attention (MLA) layer in each block. Along the depth dimension, Attention Residuals (AttnRes) enable each module to selectively retrieve representations from the embedding and preceding blocks. Along the width dimension, each attention layer is followed by a Stable LatentMoE layer that performs sparse channel mixing. For native vision, MoonViT-V2 encodes images and videos, mapping visual features into the shared embedding space. Together with Per-Head Muon, these components provide a unified architecture for scaling information flow across tokens, layers, and channels.

Kimi K3 uses a layerwise hybrid of linear and global attention. Each block contains 3 KDA layers followed by 1 Gated MLA layer, giving a 3:1 mixing ratio. An additional Gated MLA layer is placed at the end of the backbone. KDA extends the delta-rule recurrence with a channel-wise forget gate. For a sequence of hidden states $x_t$xt​, the recurrent state $S_t$St​ is updated as:

$$S_t = (I - \beta_t k_t k_t^\top) \text{Diag}(\alpha_t) S_{t-1} + \beta_t k_t v_t^\top, \quad \tilde{o}_t = S_t^\top q_t$$St​=(I−βt​kt​kt⊤​)Diag(αt​)St−1​+βt​kt​vt⊤​,o~t​=St⊤​qt​

where $\alpha_t$αt​ is the channel-wise one-step retention factor and $\beta_t$βt​ controls the delta-rule write strength. To address numerical overflow in the chunkwise parallel form, Kimi K3 changes the mapping from decay logits $z_t^h$zth​ to the per-step log-decay $g_t^h$gth​. Instead of an unbounded negative-Softplus mapping, Kimi K3 uses a scaled sigmoid to bound the log-decay from below:

$$g_t^h = g_{\min} \text{Sigmoid}(e^{A_h} z_t^h) \in (g_{\min}, 0)^{d_k}, \quad \alpha_t^h = \exp(g_t^h) \in (e^{g_{\min}}, 1)^{d_k}$$gth​=gmin​Sigmoid(eAh​zth​)∈(gmin​,0)dk​,αth​=exp(gth​)∈(egmin​,1)dk​

where $g_{\min} = -5$gmin​=−5. This finite range allows both diagonal and off-diagonal tiles to use dense Tensor Core matrix multiplications, eliminating the position-pair diagonal path.

Finally, KDA applies an input-dependent full-rank output gate:

$$y_t = W_o [\text{Sigmoid}(W_g x_t) \odot \text{RMSNorm}(\tilde{o}_t)]$$yt​=Wo​[Sigmoid(Wg​xt​)⊙RMSNorm(o~t​)]

Gated MLA compresses the key-value representation into a low-dimensional latent vector $c_t = W_c x_t$ct​=Wc​xt​. Unlike previous versions, Kimi K3 applies No Position Encoding (NoPE) to all MLA layers, relying on KDA for position-sensitive mixing. It also augments MLA with an input-dependent, channel-wise full-rank output gate:

$$y_t = W_o [\text{Sigmoid}(W_g x_t) \odot \tilde{o}_t]$$yt​=Wo​[Sigmoid(Wg​xt​)⊙o~t​]

Standard residual connections compress all prior information into a single state over depth. Attention Residuals (AttnRes) apply attention methodology to depth: each layer selectively retrieves representations from all preceding layers. For each layer $l$l, a layer-specific learnable pseudo-query $q_l = w_l$ql​=wl​ is defined, with keys and values derived from the token embedding and preceding layer outputs. The attention weights follow a softmax kernel with RMSNorm to prevent large-magnitude outputs from dominating. To reduce memory overhead, the layers are partitioned into blocks. Within a block, layer outputs are reduced to a single representation by summation. Across blocks, full attention is applied over the block-level representations.

LatentMoE separates the full model width from the routed-expert width, allowing Kimi K3 to scale channel mixing to 896 routed experts with 16 active experts per token. The shared experts process the input directly, while the routed path projects it to a compact latent space, dispatches it to selected experts, and maps the aggregate back:

$$u = \sum_{i \in \mathcal{T}_k(x)} p_i E_i^{\text{routed}}(W^\downarrow x), \quad y = \sum_{j=1}^{N_s} E_j^{\text{shared}}(x) + W^\uparrow \text{RMSNorm}(u)$$u=i∈Tk​(x)∑​pi​Eirouted​(W↓x),y=j=1∑Ns​​Ejshared​(x)+W↑RMSNorm(u)

To suppress activation explosion in the routed branch, Kimi K3 inserts RMSNorm between expert aggregation and the up-projection. It also proposes Sigmoid Tanh Unit GLU (SiTU-GLU), which applies a smooth cap to the linear factor of the Swish gate and independently to the up branch:

$$\text{SiTU-GLU}(x) = [\beta_1 \tanh(W_g x / \beta_1) \odot \text{Sigmoid}(W_g x)] \odot [\beta_2 \tanh(W_u x / \beta_2)]$$SiTU-GLU(x)=[β1​tanh(Wg​x/β1​)⊙Sigmoid(Wg​x)]⊙[β2​tanh(Wu​x/β2​)]

For load balancing, Kimi K3 adopts Quantile Balancing (QB), which sets each expert bias from the router-score quantile that matches its target load. The update rule is:

$$\hat{b}_j^{(t+1)} \leftarrow -\text{quantile}_{1-k/n}(s_{:,j} - \alpha^{(t)}), \quad b^{(t+1)} \leftarrow \hat{b}^{(t+1)} - \text{mean}(\hat{b}^{(t+1)}) \mathbf{1}$$b^j(t+1)​←−quantile1−k/n​(s:,j​−α(t)),b(t+1)←b^(t+1)−mean(b^(t+1))1

Kimi K3 is natively multimodal. The vision encoder, MoonViT-V2, is trained entirely from scratch with next-token prediction, rather than being initialized from a contrastively pre-trained model. This approach yields more stable optimization with lower gradient norms.

MoonViT-V2 is a 27-layer vision transformer that adopts RMSNorm and removes bias terms. Visual inputs are encoded and mapped by a lightweight MLP projector into the LLM. Attention is factorized into intra-frame spatial and inter-frame temporal passes, and a pixel-shuffle operation reduces the number of visual tokens.

Kimi K3 adopts Muon as the optimizer for matrix parameters. For attention projections, it uses a per-head variant that partitions momentum matrices along the head dimension and orthogonalizes each head's block separately, equalizing update scale across heads. The model is pre-trained on a curated corpus spanning text and vision domains. Scaling law studies show that the architectural improvements deliver an approximately 2.5x gain in overall scaling efficiency over Kimi K2.

The training recipe adopts a native multimodal strategy where language and vision are jointly optimized from the start. The model uses a cosine learning rate schedule with a 1% linear warmup and weight decay of 0.1. Pre-training begins with an 8k token context length, later extended to 64k tokens.

Experiment

The evaluation setup spans scaling-law re-tuning of architectural and training improvements, agentic reinforcement learning on verifiable multi-step tasks, and comprehensive public, in-house, cybersecurity, and third-party benchmarks. The scaling study confirms that architecture, data, and training changes yield a 2.5× efficiency gain and that cosine decay consistently outperforms WSD under fair hyperparameter optimization. On public benchmarks, Kimi K3 rivals the strongest proprietary models in coding and agentic domains while offering superior cost efficiency, and internal evaluations highlight strong orchestration and deep research capabilities alongside a tangible but incomplete exploit-development ability. Case studies further demonstrate the model’s capacity to autonomously perform complex real-world work such as GPU kernel optimization, compiler authoring, chip design, and multi-day knowledge tasks.

SiTU-GLU modifies the gate and up branches of GLU-like activations by applying tanh-based saturation, yielding a bounded scalar response. It closely tracks SwiGLU near the origin while capping the output at the product of its two saturation parameters for large positive inputs, unlike SwiGLU which remains unbounded. SiTU-GLU stays within the bound |f(x)| ≤ β₁β₂ = 100 for large x, whereas SwiGLU grows without limit. The inset shows SiTU-GLU and SwiGLU nearly coincide near the origin, preserving small-input behavior.

Compared to Kimi K2, Kimi K3 scales up depth by 52% and total parameters by 167%, while activated parameters grow by 220%. The mixture-of-experts layers add a latent dimension and increase the per-expert hidden dimension by 50%, leading to a reported 2.5× improvement in overall scaling efficiency. Layers increased from 61 to 93 (↑52%), total parameters from 1.04T to 2.78T (↑167%), and activated parameters from 32.6B to 104.2B (↑220%). A latent MoE dimension of 3584 (0.5× hidden size) was introduced, and the per-expert hidden dimension grew from 2048 to 3072 (↑50%). These architectural changes collectively yielded an approximately 2.5× scaling-efficiency gain over Kimi K2, as measured on held-out OOD validation data.

Kimi K3 closely trails the strongest proprietary models, Claude Fable 5 and GPT-5.6 Sol, while consistently outperforming Claude Opus 4.8, GPT-5.5, and GLM-5.2 across benchmarks. It excels in coding and agentic tasks, achieving top scores on several benchmarks, but lags behind in research-level reasoning and on Elo-rated knowledge-work suites. Kimi K3 achieves the best scores on ProgramBench and SWE-Marathon (7 points ahead of Claude Fable 5), nearly matches GPT-5.6 Sol on Terminal-Bench 2.1, and ranks second on FrontierSWE, behind only Claude Fable 5. In agentic suites, it sets state-of-the-art on BrowseComp, DeepSearchQA, and MCPMark-Verified, and is within 0.2 points of the leader on CorpFin v2 and OSWorld-Verified.

Kimi K3 demonstrates competitive coding and agent capabilities on in-house benchmarks, often matching or exceeding proprietary and open-weight models. Its performance extends to autonomous, long-horizon engineering tasks such as building a compiler stack and designing a chip, completing multi-day projects in hours. On Kimi Code Bench 2.0, Kimi K3 (max) with Claude Code harness scores 73.7, trailing the top proprietary model by only 3.2 points while that model experienced 13 fallbacks and 1 refusal. In the Coding Experience benchmark, Kimi K3 with Claude Code reaches 59.9, essentially tied with the best proprietary model (59.8) and outperforming Claude Fable 5 (58.0) and Claude Opus 4.8 (53.3). Kimi K3 autonomously constructed an end-to-end compiler and designed an inference chip prototype in a single 48-hour run, closing timing at 100 MHz within a 4 mm² area budget.

In a blind expert evaluation of web development outputs, Kimi K3 is preferred over Claude Opus 4.8 overall, with the largest win margin occurring in 3D, WebGL, and shader tasks. The model also leads in website and UI cloning, while games show a closer but still positive advantage. Kimi K3 achieves an overall win rate of 58.6% against Claude Opus 4.8, with only 27.6% losses, yielding a net win advantage of 31.0 percentage points. The strongest relative performance is in 3D/WebGL/Shader prompts, where Kimi K3 wins 72.7% of comparisons and loses just 13.6%, resulting in a 59.1-point win–loss gap.

The SiTU-GLU activation introduces bounded outputs via tanh saturation, preventing unbounded growth while preserving near-origin behavior. Scaling up Kimi K3 yields a 2.5× efficiency improvement, and the model consistently matches or surpasses leading proprietary models on coding, agentic, and long-horizon engineering benchmarks. In blind web development evaluations, Kimi K3 is preferred over Claude Opus 4.8, with the largest advantages in 3D, WebGL, and shader tasks.