One-sentence Summary

The Kimi Team releases Kimi K3, a 2.8T-parameter Mixture-of-Experts model with 104 billion activated parameters, native vision capabilities, and a 1-million-token context window, leveraging Kimi Delta Attention, Attention Residuals, and Stable LatentMoE for $\sim 2.5\times$∼2.5× scaling efficiency over Kimi K2, and its post-training reinforcement learning across general, agentic, and coding tasks yields frontier performance that outperforms all other open and proprietary models while trailing only Claude Fable 5 and GPT-5.6 Sol, with full model weights released.

Key Contributions

• Kimi K3, a 2.8T-parameter Mixture-of-Experts model with native vision and a 1M-token context window, is released with full weights; its Kimi Delta Attention, Attention Residuals, and Stable LatentMoE improve overall scaling efficiency by approximately 2.5× over Kimi K2.
• Reinforcement learning is conducted across general, agentic, and coding domains at multiple reasoning-effort levels, then the domain- and effort-specialized policies are consolidated into a unified model via multi-teacher on-policy distillation.
• Infrastructure includes KDA systems co-design, MoonEP for balanced expert-parallel training with memory-efficient management, and a co-located RL system with resumable microVM sandboxes that sustain million-token agentic trajectories.

Introduction

The scaling of large language models has evolved along two axes: pre-training larger models on more data, and increasing test-time compute through reasoning and agentic reinforcement learning. While the open-source community has made rapid progress on test-time scaling, its pre-trained models have remained in the roughly one-trillion-parameter class, causing the gap with proprietary frontier systems to widen. The authors tackle both dimensions simultaneously by introducing Kimi K3, an open 2.8-trillion-parameter mixture-of-experts model with 104 billion activated parameters per token and a one-million-token context window. Architectural innovations—Kimi Delta Attention, Attention Residuals, and Stable LatentMoE—improve long-sequence mixing, cross-layer information flow, and extreme-sparsity optimization, yielding an approximately 2.5× scaling efficiency gain over the previous generation. This foundation is then post-trained with reinforcement learning over multi-effort, long-horizon coding, reasoning, and agentic tasks, unifying diverse capabilities into a single model that establishes a new open frontier while narrowing the gap to the strongest proprietary systems.

Dataset

The authors assemble training data across pre-training, long-context cooldown, and post‑training reinforcement‑learning phases. Below is a breakdown of the main data categories, their composition, processing, and how each is used.

Pre‑training data

• Text corpus: four domains – web text, code, mathematics, and knowledge.
  • Filtering: each domain is cleaned with rule‑based heuristics, classifier‑based quality scoring, and deduplication.
  • Mixture: domain‑specific sampling rates are set through ablation studies on smaller models.
  • Knowledge and mathematics are rephrased using the Kimi K2 recipe (style‑diverse prompts, chunk‑wise autoregressive generation, fidelity verification against the source).
• Vision corpus: follows the taxonomy of Kimi K2.5.
  • Sources: open‑source collections plus in‑house synthesis, filtering, and deduplication pipelines.
  • Contents: captions, interleaved image–text documents, OCR, perception data, video, and visual coding data.
  • Programmatic multimodal data: scaled‑up pairs of code snippets with rendered visuals (SVG, 3D assets, webpages, games, CAD schematics).
  • Coordinate supervision is provided in both absolute and normalized formats for precise, resolution‑robust localization.
• Usage: both text and vision corpora are used throughout pre‑training; the domain mixture and rephrasing are applied directly to the training tokens.

Long‑context cooldown data

• Natural long documents & videos: sourced from raw long‑form content.
  • Cleaning: exact/fuzzy deduplication, perceptual hashing over video frames, heuristic and classifier‑based quality filtering, structural validation.
• Synthetic long‑context data:
  • Constructed by permuting and concatenating multimodal documents and sub‑tasks so that solving a task requires attending to information scattered across the full 1M‑token context.
• Mixture: long sequences are upsampled during the cooldown phase so the distribution is not overwhelmed by short sequences.
• Usage: employed only in the cooldown stage, while the context window is progressively extended from 256K to 1M tokens; this trains the attention mechanism to handle genuine long‑range dependencies.

Post‑training (RL) task data

The post‑training phase uses diverse, synthetically generated task suites; below are the key datasets.

• Kernel optimization tasks

  • A large‑scale suite of GPU kernel tasks, from single‑operator kernels to fused mega‑kernels, sourced from high‑quality GitHub repositories (e.g., Flash Linear Attention).
  • Covers multiple GPU programming approaches (CUDA, Triton, CuTe DSL, Gluon, ThunderKittens, TileLang) and numerical formats (BF16, FP8, FP4).
  • Rewards combine correctness checks (against a PyTorch reference) and performance scoring versus an expert implementation, with a hacking‑detection system that penalizes reward‑hacking strategies.
  • Used as an environment for RL to strengthen GPU kernel optimisation capability.

• Knowledge‑graph‑guided task synthesis

  • A self‑evolving, hierarchically organized knowledge graph is built by agents that recursively expand from seed nodes via web searches. Nodes represent fine‑grained concepts in knowledge‑intensive and coding domains.
  • Tasks are synthesized by sampling nodes at various granularities, retrieving real‑world materials through web queries that combine node keywords with ancestor context, and then assembling the materials into training tasks of multiple types.
  • Used to generate a broad, controllable distribution of RL tasks that target specialized and underrepresented knowledge.

• Personal assistant tasks

  • Based on realistic mock implementations of widely used applications (Gmail, Notion, Slack, Canvas) that preserve core semantics without external APIs.
  • Complex, long‑horizon professional workflows (e.g., HR, legal, finance) play out over multiple simulated days, creating persistent environments with dozens of interdependent events across apps. A single rollout can involve thousands of tool calls and millions of context tokens.
  • The initial workspace is built by agents that autonomously search the web for reference materials and transform them into a coherent task‑relevant environment.
  • Used for RL training of long‑horizon agent behaviour, with evaluation at the individual event level.

• Web development tasks

  • An expert‑curated suite covering a range of artifacts: websites, interactive games, 3D/WebGL scenes, data visualizations, SVGs, and full‑stack applications.
  • Input complexity varies from one‑line scene descriptions to multi‑paragraph specifications.
  • Each task runs in a containerized sandbox; rewards include deterministic functional checks (build success, error‑free runs, structural/pixel‑level similarity for reference replicas) and model‑based judging (source code inspection, artifact interaction). Faked implementations are penalised.
  • Used for RL across diverse agent scaffolds, promoting cross‑scaffold generalization.

Method

The Kimi K3 architecture is designed to scale information flow along three complementary dimensions: sequence length, network depth, and model width. Together with refined training and data recipes, these components yield an approximately 2.5x improvement in overall scaling efficiency over the previous generation.

As shown in the framework diagram below, the model integrates these dimensions into a unified structure.

Hybrid Attention Along the sequence dimension, Hybrid Attention combines three Kimi Delta Attention (KDA) layers with one Gated MLA layer in each block. This 3:1 mixing ratio provides an efficient mechanism for long-context token mixing while retaining selective high-capacity attention.

KDA extends the delta-rule recurrence with a channel-wise forget gate. To address numerical stability in the chunkwise parallel form, the authors replace the unbounded negative-Softplus mapping with a scaled sigmoid to bound the log-decay from below:

$$\pmb{g}_t^h = g_{\min} \operatorname{Sigmoid}(e^{A_h} \pmb{z}_t^h)$$gth​=gmin​Sigmoid(eAh​zth​)

This lower-bounded decay ensures that the reciprocal rescaling factor remains within the BF16 dynamic range, allowing all causal tiles to use dense Tensor Core matrix multiplications rather than explicit position-pair computations for diagonal tiles.

The Gated MLA layers compress the key-value representation into a low-dimensional latent vector, reducing the KV-cache footprint. Kimi K3 applies No Position Encoding (NoPE) to all MLA layers, relying on the intervening KDA layers for position-sensitive mixing. Both KDA and MLA utilize an input-dependent, channel-wise full-rank output gate.

Attention Residuals To extend information access beyond conventional sequential residual accumulation, the authors introduce Attention Residuals (AttnRes). This mechanism allows each layer to selectively retrieve representations from the embedding and preceding blocks. To reduce memory overhead, the layers are partitioned into blocks. Within each block, layer outputs are reduced to a single representation by summation, and full attention is applied across block-level representations. This block structure bounds the inference-time state and enables efficient merging via online softmax.

Stable LatentMoE Along the width dimension, each attention layer is followed by a Stable LatentMoE layer. This module separates the full model width from the routed-expert width, enabling the model to scale channel mixing to 896 routed experts with 16 active experts per token. To address activation explosion and load balancing issues inherent in such extreme sparsity, the authors introduce three components:

1. Normalized LatentMoE: An RMSNorm is inserted between expert aggregation and the up-projection to reduce sensitivity to scale variation.
2. SiTU-GLU: A Sigmoid Tanh Unit GLU activation applies a smooth cap to the linear factor of the Swish gate and the up branch, preserving local response while controlling large-value growth.
3. Quantile Balancing (QB): This auxiliary-loss-free routing method sets each expert bias from the router-score quantile that matches its target load, ensuring balanced dispatch without altering mixture weights.

Native Vision For native multimodal capabilities, the model processes text, images, and videos within a single shared backbone. The vision encoder, MoonViT-V2, is trained entirely from scratch using next-token prediction. This approach yields more stable optimization with lower gradient norms compared to encoders initialized from contrastively pre-trained models.

Visual inputs are encoded by MoonViT-V2 and mapped by a lightweight MLP projector into the shared embedding space. The encoder factorizes attention into intra-frame spatial and inter-frame temporal passes, and employs pixel-shuffle downsampling to reduce visual token count.

Per-Head Muon The model is optimized using Per-Head Muon, a variant of the Muon optimizer where Newton-Schulz orthogonalization is applied to each head's momentum block separately rather than the full projection matrix. This equalizes update scales across heads and improves training stability at larger scales.

Experiment

Kimi K3 is evaluated against state-of-the-art proprietary models across reasoning, coding, agentic, and vision benchmarks, as well as internal evaluations and cyber security tasks. It closely trails the strongest models on reasoning, leads on many agentic and coding suites, and achieves superior cost efficiency, but lags on research-level reasoning, kernel exploitation, and agent behavior. The model excels at orchestration, research agency, and coding, while cyber tests show meaningful exploit-development capability yet a clear gap to human experts on hardened targets. Case studies demonstrate strong autonomous performance on complex technical tasks like kernel optimization, compiler design, and chip design.

SiTU-GLU introduces bounded saturation to both branches of the gated linear unit, closely matching SwiGLU near the origin but limiting the response for large positive inputs. The gate branch uses a sigmoid-weighted tanh, while the up branch uses a standalone tanh, resulting in an output that approaches the product of their scaling factors β1β2 = 100, unlike the unbounded growth of SwiGLU. SiTU-GLU replaces the linear up branch of SwiGLU with a β2-scaled tanh, capping its contribution for large inputs. Near the origin, SiTU-GLU’s curve closely follows SwiGLU, but as x increases it saturates toward the bound |f̄(x)| ≤ 100, while SwiGLU remains unbounded.

Kimi K3 scales up the Mixture-of-Experts architecture by increasing depth, total and activated parameters, introducing a latent MoE dimension, and expanding per-expert hidden size while keeping the hidden dimension unchanged. Together with data and training changes, these structural updates contribute to an approximately 2.5× improvement in overall scaling efficiency compared to Kimi K2. Depth grows by 52% (from 61 to 93 layers). Total parameters increase by 167% and activated parameters by 220%, moving to a much sparser expert configuration. A new latent MoE dimension is introduced, set to half the hidden dimension. Per-expert hidden dimension expands by 50%, while the model hidden dimension remains unchanged. The architectural changes, combined with data and training improvements, yield a scaling efficiency gain of about 2.5× over Kimi K2.

Kimi K3 closely trails the strongest proprietary models, Claude Fable 5 and GPT-5.6 Sol, while consistently outperforming Claude Opus 4.8, GPT-5.5, and GLM-5.2 across most benchmarks. It achieves top or near-top scores in coding and agentic tasks, notably leading on ProgramBench and ranking second on the long-horizon FrontierSWE, but shows a clear gap to the frontier on research-level reasoning challenges like HLE-Full and CritPt. On SWE-Marathon, a GPU-kernel coding suite, Kimi K3 leads Claude Fable 5 by 7 points. Kimi K3 attains the best score on ProgramBench among all evaluated models. In agentic tasks, Kimi K3 sets state-of-the-art results on BrowseComp, DeepSearchQA, ResearchRubrics, and several other suites, while placing second or third on Elo-rated knowledge-work benchmarks GDPval-AA v2 and AA-Briefcase. On FrontierSWE, a long-horizon benchmark, Kimi K3 ranks second with a score of 81.2%, behind only Claude Fable 5. Kimi K3 nearly matches GPT-5.6 Sol on Terminal-Bench 2.1 (88.3% vs. 88.8%). For reasoning, Kimi K3 is competitive on GPQA Diamond (93.5%) but trails Claude Fable 5 and GPT-5.6 Sol on the research-level CritPt and on HLE-Full both with and without tools.

On in-house coding benchmarks, Kimi K3 performs competitively, matching the top proprietary system on Coding Experience and trailing by a small margin on Kimi Code Bench 2.0. The open-weight Claude Fable 5 model falls behind, with lower scores and higher refusal rates. These results reflect evaluation at maximum reasoning effort, with harness assignments shown for each model. Kimi K3 achieves the top score on the Coding Experience benchmark, narrowly ahead of the best proprietary model. On Kimi Code Bench 2.0, the proprietary model leads but required 13 fallbacks and 1 refusal, while Kimi K3 reaches a close second without such issues.

In blind expert evaluation of web development outputs, Kimi K3 (max) outperforms Claude Opus 4.8 (max) overall, with a win rate of 58.6% versus 27.6% losses. The advantage is largest in 3D/WebGL/Shader tasks, where Kimi K3 wins 72.7% of prompts and loses only 13.6%. Website and UI cloning tasks are more competitive, showing the highest tie rate. Kimi K3 leads by 31.0 percentage points overall (win minus loss). In 3D/WebGL/Shader tasks, the win-minus-loss margin widens to +59.1 points. Website/UI Clone tasks have the highest tie rate (21.1%), indicating closer performance.

The evaluation covers Kimi K3's activation design, scaling efficiency, and performance across diverse tasks. SiTU-GLU bounds the gated unit's output to prevent unbounded growth, while structural changes to the MoE architecture yield a 2.5x improvement in efficiency. On public and in-house benchmarks, Kimi K3 matches or surpasses strong proprietary models in coding, agentic, and web development tasks, but falls short of the frontier on research-level reasoning challenges.