One-sentence Summary

Google DeepMind introduces Gemma 4, a family of open-weight multimodal models ranging from 2.3B to 31B parameters that feature dense and Mixture-of-Experts (MoE) architectures, an encoder-free 12B variant ingesting raw audio and image patches, and a thinking mode, achieving state-of-the-art performance on STEM, multimodal, and long-context tasks while rivaling larger frontier models.

Key Contributions

• The paper introduces a unified, encoder-free architecture in the 12B model that processes raw audio and image patches directly.
• A thinking mode is integrated that generates reasoning traces before responding, which improves overall model performance.
• System-level efficiency improvements including Quantization-Aware Training, multi-token prediction drafters, and KV cache sharing enhance inference speed, memory, and long-context handling, leading to a performance leap over Gemma 3 and competitive results with larger open models.

Introduction

Large language models increasingly demand open-weight architectures that combine multimodal understanding, complex reasoning, and efficient deployment. Prior models, including earlier Gemma versions, struggled with long-context KV cache memory explosion, memory fragmentation from separate modality encoders, and limited reasoning depth. The authors introduce Gemma 4, a family of dense and mixture-of-experts models that address these gaps with a thinking mode that generates reasoning traces before responding, local-to-global attention ratios and KV cache sharing to reduce memory overhead, multi-token prediction for speculative decoding, quantization-aware training for memory efficiency, and a unified encoder-free architecture that directly projects raw audio and image patches into the LLM embedding space, achieving frontier-level performance comparable to much larger open models.

Method

The authors leverage a decoder-only Transformer architecture for the Gemma 4 models, incorporating pre-norm and post-norm with RMSNorm, as well as QKNorm. The model family includes dense architectures with varying effective parameter counts and a Mixture of Experts (MoE) model. To optimize long-context efficiency, the authors employ a local-to-global attention ratio pattern and improve memory efficiency by reusing keys as values in global attention layers. Positional encoding utilizes $\alpha$α-RoPE on global attention layers and standard RoPE on local layers, significantly reducing the global KV cache.

For multimodal capabilities, the models integrate vision and audio modalities. Smaller models utilize dedicated Vision Transformers and audio encoders based on the Universal Speech Model. However, for the unified 12B model, the authors introduce an encoder-free architecture. This paradigm replaces separate encoders with lightweight projection modules. Vision patches are processed via a large matrix multiplication with 2D coordinate-based positional embeddings, while raw audio chunks are projected directly into the LLM embedding space without additional positional encoding.

To accelerate inference, the authors train a small autoregressive Multi-Token Prediction (MTP) drafter head for speculative decoding. In this procedure, the model's last layer activations from the previous step and token embeddings are fed into the MTP head. The MTP head generates future tokens sequentially using a separate embedder and a 4-layer Transformer block that cross-attends to the key-value pairs of the main model, as shown in the figure below:

This design eliminates the need for MTP prefill and supports any draft length. For efficient decoding in smaller models, the authors reduce overhead by replacing the projection operation to the entire vocabulary with a top-k operation on clusters of tokens, significantly reducing the final matrix multiplication size while maintaining a similar acceptance rate.

The pre-training process utilizes a large-scale, diverse dataset encompassing web documents, code, images, and audio, processed with a SentencePiece tokenizer. The authors apply rigorous data filtering to decontaminate benchmarks and mitigate unsafe content. Following pre-training, models undergo instruction tuning to enable a thinking mode, allowing the generation of reasoning traces before answering. This phase involves careful data filtering to optimize performance and minimize hallucinations.

Finally, the authors implement Quantization-Aware Training to provide efficient quantized models. They focus on mobile quantization with low bitwidth weights and activation quantization, as well as blockwise quantization. Quantization-Aware Training is also applied to the image and audio encoders, yielding substantial reductions in memory footprint and on-device latency.

Experiment

The final Gemma 4 models are evaluated through blind human side-by-side comparisons on Arena, a variety of static benchmarks spanning text, vision, audio, and long-context tasks, and rigorous safety testing conducted without safety filters. Gemma 4 31B is the top open dense model, and the 26B-A4B variant performs on par with much larger open models, while the E2B and E4B models match or surpass the much larger Gemma 3 27B across all benchmarks. The models also show major improvements in content safety across all categories relative to previous Gemma generations, with minimal policy violations and low unjustified refusals.

Gemma 4 encompasses a range of model sizes from compact audio-visual models (E2B, E4B) to large dense and mixture-of-experts (MoE) models up to 31B parameters. The 31B dense model achieves top open-model performance on human evaluations, while the efficient E2B matches the much larger Gemma 3 27B with roughly 10x fewer parameters. All models show significant improvements over previous Gemma generations in benchmarks, long-context tasks, and content safety. E2B, with roughly 10x fewer parameters, reaches performance comparable to Gemma 3 27B on static benchmarks. Gemma 4 31B is the leading open dense model in human-rated side-by-side evaluations. The 26B-A4B mixture-of-experts model performs on par with much larger open models on human evaluations. E4B equals or surpasses Gemma 3 27B on all vision benchmarks. E4B outperforms Gemma 3 27B on long-context capabilities. All Gemma 4 models show major content safety gains over Gemma 3 and 3n, with minimal policy violations and low unjustified refusals.

Pre-training spans TPUv5p and TPUv6e chips, with each configuration tuned to minimize step time. Data sharding is fixed at 16 across all models, while sequence and replica sharding scale with model size. Larger models benefit from slice-granularity elasticity, reducing failure recovery from minutes to seconds, and ZeRO-3 optimizer state sharding. Data sharding is consistently set to 16 for every model, from E2B to 31B. Replica counts vary non-monotonically with model size, peaking at 48 for the 12B model on TPUv5p.

Quantization significantly reduces memory footprint for all model sizes, with mobile quantization shrinking the smallest model to under 1 GB and Q4_0 compressing the largest model by roughly 70%. The int8 KV cache for a 32k-token context adds only a modest overhead, ranging from 0.05 GB to 1.10 GB, keeping quantized inference practical. Mobile quantization (mixed int2/int4) compresses the E2B checkpoint from 4.6 GB to just 0.8 GB, a reduction of over 80%. Q4_0 quantization reduces the 31B model's weight memory from 64 GB to 19.2 GB, while its KV cache adds only 1.1 GB. The 26B-A4B mixture-of-experts model sees both total and active weight memory shrink under Q4_0, from 52.0/7.6 GB to 16.2/2.8 GB.

On the Arena Text leaderboard, open-weight models with mixture-of-experts architectures and hundreds of billions of parameters dominate the top ranks, yet the dense Gemma 4 31B achieves the highest Elo among open dense models and matches the performance of these far larger systems. The smaller Gemma 4 26B-A4B likewise competes on par with models over 30 times its total parameter count. The top open dense model is a 31B-parameter system, outperforming mixture-of-experts models with up to 1.6 trillion total parameters. Gemma 4 26B-A4B matches the Elo of open models that have over 700 billion total parameters, despite being only a fraction of their size.

Gemma 4 models in thinking mode consistently outperform the non‑thinking Gemma 3 27B. The 31B variant achieves the strongest results across all benchmarks, while the lightweight E2B model roughly matches the 27B baseline despite having an order of magnitude fewer parameters. The 26B‑A4B and 12B models also show strong performance, with the 26B‑A4B closely trailing the 31B on several tasks. Gemma 4 31B leads by a wide margin on all six benchmarks, including MMLU Pro and AIME 2026. The E2B model, with 10× fewer parameters, achieves competitive performance on GPQA Diamond and outperforms the 27B baseline on math and coding benchmarks. Gemma 4 26B‑A4B scores only slightly below the 31B on MMLU Pro and AIME 2026, making it a strong smaller alternative. The 12B model, the smallest Gemma 4 variant, already surpasses the 27B baseline on MMLU Pro.

Experiments across model sizes, from compact audio-visual models to the 31B dense and mixture-of-experts architectures, demonstrate that the Gemma 4 31B achieves top open-model performance on human evaluations and matches much larger open systems, while the efficient E2B matches the previous Gemma 3 27B with roughly 10x fewer parameters. All models show significant improvements over prior generations in benchmarks, long-context tasks, and content safety, and quantization techniques enable memory-efficient deployment, with the smallest model compressed to under 1 GB. In thinking mode, the entire Gemma 4 family surpasses the non-thinking Gemma 3 27B, with the 31B leading all benchmarks.