One-sentence Summary

Qwen Team (Alibaba Group) et al. systematically compare 30 variants of gating-augmented softmax attention on 15B MoE and 1.7B dense models trained on 3.5 trillion tokens, finding that a head-specific sigmoid gate applied after SDPA introduces non-linearity and query-dependent sparsity that improve performance, training stability, and long-context extrapolation while mitigating attention sink, a mechanism subsequently adopted in Qwen3-Next.

Key Contributions

• A comprehensive comparison of 30 gating-augmented softmax attention variants in 1.7B dense and 15B Mixture-of-Experts models trained on 3.5 trillion tokens shows that a head-specific sigmoid gate after Scaled Dot-Product Attention consistently improves performance.

• The paper attributes this improvement to non-linearity on the low-rank mapping inside softmax attention and to query-dependent sparse gating scores that modulate the SDPA output. The gated models also show improved training stability, tolerance for larger learning rates, and improved scaling behavior.

• Post-SDPA sparse gating eliminates attention sinks in dense and Mixture-of-Experts models, mitigates massive activation, and improves long-context extrapolation and context-length extension without retraining. Related attention-sink-free models and code are released, and the most effective SDPA output gating is used in Qwen3-Next models.

Introduction

Gating mechanisms are widely used in neural networks and modern Transformer attention to control information flow, improve training, and modulate token mixing. Although gating appears in recent architectures such as state space models, sparse attention, and mixture of experts designs, prior work often leaves its direct contribution unclear because gating effects can be mixed with routing, sparsity, or other architectural changes. The authors address this by systematically testing gating at different positions inside standard softmax attention and find that elementwise gating after the scaled dot product attention output is especially effective. This simple change adds non linearity, introduces input dependent sparsity, improves perplexity and MMLU, reduces loss spikes, enables larger learning rates, eliminates attention sinks, and improves length generalization.

Dataset

• Training data composition: the authors train on subsets of a 4T-token, high-quality corpus that spans multilingual, math, and general knowledge content.

• Source details: the paper does not name the exact original data sources, subset sizes, mixture ratios, or detailed filtering rules beyond describing the corpus as high quality and broad in domain.

• Processing details: training samples are prepared with a fixed sequence length of 4096; no additional cropping strategy, metadata schema, or post-tokenization filtering is described.

• Training use: the model variants are trained on selected portions of this high-quality token pool rather than necessarily the full 4T tokens.

• Evaluation data: performance is measured on held-out perplexity test sets covering English, Chinese, Code, Math, Law, and Literature, as well as few-shot benchmark datasets including HellaSwag, MMLU, GSM8K, HumanEval, C-Eval, and CMMLU.

Method

The authors leverage a gated attention layer that augments standard multi-head attention with dynamic filters placed inside the attention computation. The basic gate is formulated as:

where $Y$Y is the feature tensor to be modulated, $X$X is the input used to compute the gating score, $W_{\theta}$Wθ​ are learnable gate parameters, $\sigma$σ is an activation function such as sigmoid, and $Y'$Y′ is the gated output. In this formulation, $\sigma(XW_{\theta})$σ(XWθ​) acts as a learnable and input-dependent mask that controls information flow by preserving or suppressing individual features. The gate is trained jointly with the attention weights and does not require a separate routing or top-level selection module.

The gating mechanism can be inserted at several distinct positions in the attention block. Refer to the framework diagram.

The displayed architecture marks the possible gate locations after the query, key, and value projections, denoted by $G_4$G4​, $G_3$G3​, and $G_2$G2​; after the scaled dot product attention output, denoted by $G_1$G1​; and after the final dense projection, denoted by $G_5$G5​. The most effective placement is the SDPA output position $G_1$G1​, where the gate directly modulates the aggregated attention output before the dense projection. The gate can be headwise, using one scalar score per head, or elementwise, using a score vector with the same dimensionality as the output. The strongest variant uses elementwise modulation with head-specific parameters. Multiplicative gating is the primary form, while additive gating is also considered with SiLU because the additive term needs an unbounded activation.

This design is not merely a normalization or regularization trick. In a standard attention head, the output for token $i$i and head $k$k can be written as:

Here $S_{ij}^{(k)}$Sij(k)​ is the attention score from token $i$i to token $j$j in head $k$k, and $X_j$Xj​ is the input representation of token $j$j. The two linear transformations $W_V^{(k)}$WV(k)​ and $W_O^{(k)}$WO(k)​ can be combined into one low-rank linear map:

This merging exposes a structural limitation: the path from value projection to dense output is effectively a low-rank linear transformation, and grouped query attention can reduce this expressiveness further. Gating at $G_2$G2​ or $G_1$G1​ breaks the purely linear character of this path by introducing non-linearity in different places. For example, gating after the value projection can be viewed as:

while gating after the SDPA output corresponds to:

In both cases, the gate injects a non-linear operation between the value and dense transformations. This improves the expressive capacity of the otherwise low-rank attention pathway. Additive gating can introduce some non-linearity, but multiplicative gating provides a stronger input-dependent modulation. RMSNorm-like operations can also improve performance by introducing normalization and mild non-linear stabilization, but they do not match the same query-conditioned filtering behavior of the full gate.

During training, the gate parameters are learned end to end with the model. In practical dense model training, the parameter budget may be held fixed by reducing the feedforward width when the gate is added. The main trained configuration uses head-specific multiplicative gating with sigmoid scores. This configuration has been observed to reduce final loss and mitigate loss spikes under larger learning rates and larger batch sizes, enabling more stable optimization. In the long training comparison shown with the architecture, the gated model follows a smoother loss trajectory and reaches a lower final loss than the baseline under comparable settings.

The gating score distribution reveals why some gate placements are stronger than others. As shown in the figure below:

The displayed distributions reveal that effective gating produces sparse score distributions. SDPA output gating has the lowest mean score and the strongest concentration near zero, meaning the gate frequently suppresses many attention features. Value-position gating produces higher mean scores and a weaker sparse pattern, while head-shared gating also increases the average score and weakens the filtering behavior. This supports the need for head-specific gating, because each attention head can specialize its own filtering rule rather than being forced to share a single gate with other heads.

A further distinction is query dependency. In SDPA output gating, the gate is computed from the hidden state associated with the current query. This allows the gate to filter contextual contributions that are irrelevant to the current token being processed. In contrast, value-position gating is driven by past value hidden states and is therefore less aligned with the current query. The observed performance difference suggests that useful gating is not only sparse, but also input-dependent and query-conditioned. To test this, the authors also evaluate input-independent gating by using fixed or zero-initialized learnable parameters with a sigmoid. This still improves the baseline, likely through added non-linearity, but it does not reach the performance of query-dependent gating. Additional ablations with non-sparse sigmoid variants, which constrain scores to a high range and remove strong suppression, reduce the gains, confirming that sparse gating scores are a central component of the method.

Experiment

Experiments are conducted on both sparse MoE and dense transformer models trained on large multilingual, coding, and reasoning corpora and evaluated on broad benchmarks covering knowledge, reasoning, coding, language proficiency, and perplexity to test the effectiveness of alternative attention gating positions and gate designs. The results show that element-wise or headwise sigmoid gating at the SDPA output consistently improves both perplexity and downstream task performance, outperforming baselines and simple parameter-expansion methods. Further analyses indicate that SDPA output gating works by introducing useful nonlinearity in low-rank attention mappings and inducing query-dependent, head-specific sparsity, which reduces massive activations and attention-sink effects, thereby stabilizing training and enabling more robust scaling to larger learning rates, batch sizes, and longer contexts.

Gating applied at the SDPA output and the value map improves model performance over the baseline and parameter-expansion baselines. SDPA output gating provides the strongest gains, while value map gating also remains effective. Head-specific gating is especially important, and its benefits are linked to sparse, input-dependent gating behavior. SDPA output gating gives the best perplexity and benchmark improvements among the gating variants Value map gating also reduces perplexity and improves benchmarks, though less strongly than SDPA output gating Head-specific gating outperforms shared gating across attention heads Sparse input-dependent gating is associated with better performance and reduced attention artifacts

Across dense model configurations, training scales, and hyperparameter settings, element-wise sigmoid gating after SDPA consistently improves validation perplexity and downstream benchmark performance relative to the baseline, even when FFN width is reduced to preserve parameter count. The gating mechanism also appears to improve optimization stability, enabling effective training with larger learning rates and batch sizes where baseline variants can become unstable. SDPA output gating improves perplexity and multiple evaluation scores across different model and training settings. Gating supports more stable training under larger learning rates and batch sizes than the baseline. Sandwich normalization can restore convergence in some unstable settings, but gives smaller gains than SDPA gating.

Adding nonlinear augmentations to attention outputs generally lowers perplexity and improves downstream scores compared with the baseline. Sigmoid gating of the SDPA output gives the clearest gains, while RMSNorm also improves perplexity with almost no additional parameters. The analysis suggests that input-dependent gating increases sparsity and reduces massive activations, which may also support more robust context length extension. Sigmoid elementwise gating outperforms the baseline across perplexity and downstream tasks. RMSNorm on each attention head reduces perplexity without adding meaningful parameters. Gating with reduced input dependence is associated with stronger massive activations and attention sink effects. Models with SDPA gating show stronger performance than the baseline at extended context lengths.

Across the compared gating settings, the strongest qualitative trends point to the SDPA output gating as the most effective variant, especially when gating is elementwise or headwise and query dependent. These settings show sparser gating scores together with lower first-token attention scores and reduced maximum hidden state activations, consistent with weaker attention sink behavior. In contrast, shared gating, value-only gating, or less sparse gating variants produce higher gate scores and weaker reductions in first-token attention, indicating that input-dependent head-specific sparsity is important. Sparse gating scores are strongest with SDPA output gating. Head-specific gating reduces first-token attention more than shared gating. Value gating lowers activations but does not reduce first-token attention as effectively. Less sparse gating increases both massive activations and attention sink.

Within the trained sequence-length range, gated attention gives a slight advantage over the baseline. After context extension, both models decline on previously supported lengths, but the gated model suffers a smaller drop. At longer evaluated lengths, the gated model performs substantially better, suggesting it adapts more robustly to context extension. Gated attention provides modest gains at the original training length. Context extension degrades performance for both methods, with the baseline declining more sharply. The gated model retains stronger accuracy at extended sequence lengths.

The experiments evaluate attention gating across dense model configurations, training scales, hyperparameter settings, ablations of gate placement, and context extension, testing whether gating improves language modeling, downstream tasks, optimization stability, and attention behavior. Element-wise sigmoid gating after SDPA emerges as the strongest variant, with value-map gating also helpful, and head-specific, input-dependent sparse gating provides the clearest benefits over shared or value-only alternatives. These results indicate that SDPA output gating improves perplexity and benchmark performance while reducing massive activations and attention sink effects, supporting more stable training and more robust context extension.