One-sentence Summary

The Qwen Team presents Qwen3.8-Flash-Next, a $125\text{B-parameter}$125B-parameter sparse MoE with $6\text{B}$6B activated tokens and $51\text{B}$51B n-gram embedding parameters, which outperforms its $397\text{B-A}17\text{B}$397B-A17B predecessor on eight of fourteen benchmarks at $\frac{1}{3}$31​ the activated parameters, 1/3 the training tokens, and roughly $\frac{1}{9}$91​ the FLOPs, using a layer-wise hybrid of Gated DeltaNet and global attention (replaced by Qwen Sparse Attention during continued pretraining), a four-branch Gated Residual stream, and prefetched n-gram tables, while jointly optimizing loss, downstream accuracy, efficiency, and stability with the Muon optimizer.

Key Contributions

• Introduces Qwen3.8-Flash-Next, a 125B-parameter sparse mixture-of-experts model with 6B activated parameters per token, matching or nearly matching the prior 397B-A17B flagship on fourteen pre-training benchmarks while using one-third of the activated parameters, one-third of the training tokens, and roughly one-ninth of the training FLOPs.
• Combines a layer-wise hybrid of Gated DeltaNet and global attention, with Qwen Sparse Attention replacing full attention at continued pretraining, a four-branch Gated Residual stream read through an elementwise gate, and a host-memory n-gram embedding layer; these changes shift optimal hyperparameters upward, eliminate batch-size warmup, and improve stability, as shown by stress tests at four times the optimal learning rate where the new recipe remains stable while the previous structure spikes.
• Demonstrates that loss and downstream accuracy do not always move together, as enlarging the n-gram vocabulary lowers loss monotonically while downstream accuracy saturates, and validates the design via a joint evaluation of loss, benchmarks, training/prefill/decode cost, and stability, yielding a recipe that is simultaneously more efficient, more capable, and more stable.

Introduction

The authors introduce Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B total parameters and 6B activated per token, plus 51B parameters in n-gram embedding tables stored off-accelerator. The goal is to match the quality of the previous 397B-A17B flagship while using about a third of the activated parameters, a third of the training tokens, and roughly a ninth of the training FLOPs. Prior work faced trade-offs where architectural changes affect downstream performance, training and serving costs, and stability simultaneously, and simple fixes often fail under late-stage evaluation or stress. The authors’ main contribution is a coupled design across four components: a layer-wise hybrid of Gated DeltaNet and global attention for token mixing, Qwen Sparse Attention for efficient long-context prefill, a widened residual stream with an elementwise gate for capacity and stability, and an n-gram embedding layer for extra parameters without added per-token compute. They also refit the scaling law for the new optimizer and architecture, and validate stability under stress tests that expose instabilities at moderate scale. The result is a base model that leads the predecessor on eight of fourteen benchmarks and trails by at most 2.6 points on the rest, with no loss spikes during full-scale training.

Method

The authors address the tension between efficient local processing and persistent content-dependent memory by introducing a layer-wise hybrid of Gated DeltaNet (GDN) and global attention. GDN compresses the prefix into a fixed-size recurrent state and updates that state according to the current content, while interleaved global-attention layers retain direct token-level retrieval. Specifically, one full-attention layer is placed in every four layers, with the remaining three using GDN.

The GDN token-mixing path is illustrated below.

Given a normalized residual-stream input, GDN computes content features using learned projections followed by short depthwise causal convolutions. Queries and keys are L2-normalized to bound magnitudes and stabilize the rank-one delta transition. The recurrent state is updated using a gated delta rule, where a data-dependent decay gate globally controls the lifetime of the existing state, and a write gate controls the delta update. This targeted erase-and-write operation distinguishes GDN from purely additive linear attention. The head outputs are then normalized and modulated by a sigmoid output gate. For efficiency, the GDN kernel is optimized using a fused linear-attention kernel library.

To alleviate the quadratic computational bottleneck of softmax attention in long-context scenarios, the authors adopt Qwen Sparse Attention (QSA). QSA employs a lightweight indexer that compresses the sequence into micro-block representations, estimates their importance, and selects the most relevant context for attention computation.

As shown in the figure below:

The compressed lightweight indexer adopts a multi-query attention structure. Keys are partitioned into non-overlapping blocks and compressed via average pooling before applying partial rotary position embeddings. Block-level importance scores are obtained through block-causal scoring, allowing each query to score only fully observed blocks. Given a token budget, the top-scoring blocks are selected, expanded to original token indices, and combined with tail tokens for core attention computation.

The training of QSA involves two stages during continued pretraining. In the first stage, dense distillation, the full-sequence attention distribution of the backbone is distilled into the indexer. The token-level teacher distribution is max-pooled to align with block-level indexer scores, and the KL divergence is minimized to train the indexer. In the second stage, sparse training, the entire backbone is trained under the guidance of the indexer to adapt to sparse attention patterns. The indexer KL loss is computed only over the selected top-k blocks.

The effectiveness of this training process is demonstrated in the training loss curves.

The loss curves for QSA and full attention remain highly consistent, with the overall loss difference on the order of $10^{-4}$10−4.

Architecture ablations highlight the benefits of the intra-layer compression used in QSA.

QSA matches the full-attention baseline at a relative indexer latency of 0.25, outperforming cross-layer index sharing methods. Furthermore, QSA maintains performance with a small number of indexer query heads.

The efficiency gains of QSA are substantial at long sequence lengths.

By compressing the key sequence, QSA reduces indexer complexity and achieves significant kernel-level speedups for both prefill and decode stages compared to dense grouped-query attention.

To address signal attenuation in pre-normalization networks, the authors introduce Gated Residual (GR) connections, which widen the residual stream into multiple parallel branches. GR replaces block pre-normalization with a read operator that predicts elementwise gating scores per branch and channel from all branches, averaging the gated branches into the block input. The block output is written to every branch through a data-dependent scalar per branch. This design allows specific branches to preserve early attention outputs across many layers while others stay local, improving both training stability and downstream performance.

For optimization, the authors utilize the matrix-based optimizer Muon, applying Newton-Schulz iterations to orthogonalize momentum for two-dimensional weights. Fused parameters like attention query-key-value projections are split at per-head granularity before orthogonalization to avoid mixing singular directions across unrelated sub-blocks. Additionally, an updated hyperparameter scaling law is developed, predicting substantially larger batch sizes and learning rates. The new recipe eliminates the need for batch-size warmup, as Muon preserves data efficiency at larger batch sizes, leading to more stable and efficient large-scale training.

Experiment

The experiments validate the Qwen3.8-Flash-Next architecture across its key components. The GDN hybrid attention improves over both full-attention and SWA baselines on most benchmarks, while QSA sparse attention matches full-attention performance on short-context tasks and delivers substantial gains on long-context retrieval, with kernel-level speedups up to 7.6x at 1M context. N-gram embedding ablations show a single layer at Layer 2 is sufficient, and scaling its vocabulary improves loss and Chinese benchmarks without consistent downstream gains. Hyperparameter scaling experiments confirm the new recipe's predicted batch size and learning rate are near-optimal, yielding stable training with no need for batch-size warmup. Stress tests demonstrate the gated residual and Muon optimizer provide a large stability margin over the AdamW baseline, and the full model achieves results comparable to a much larger baseline at roughly one-ninth the training cost.

The GDN hybrid architecture outperforms both the full-attention and SWA-hybrid baselines on most benchmarks, particularly in reasoning, math, and code tasks. While full attention leads on some general knowledge benchmarks, the GDN hybrid achieves the highest average score across all nine tasks. GDN hybrid improves on 8 of 9 benchmarks relative to full attention and on 7 of 9 relative to SWA hybrid. GDN hybrid shows the largest gains in math and code benchmarks, such as MATH and EvalPlus. Full attention retains an edge on MMLU-Pro, while SWA hybrid leads on MMLU and SuperGPQA. The GDN hybrid achieves the best average score across all nine benchmarks.

Replacing full attention with Qwen Sparse Attention (QSA) maintains or slightly improves performance across most benchmarks, with the largest gains in reasoning and coding tasks. The only notable decline is on the SuperGPQA benchmark, while average scores across all categories improve modestly. QSA improves average performance over full attention, driven by gains in reasoning and coding benchmarks. The largest relative improvement with QSA is on the MATH benchmark, while SuperGPQA shows a slight drop. Multilingual and STEM results also improve with QSA, though the gains are smaller than those in reasoning and coding.

Applying Qwen Sparse Attention (QSA) to the model improves average long-context retrieval performance across RULER and MRCR benchmarks, with the largest gains observed at longer sequence lengths. While full attention retains a slight edge on some shorter-range and MRCR settings, QSA yields substantial improvements on the most challenging 512K–1M and 512K/1M MRCR cases. QSA raises the average score across both benchmarks from 78.76 to 80.93. The largest RULER gain with QSA occurs in the 512K–1M range, where the score jumps from 90.08 to 93.00. On MRCR, QSA more than doubles the full-attention score at 512K (from 30.66 to 40.53) and improves the 1M score from 20.71 to 26.44. Full attention remains slightly better on MRCR at 128K and 256K, but QSA closes the gap and outperforms at longer lengths.

Under four-step speculative decoding, replacing full attention with Qwen Sparse Attention (QSA) yields slightly higher mean MTP accepted lengths across all evaluated benchmarks. The gains are consistent but modest, with the average accepted length improving from 4.06 to 4.07. This indicates that QSA maintains or slightly improves decoding efficiency while reducing attention indexing overhead. QSA improves the average MTP accepted length from 4.06 to 4.07 compared to full attention. Gains are observed on every benchmark, with the largest relative improvement on MT-Bench (from 3.44 to 3.47). The improvements are consistent but small, suggesting QSA preserves decoding quality while reducing indexing cost.

Residual stream widening with multiple branches improves performance over standard pre-norm across all evaluated benchmarks. Dynamic gating of branch weights outperforms static weighting, and a further refined gating variant (GR) achieves the best overall results, particularly in STEM and multilingual tasks. All widened residual variants reduce loss and improve benchmark scores compared to the pre-norm baseline. Dynamic branch weighting consistently outperforms static weighting across knowledge, STEM, reasoning, and multilingual benchmarks. The refined gating variant (GR) achieves the highest average MMLU and the best scores on STEM and multilingual tasks, with the lowest loss among the compared configurations.

The experiments validate the proposed hybrid attention and architectural modifications across diverse benchmarks. The GDN hybrid attention achieves the best average performance across nine tasks, with notable gains in reasoning, math, and code, though full attention retains an edge on some general knowledge tasks. Replacing full attention with Qwen Sparse Attention (QSA) maintains or slightly improves most benchmark scores, with the largest gains in reasoning and coding, and also improves long-context retrieval, especially at 512K and 1M sequence lengths. Under speculative decoding, QSA preserves decoding efficiency with a marginal improvement in accepted token length. Finally, widening the residual stream with multiple branches and dynamic gating consistently outperforms the standard pre-norm baseline, with the refined gating variant yielding the best results overall.