One-sentence Summary

Researchers from Zhejiang University, Tencent, and an independent researcher propose STEER, a principled entropy-modulation method for Reinforcement Learning with Verifiable Rewards (RLVR) that derives an approximate token-level entropy change expression identifying four governing factors, exposes the heuristic limitations of prior entropy interventions, and adaptively reweighs tokens to mitigate entropy collapse, consistently outperforming state-of-the-art baselines across six math and three coding benchmarks.

Key Contributions

• Derives an approximate expression for token-level entropy change at each policy update step in RLVR, identifying four governing factors: clipping strategy, advantage, token probability, and conditional entropy. This provides a unified framework for explaining how existing entropy intervention methods influence these factors.
• Exposes a fundamental limitation of current entropy mitigation approaches: they rely on heuristic adjustments to only one or two of the four identified factors, leaving other relevant factors unconsidered, which inherently limits their effectiveness in preventing entropy collapse.
• Introduces STEER, a principled entropy-modulation method that adaptively reweighs tokens based on estimated entropy change. Experiments across six math reasoning and three coding benchmarks demonstrate that STEER mitigates entropy collapse and consistently outperforms state-of-the-art baselines.

Introduction

Large language models have shown strong reasoning abilities, and a key driver is Reinforcement Learning with Verifiable Rewards (RLVR), which uses policy-gradient algorithms like PPO and GRPO to optimize policies against verifiable reward signals. This approach has been central to post-training scaling of reasoning performance. However, a major pitfall is entropy collapse, where policy entropy drops rapidly during training, weakening exploration and making rollouts homogeneous. In group-based algorithms like GRPO, this reduces the discriminative power of advantage scores and stalls training. Existing mitigation strategies, such as Clip-Higher, Positive-Reweighting, and Entropy-Aware Advantage, are largely heuristic, with poorly understood mechanisms, so their effectiveness is limited and entropy remains loosely controlled.

The authors provide a principled analysis of token-level entropy dynamics, deriving an approximate expression for entropy change governed by four factors: clipping strategy, advantage, token probability, and conditional entropy. They show how existing methods affect these factors and identify limitations, including that prior methods apply modulation to only a small subset of tokens and consider only partial factors. Based on this, they introduce STEER (Stabilizing Token-level Entropy-changE via Reweighting), which adaptively down-weights tokens with large entropy changes to regulate per-step entropy dynamics and sustain exploration. Experiments across six math and three coding benchmarks show STEER outperforms 10 baselines and generalizes across model scales, families, and RL algorithms.

Method

The authors establish a theoretical framework to quantify token-level entropy dynamics during reinforcement learning from verification (RLVR) updates. By analyzing the policy gradient of the Group Relative Policy Optimization (GRPO) algorithm, they derive an approximation for the per-token entropy change, denoted as $\Omega(s)$Ω(s). This metric reveals that entropy evolution is jointly influenced by the clipping indicator, the advantage signal, the old policy probability, the current token probability $\pi_\theta$πθ​, and the state entropy $\mathcal{H}(s)$H(s).

To illustrate the interplay between token probability and state entropy, the authors define a function $\delta(\pi_\theta, \mathcal{H})$δ(πθ​,H) that captures the contribution of these specific terms to the overall entropy change. The behavior of this function is visualized across different entropy levels, demonstrating how the interaction between generation probability and entropy dictates whether the policy distribution becomes more peaked or more uniform.

Building on this quantitative analysis, the authors categorize token updates into four distinct qualitative quadrants within the advantage-probability space. This framework elucidates the competing forces at play during training: Quadrant I represents exploitation where high-probability tokens in correct outputs decrease entropy; Quadrant II corresponds to exploration where low-probability tokens in correct outputs increase entropy; Quadrant III involves suppression where low-probability tokens in incorrect outputs decrease entropy; and Quadrant IV signifies error correction where high-probability tokens in incorrect outputs increase entropy. Entropy collapse is understood as a state where the entropy-decreasing forces in Quadrants I and III overwhelm the entropy-increasing forces in Quadrants II and IV.

To address the instability caused by these competing dynamics, the authors propose a method termed STEER (Stabilizing Token-level Entropy-change via Reweighting). This approach introduces an adaptive weight $\lambda(s)$λ(s) for each token to modulate its entropy change from $\Omega(s)$Ω(s) to $\lambda(s)\Omega(s)$λ(s)Ω(s). The weight is computed using an exponential decay function based on the magnitude of the normalized entropy change:

Here, $\alpha$α controls the decay rate, effectively mapping entropy changes to a range $[\lambda_{min}, 1]$[λmin​,1]. By attenuating tokens with large entropy changes (whether increasing or decreasing), STEER stabilizes the training dynamics and prevents the policy from collapsing into low-entropy states while maintaining sufficient exploration capabilities.

Experiment

The evaluation spans math reasoning across six benchmarks and coding tasks across three benchmarks, comparing STEER against ten baselines including GRPO, OPO, and Clip-Cov. STEER consistently outperforms all baselines on math reasoning, improving average performance by 2.2 points over the second best method, and exceeds GRPO on coding tasks by at least 1% across model scales. In extreme entropy scenarios where baseline methods suffer from entropy collapse or uncontrolled increases, STEER maintains stable entropy and achieves the highest accuracy. The method also generalizes across multiple model architectures, sizes, and RLVR algorithms, including RLOO and OPO.

The table compares covariance-based estimation with the proposed method across three models. The proposed method consistently yields much lower mean squared error and substantially higher positive correlation coefficients, while covariance-based estimates show near-zero or negligible correlations. Overall, the proposed approach provides a more reliable quantitative characterization of per-token entropy changes. The proposed method reduces mean squared error by several orders of magnitude compared to covariance-based estimation across all tested models. Correlation metrics (PCC and SRCC) are near zero or slightly positive with covariance estimation, but become strongly positive with the proposed method. The improvement is consistent across different model sizes, suggesting robust generalization of the entropy-intervention mechanism.

The table compares how entropy intervention methods incorporate factors like clipping, policy probability, advantage, and state entropy, revealing that each existing method only covers a subset of these. In contrast, the proposed approach explicitly regulates entropy to maintain stable training dynamics and consistently outperform baselines across math reasoning, coding, and generalization settings. STEER prevents entropy collapse or excessive uncertainty, achieving stable entropy after an initial decline, unlike other methods. Even under extreme conditions with almost no ratio clipping, STEER maintains relatively stable performance and the highest accuracy across test sets. STEER yields consistent improvements across different backbones and RL algorithms, indicating its general applicability.

STEER consistently outperforms all baseline methods across math reasoning benchmarks, achieving the highest average score. It surpasses the second-best method by 2.2 points and the third-best by 2.9 points, with improvements observed on every dataset. The gains are attributed to explicit entropy regulation that preserves exploration for more informative trajectories. STEER achieves the top average performance across all listed math reasoning datasets. STEER improves average accuracy by 2.2 points over the runner-up and 2.9 points over the third-place method.

On Qwen2.5-14B, STEER surpasses all baseline methods across every math reasoning dataset, achieving the highest average performance. It consistently leads over strong RLVR and entropy intervention baselines, indicating robust generalization in math reasoning tasks. STEER achieves the top average score of 45.2%, ahead of all competing methods. STEER outperforms the second-best method (OPO) by about 2 points on average. STEER records the highest scores on individual datasets such as AIME24, AMC23, and MATH500.

The table compares STEER against GRPO on real-world coding tasks across three Qwen2.5-coder model sizes. STEER consistently outperforms GRPO on every dataset and model configuration, with improvements of at least one percentage point and often larger gains at the 14B scale. STEER achieves higher scores than GRPO on all three datasets and all three model sizes. The performance advantage is more pronounced on the 14B models, indicating better scalability with larger backbones.

The proposed entropy estimation method consistently outperforms covariance-based estimation across all tested models, yielding far lower mean squared error and strongly positive correlations, whereas covariance estimates show near-zero or negligible correlations. STEER, the proposed entropy intervention method, explicitly regulates entropy to maintain stable training dynamics and outperforms existing baselines across math reasoning, coding, and generalization tasks, showing robust improvements across different backbones, model scales, and RL algorithms. On math reasoning benchmarks, STEER achieves the highest average accuracy, surpassing the second-best method by over two points, and on coding tasks, it consistently beats GRPO across all tested model sizes with larger gains at larger scales. These results confirm the reliability and general applicability of the proposed approach.