One-sentence Summary

Tsinghua University researchers propose Riemannian Isometric Policy Optimization (RIPO), demonstrating that PPO-Clip’s exploration collapse stems from an inconsistent Euclidean metric on the policy Riemannian manifold and instead enforcing isometric updates to balance exploration and exploitation, yielding up to 60% improvement over GRPO on competition-level reasoning benchmarks.

Key Contributions

• PPO-Clip's fundamental limitation is a geometric mismatch: it implicitly uses a Euclidean metric to constrain policy updates, which conflicts with the intrinsic Riemannian manifold geometry of policy distributions and causes exploration collapse.
• Riemannian Isometric Policy Optimization (RIPO) enforces isometric updates on the policy Riemannian manifold to balance exploration and exploitation.
• RIPO induces statistical homoscedasticity for a favorable bias-variance trade-off, and experiments on seven competition-level benchmarks show it significantly outperforms existing LLM reinforcement learning methods, including a 60% improvement over GRPO on AIME24.

Introduction

Reinforcement learning (RL) has become a key paradigm for enhancing the reasoning capabilities of large language models, particularly in long-horizon tasks such as mathematical problem solving. However, the PPO-Clip algorithm widely adopted in these settings suffers from exploration collapse: the policy rapidly narrows to a small set of high-probability actions, systematically under-updating rare but informative actions and stifling performance. Prior attempts to mitigate this, such as raising clipping boundaries, offer heuristic improvements within the PPO-Clip framework but do not identify or resolve the root cause. The authors uncover a fundamental geometric mismatch: PPO-Clip measures policy discrepancies with a Euclidean metric on importance ratios, while the natural policy space is governed by KL divergence and carries a Riemannian geometry. This mismatch produces pathological updates that are overly conservative for low-probability actions and overly aggressive for high-probability ones, driving premature concentration. To correct this, they propose Riemannian Isometric Policy Optimization (RIPO), which reinterprets policy updates on the Riemannian manifold, dynamically adapting the clipping boundary to permit larger changes for low-probability actions and restrain high-probability changes. The resulting isometric update balances exploration and exploitation, significantly improving reasoning performance, and the geometric isometry also induces statistical homoscedasticity, leading to a favorable bias-variance tradeoff.

Method

The authors first identify a fundamental geometric flaw in the standard PPO‑Clip mechanism that leads to exploration collapse in policy optimization for large language models. PPO‑Clip restricts the importance ratio $r_{s,a}(\theta) = \pi_\theta(a|s) / \pi_{\theta_{\mathrm{old}}}(a|s)$rs,a​(θ)=πθ​(a∣s)/πθold​​(a∣s) to a fixed interval $[1-\epsilon, 1+\epsilon]$[1−ϵ,1+ϵ], but this constraint is symmetric and independent of the current action probability. As a result, high‑probability actions can easily grow toward their maximum allowed value (e.g., from 0.8 to 0.96 for $\epsilon=0.2$ϵ=0.2), while low‑probability actions receive an almost negligible absolute increase (e.g., from 0.01 to 0.012). This asymmetry suppresses exploration diversity and causes the policy to collapse to a narrow set of already favored tokens.

The root cause of this behavior is a geometric mismatch between the Euclidean metric implicitly used by PPO‑Clip and the intrinsic Riemannian geometry of policy space. PPO‑Clip treats the squared ratio deviation $(r(\theta)-1)^2$(r(θ)−1)2 as a global measure of policy change, assuming that equal deviations imply equal trustworthiness. In reality, the proper measure of policy discrepancy is the KL divergence, which induces a Riemannian metric on the statistical manifold. A second‑order Taylor expansion of the KL divergence reveals that the geometric distance between two policies is

Thus, the distance depends on the local probability simplex: high‑probability regions expand distances, while low‑probability regions shrink them. PPO‑Clip’s clipping boundary completely ignores this scaling, allowing high‑probability tokens to consume far more trust‑region budget and low‑probability tokens to be severely constrained, even when the Euclidean deviation is identical.

To correct this, the authors propose a Riemannian Isometric Clip (RIC). Instead of enforcing a uniform ratio deviation, RIC requires that every state‑action update moves exactly the same geometric distance on the policy Riemannian manifold. They impose

which, after solving for the ratio, gives a distribution‑dependent clipping threshold:

In practice this is implemented as a dynamic clipping boundary $\epsilon_{s,a}(\pi_{\theta_{\mathrm{old}}}) = \sqrt{\delta / \pi_{\theta_{\mathrm{old}}}(a|s)}$ϵs,a​(πθold​​)=δ/πθold​​(a∣s)​, where $\delta$δ is the maximum permitted geometric distance (trust‑region radius). Consequently, low‑probability actions are granted a wider clipping range, enabling meaningful exploration, while high‑probability actions are kept within a tighter bound, preventing them from monopolizing the update budget. Revisiting the earlier example, with $\delta=0.02$δ=0.02, the high‑probability action (0.8) is capped at 0.92 (instead of 0.96 under PPO), while the low‑probability action (0.01) can rise to 0.024 (instead of 0.012). Both updates consume precisely the same geometric distance, achieving isometry.

The authors further show that this geometric isometry implies statistical homoscedasticity in off‑policy importance sampling. The variance of an importance‑weighted advantage estimator is dominated by terms proportional to $\pi_{\theta_{\mathrm{old}}}(x)\, r(x)^2$πθold​​(x)r(x)2. PPO‑Clip reduces variance by capping $r(x)$r(x), but it introduces bias by discarding low‑probability samples. RIC’s dynamic clipping yields a variance contribution near the boundary that is $\mathcal{O}(\delta)$O(δ), independent of the action probability. Thus, RIC achieves a principled bias–variance trade‑off: variance is controlled, while bias is substantially lower than PPO‑Clip because rare yet valuable tokens are no longer overly suppressed. The geometric equivalence of updates on the policy manifold directly translates to equalized second‑order contributions across samples, delivering homoscedastic variance.

Building on this, the full Riemannian Isometric Policy Optimization (RIPO) algorithm is designed for LLM reinforcement learning tasks. RIPO inherits the group‑relative advantage estimator from GRPO: for each query $q$q, a group of $G$G responses is sampled from the old policy, and the advantage $\hat{A}_{i,t}$A^i,t​ is computed by normalizing the sparse reward with the group mean and standard deviation. The loss function then applies token‑level policy gradient clipping, as in DAPO, to balance the gradient influence across responses of varying lengths. The key difference is that the clipping boundary is now the dynamic RIC threshold $\epsilon_{i,t}(\pi_{\theta_{\mathrm{old}}})$ϵi,t​(πθold​​) rather than a fixed constant. Formally, the RIPO objective is

with $\epsilon_{i,t} = \sqrt{\delta / \pi_{\theta_{\mathrm{old}}}(o_{i,t} \mid q, o_{i,<t})}$ϵi,t​=δ/πθold​​(oi,t​∣q,oi,<t​)​. This design ensures that every token update respects the same trust‑region budget on the policy manifold, directly counteracting the exploration collapse of PPO‑Clip while maintaining stable and sample‑efficient training for large language models.

Experiment

RIPO is evaluated on four large language models of varying sizes and architectures using DAPO-Math-17k for training and seven competition-level mathematical reasoning benchmarks, compared against six reinforcement learning algorithms with different clipping strategies. The method consistently outperforms baselines, achieving faster convergence and stable training dynamics without entropy collapse or gradient spikes, while ablation studies confirm robustness to its hyperparameter and the importance of symmetric clipping. Transfer to PPO objectives, coding, and search tasks, along with Pass@k analysis, further demonstrate RIPO's broad applicability and its ability to break capacity boundaries by mitigating exploration collapse in long-horizon reasoning.

RIPO consistently achieves the highest average performance across all four base model scales, outperforming GRPO by up to 37.2% and surpassing improved clipping methods like DCPO. Gains are especially large on difficult benchmarks such as AIME24 and BRUMO25, and the advantage grows with model size, demonstrating scalable and robust reasoning improvements. RIPO delivers the best overall average, exceeding GRPO by 37.2% on Qwen3-1.7B-Base and consistently leading over DAPO, GSPO, GMPO, and DCPO across all tested backbones. Improvements are most pronounced on harder datasets like AIME24 and BRUMO25 and scale up with larger models, highlighting RIPO's ability to break through capacity boundaries and maintain policy diversity.

An ablation study on the RIPO clipping parameter δ for AIME24 shows that symmetric values from 0.02 to 0.08 yield stable performance around 40–44 Avg@8, peaking at 0.05. Introducing asymmetry with a tighter upper clipping bound (δ_high) relative to the lower bound leads to a sharp drop below 29 Avg@8, confirming that balanced policy update constraints are critical. Symmetric clipping with δ between 0.02 and 0.08 all achieve Avg@8 above 40, with the best result of 43.8 at δ=0.05. Setting the upper clipping bound smaller than the lower bound (e.g., δ_low=0.08, δ_high=0.02) causes Avg@8 to fall below 29, a substantial decline from symmetric settings.

RIPO consistently outperforms all compared methods, achieving the top scores on every benchmark with improvements over GRPO ranging from 3.8 to over 14 points. While GPPO and Clip-Cov generally improve upon GRPO, Clip-Cov shows a slight decline on HMMT25, underscoring its less stable behavior. The results demonstrate RIPO’s superior stability and effectiveness across diverse reasoning tasks. RIPO delivers double-digit gains over GRPO on four of the six benchmarks, including a 14.2-point boost on BRUMO25 and SMT25. GPPO provides modest but consistent improvements over GRPO, never falling below GRPO on any task. Clip-Cov improves over GRPO on most benchmarks but drops by 1.2 points on HMMT25, revealing occasional instability. On the challenging AIME25 benchmark, RIPO (29.2) gains 8.4 points over GRPO, far exceeding the next best method.

RIPO-Clip (RIC) consistently achieves the highest Avg@1 across all model sizes on the GSM8k math dataset when used with the PPO objective, outperforming PPO-Clip, DAPO-Clip, and DCPO-Clip. The gains are largest for smaller models, with RIPO-Clip adding 3.1 points on the 0.5B model. Training dynamics show that RIPO-Clip avoids the entropy collapse and gradient spikes observed with standard PPO-Clip, promoting more stable optimization and sustained exploration. RIPO-Clip delivers absolute Avg@1 improvements over PPO-Clip of +3.1 (0.5B), +2.4 (1.5B), +1.8 (7B), and +1.2 (14B) on GSM8k. On the 1.5B model, RIPO-Clip maintains policy entropy and exhibits smoother gradient norms, while PPO-Clip suffers rapid entropy decay and sharp gradient fluctuations.

On AIME-25 and HMMT-25, the base Qwen3-8B model's pass@k saturates early, while RL-trained methods improve scaling with k. RIPO consistently achieves the best pass@k at every budget and continues to benefit from larger k, breaking through the base model's capacity ceiling. The base model's pass@k plateaus at 16.7% on AIME-25 and 6.7% on HMMT-25 as k increases, indicating inherently limited reasoning capacity. RIPO attains a pass@128 of 60.0% on AIME-25 and 45.3% on HMMT-25, outperforming all other RL algorithms and showing continuous improvement with more samples. The gap between RIPO and other RL methods grows with larger sampling budgets, with DCPO, GRPO, and DAPO trailing at all measured k values.

RIPO is evaluated across multiple model sizes and challenging math reasoning benchmarks, comparing against several reinforcement learning baselines including GRPO and its variants. The experiments show that RIPO consistently delivers the best overall performance, particularly on difficult tasks, with gains growing as model scale increases and as sampling budgets expand, demonstrating its ability to push beyond base-model capacity limits. Ablations confirm that symmetric clipping is critical for stability, and additional studies reveal that RIPO prevents entropy collapse and gradient spikes during training, yielding smoother optimization. Overall, the findings highlight RIPO’s superior robustness, scalability, and training stability across diverse settings.