One-sentence Summary

Researchers from Tsinghua University propose Single-rollout Asynchronous Optimization (SAO), an asynchronous reinforcement learning framework for large language models that replaces GRPO’s group-wise sampling with single-rollout sampling and employs strict double-side token-level clipping to improve training stability and off-policy robustness, leading to superior performance on agentic coding and reasoning benchmarks and successful deployment in the GLM-5.2 (750B) training pipeline.

Key Contributions

• Single-rollout Asynchronous Optimization (SAO) replaces group-wise sampling with single-rollout sampling (one rollout per prompt) and incorporates value-model training to reduce off-policy effects in asynchronous RL for large language models.
• A strict double-side token-level clipping strategy is introduced to improve optimization stability, enabling stable training for one thousand steps.
• SAO consistently outperforms GRPO and its variants on agentic coding and reasoning benchmarks such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench, and is deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).

Introduction

The push toward scaling reinforcement learning (RL) for large language models (LLMs) has exposed a bottleneck: synchronous training pipelines idle while waiting for the longest rollouts, especially in agentic and coding tasks with highly variable trajectory lengths. Asynchronous RL avoids this by continuously feeding rollouts to the optimizer, but it introduces off-policy instability because trajectories come from stale policy versions. Group-wise advantage methods like GRPO compound the problem, as they require all responses for a prompt to finish before updating, causing latency-driven policy lag and proving incompatible with online environments that provide only single trajectory feedback. The authors propose Single-rollout Asynchronous Optimization (SAO), an asynchronous RL algorithm that uses single-rollout updates instead of group sampling. SAO stabilizes training under policy lag with token-level importance sampling and strict double-sided clipping, strengthens value-model training via more frequent critic updates with frozen attention, and handles multi-turn agent trajectories with a skip-observation GAE estimator that computes advantages across action boundaries without propagating noise through environment observation tokens. Evaluated on agentic coding and math reasoning benchmarks, SAO trains stably for approximately a thousand steps and outperforms improved GRPO, while its single-rollout design adapts naturally to dynamic online learning.

Method

The authors introduce SAO to tackle training instability and off-policy drift in asynchronous reinforcement learning (RL) training. By leveraging a simple token-level clipping strategy and single rollout as an alternative to group-wise sampling, asynchronous RL can be stably scaled to thousands of training steps and achieve significant performance improvements.

As shown in the figure below:

In contrast to group-wise sampling algorithms like GRPO, which must wait for all trajectories in a group to be generated before training begins, the SAO framework allows each trajectory to become available for training immediately upon completion. This single-rollout design eliminates the imbalanced generation bias and waiting time inherent in group-wise approaches.

To address the policy lag between rollout models and training models in asynchronous RL, the authors propose Direct Double-Sided Importance Sampling (DIS). In decoupled PPO, importance sampling typically tracks three distinct models. However, tracking exact behavior probabilities in asynchronous RL is computationally prohibitive. To resolve this, the authors directly use the rollout policy $\pi_{\mathrm{rollout}}$πrollout​ as the behavior proxy and the current policy $\pi_{\theta}$πθ​ for importance sampling, computing the probability ratio as: $r_t(\theta) = \exp(\log \pi_{\theta}(a_t | s_t) - \log \pi_{\mathrm{rollout}}(a_t | s_t))$rt​(θ)=exp(logπθ​(at​∣st​)−logπrollout​(at​∣st​)) This eliminates the need for separate old-policy inference. Furthermore, they employ a double-sided calibration token-level masking strategy. Unlike standard PPO clipping, which only clips selected off-policy tokens, this approach restricts the trust region to the interval $[1 - \epsilon_{\ell}, 1 + \epsilon_h]$[1−ϵℓ​,1+ϵh​]. Tokens falling outside this range are masked from gradient computation entirely to prevent instabilities from extreme policy divergence. The optimization objective is formulated as: $L(\theta) = \hat{\mathbb{E}}_t \left[ f(r_t(\theta), \epsilon_{\ell}, \epsilon_h) \hat{A}_t \log \pi_{\theta}(a_t | s_t) \right]$L(θ)=E^t​[f(rt​(θ),ϵℓ​,ϵh​)A^t​logπθ​(at​∣st​)] where the calibration function $f(x; \epsilon_{\ell}, \epsilon_h)$f(x;ϵℓ​,ϵh​) is defined as: $f(x; \epsilon_{\ell}, \epsilon_h) = \begin{cases} x, & \text{if } 1 - \epsilon_{\ell} < x < 1 + \epsilon_h \\ 0, & \text{otherwise} \end{cases}$f(x;ϵℓ​,ϵh​)={x,0,​if 1−ϵℓ​<x<1+ϵh​otherwise​

To further reduce off-policy drift and high variance in gradient estimation inherent to single-rollout optimization, the authors implement several strategies to optimize value modeling. First, they introduce a Faster Value Update mechanism. Recognizing that inaccurate value models $V_{\phi}$Vϕ​ lead to noisy advantage estimates $\hat{A}_t$A^t​, they decouple the optimization frequencies of the policy and value models. For every single gradient update applied to the policy $\pi_{\theta}$πθ​, they enforce $K$K updates to the value network $V_{\phi}$Vϕ​ (where $K > 1$K>1), facilitating faster adaptation of value estimates.

Second, to stabilize value model training, they employ a Frozen-Attention strategy. Observing that gradient instability originates primarily from Full Attention layers rather than Mixture-of-Experts (MoE) layers, they freeze the parameters of the attention modules in $V_{\phi}$Vϕ​ during RL training and optimize only the MoE projections. This effectively regularizes the value model by relying on pre-trained attention weights for semantic capability.

Third, for agentic tasks with trajectory structures containing model actions and environment feedback, standard Generalized Advantage Estimation (GAE) introduces noise when calculating advantage across the discontinuous boundary between actions and observations. The authors derive a Skip-Observation GAE that explicitly modifies the Bellman target to bypass environment feedback tokens. The advantage is defined as: $\hat{A}(a_{i, N}) = \delta + \gamma \lambda \hat{A}(a_{i+1, 0})$A^(ai,N​)=δ+γλA^(ai+1,0​) where the temporal difference residual $\delta$δ bridges the observation gap: $\delta = r_t + \gamma V(a_{i+1, 0}) - V(a_{i, N})$δ=rt​+γV(ai+1,0​)−V(ai,N​) This constrains advantage estimation to rely purely on model outputs. Finally, to support these mechanisms, the authors scale the data used for value model pretraining to overcome the cold start problem in value estimation, providing a robust initialization point for the single-rollout and faster update mechanisms.

Experiment

The experiments evaluate the SAO method across agentic reasoning, coding, and simulated online learning tasks, using a Qwen3-30B-A3B-Thinking-2507 model finetuned on tool-integrated reasoning data. SAO consistently outperforms baselines such as standard GRPO and VAPO, which suffer from training collapse, while its design choices, including faster value updates and frozen-attention critic training, prove essential for stability and performance. Training dynamics analysis shows that SAO achieves higher explained variance, smoother critic gradients, and effective clipping, and in an online writing simulation with shifting style preferences, it adapts faster than a running-mean baseline by leveraging a state-dependent value model.

GLM-4.7 and GPT-5 High lead across all math reasoning benchmarks, with GLM-4.7 achieving the highest accuracy on AIME2025, HMMT, and IMOAnswerBench. Qwen3-30B-A3B performs competitively without Python, but enabling Python execution causes a drastic accuracy collapse. Supervised fine-tuning on Python data partially recovers performance but remains below the no-Python setting. GLM-4.7 reaches the top accuracy on AIME2025 (95.7%), HMMT (93.5%), and IMOAnswerBench (82.0%), with GPT-5 High close behind. Enabling Python tool use in Qwen3-30B-A3B drops AIME2025 accuracy from 85.0% to 14.6%, while SFT with Python partially recovers to 80.4% but still trails the no-Python variant.

On the SWE-Bench Verified benchmark, SAO reaches 29.8% accuracy, surpassing the base Qwen3-30B-A3B model (23.0%) and the stable GRPO variant with DIS (27.0%). Standard GRPO collapses early during training, while the DIS strategy stabilizes learning; SAO’s critic-based advantage estimation then provides an additional performance boost after roughly 400 training steps. SAO achieves the highest accuracy (29.8%) on SWE-Bench Verified, outperforming both the base model (23.0%) and GRPO with DIS (27.0%). GRPO with DIS prevents the early training collapse seen in standard GRPO, and SAO further widens the performance gap after about 400 steps, demonstrating both stability and improved final results.

Ablations show that updating the value model twice per policy step yields higher accuracy than a single update, as more frequent critic updates better track policy changes. Frozen-attention value training outperforms full-parameter updates, with the latter causing a large accuracy drop on AIME2025 and a slight decline on BeyondAIME, indicating that freezing attention regularizes critic optimization for complex reasoning. Reducing critic updates from two to one per policy step lowers accuracy on both evaluated datasets. Full-parameter value training degrades performance relative to frozen-attention training, with a particularly severe impact on the AIME2025 benchmark.

Ablation experiments confirm that both faster critic updates and frozen-attention value training are essential for SAO's performance. Removing faster value updates or freezing attention leads to consistent accuracy drops, while a running-mean baseline and vanilla VAPO both substantially underperform, validating each component. Reducing critic updates from two per batch to one degrades accuracy, indicating that a single update cannot reliably track rapid policy changes. Switching from frozen-attention to full-parameter value updates hurts performance, suggesting that attention-freezing helps regularize critic optimization in complex reasoning tasks. Replacing the learned value model with a running-mean reward baseline causes a large accuracy drop, demonstrating the necessity of a state-dependent critic for reliable advantage estimation.

Math reasoning benchmarks show GLM-4.7 and GPT-5 High achieving top accuracy, while enabling Python tool use causes a severe accuracy collapse for Qwen3-30B-A3B that supervised fine-tuning only partially recovers. On SWE-Bench Verified, the SAO method surpasses base and GRPO variants by using a critic-based advantage estimator with frozen-attention value training and frequent critic updates, which stabilizes reinforcement learning and prevents early training collapse. Ablations confirm that both faster value model updates and frozen attention are essential, and replacing the learned critic with a running-mean baseline substantially degrades performance, highlighting the importance of a state-dependent value function for complex reasoning tasks.