One-sentence Summary

Researchers from Renmin University of China, Ant Group, and colleagues present Ring-2.5-1T-Zero, a 1-trillion-parameter model trained with zero RL and a stable pipeline featuring clipped importance sampling, training-inference ratio correction, and mixed-precision control, which exhibits emergent reasoning behaviors like self-verification and parallel reasoning, achieving competitive math benchmark performance and structured, concise chain-of-thought.

Key Contributions

• A stable training pipeline for zero reinforcement learning at the trillion-parameter scale is built, combining clipped importance sampling, training-inference ratio correction, and mixed-precision control to resolve numerical and structural bottlenecks.
• Scaling to 1T parameters significantly improves sample efficiency and performance, revealing a two-phase learning process (discovery then sharpening) and the spontaneous emergence of advanced reasoning behaviors such as self-verification and structured formatting, making hand-crafted heuristics unnecessary.
• A structured evaluation framework is proposed to assess chain-of-thought quality along comprehensibility, reproducibility, and efficiency, and the resulting model, Ring-2.5-1T-Zero, produces clear, concise reasoning traces while achieving competitive results on seven mathematical benchmarks.

Introduction

The authors tackle the challenge of scaling "zero RL" for chain-of-thought reasoning to trillion-parameter models. Prior work demonstrated that reinforcement learning with verifiable rewards applied directly to a base model can elicit emergent reasoning without supervised fine-tuning, but these studies were limited to smaller scales due to computational constraints. Naive application at scale introduced critical issues: reasoning traces became unreadable and unstructured, standard algorithms like GRPO introduced length bias leading to uncontrolled verbose outputs, and models were locked into a single reasoning depth regardless of problem complexity. The authors’ main contribution is a minimalist yet stable self-iterative training pipeline for a 1-trillion-parameter model that relies on clipped importance ratio policy optimization, training-inference ratio correction, sample-level loss normalization, and tier-based adaptive training. This pipeline avoids heavy human-crafted heuristics and instead unlocks spontaneous emergence of structured formatting, self-verification, parallel reasoning, and other advanced cognitive strategies purely through scale, validating the “bitter lesson” that computation overtakes human-engineered design.

Method

The authors adopt a minimalist training framework designed to elicit and scale reasoning capabilities from a base pretrained model without any human-annotated data. The core philosophy is to introduce algorithmic modifications only when strictly necessary to stabilize the iterative learning process. The entire training pipeline consists of four distinct phases: an initial reinforcement learning stage that bootstraps chain-of-thought reasoning, a self-distillation phase that compresses and stabilizes the model, a second RL stage that sustains optimization without uncontrolled length growth, and a final RL stage that equips the model with adaptive reasoning depth for different difficulty levels. Infrastructure optimizations for mixed-precision and context parallelism ensure that the long-context RL training remains numerically stable and computationally efficient.

In the first RL stage, reasoning elicitation, the model rarely generates step-by-step derivations because reasoning tokens have very low probability under the base policy. To amplify these tokens aggressively, the authors use a clipped importance-sampling policy gradient. Unlike standard PPO-clip, which zeroes out gradients outside the clipping range, this variant applies a stop-gradient to the clipped ratio while allowing gradient flow for all tokens, ensuring every token contributes to building reasoning abilities. The objective is:

where $q$q is a question from dataset $\mathcal{D}$D, $\{o_i\}${oi​} are group rollout responses from the inference engine $\pi_{\mathrm{S}}$πS​, $\hat{A}_{i,t}$A^i,t​ are advantage estimates computed with group-normalized rewards, $\operatorname{sg}(\cdot)$sg(⋅) is the stop-gradient operator, and $\hat{\rho}_{i,t}$ρ^​i,t​ is the clipped importance ratio.

A critical stability challenge arises from numerical discrepancies between the training engine (Megatron) and the inference engine (SGLang) due to differences in floating-point precision and kernel implementations. To prevent these micro-discrepancies from compounding into macroscopic collapses, the numerator of the importance sampling ratio is replaced with the actual training-engine logits:

where only an upper bound $\epsilon_{\mathrm{high}}$ϵhigh​ is applied to prevent excessively large updates while no lower bound is enforced. This correction accurately reflects the true divergence between engines. Additionally, a KL divergence penalty against a frozen reference model constrains the policy drift, and a token-level loss is used, summing over all tokens without normalizing by response length, which explicitly encourages longer generation. The response window length is gradually expanded in a curriculum manner. The combined first-stage loss is:

with $\beta$β balancing exploration and stability.

While the first stage successfully elicits reasoning, the unconstrained token-level loss leads to excessively long, redundant chain-of-thought traces, and the accumulated numerical errors between engines cause optimization instability. To resolve these issues, a self-distillation phase is introduced. The first-stage expert policy $\pi_{\mathrm{expert}}$πexpert​ samples multiple rollouts per query. A two-step length refinement process selects the shortest correct reasoning trace and then prompts the model to self-evaluate and remove any remaining redundant segments. The refined corpus is used to fine-tune the original base model via standard supervised learning:

The distilled model retains reasoning abilities while producing concise traces and effectively resets the accumulated numerical errors, providing a clean starting point for the next RL stage.

In the second RL stage, sustained optimization, the same clipped importance-sampling policy gradient and ratio correction are retained, but the loss strategy is changed to a sample-level normalization. Instead of summing unnormalized token losses, the gradient is normalized by the response length:

This eliminates the bias toward length growth inherited from the token-level loss while still allowing the model to optimize for accuracy. The KL penalty is also removed because the distilled model already provides a strong starting point, and further regularization would limit exploration.

The third RL stage introduces adaptive reasoning depth through a tier-based training strategy. Questions are partitioned into three difficulty tiers—low, medium, high—each associated with a maximum token length and a specific system prompt $p_k$pk​. The objective incorporates these prompts:

This conditioning teaches the model to allocate compute adaptively: concise answers for trivial queries and exhaustive chain-of-thought reasoning for complex problems. During inference, users can control the reasoning mode via the system prompt.

To make long-context RL practical at scale, the authors implement two infrastructure optimizations. First, mixed-precision control stabilizes training by keeping the main model body in BF16 but computing the attention softmax and the LM head in FP32. These components involve exponentiation, which naturally amplifies small numerical errors; high precision in these spots eliminates sudden loss spikes and closes the numerical gap between the training and inference engines. Second, a tailored context parallelism strategy breaks the latency bottleneck of standard ring attention. For MLA layers, an all-to-all operation along the head dimension allows each device to compute full attention for a subset of heads across the entire sequence, with low communication volume due to compressed key-value latent representations. For Lightning Attention layers, a single AllGather operation broadcasts local state matrices to all devices simultaneously, enabling immediate computation. Both optimizations are mathematically equivalent to ring attention and produce identical gradients.

Experiment

A multi-stage zero RL pipeline starting from pretrained base models is evaluated on mathematical benchmarks, showing that reinforcement learning alone can develop competitive reasoning capabilities without human demonstrations. The evaluation extends beyond accuracy to measure chain-of-thought quality along three axes: comprehensibility via LLM-as-judge pairwise comparisons, reproducibility through distillation gains for weaker students, and efficiency by comparing token counts on correct traces, with results indicating that the emergent traces are more logically structured, easier to transfer, and less verbose than strong baselines. Additional analyses reveal that careful stabilization such as KL penalties and logit mismatch correction, model scaling, and a dynamic curriculum are essential for stable training, while the RL process undergoes a distinct shift from boundary expansion to solution sharpening, and qualitative inspection uncovers spontaneous self-verification and parallel reasoning behaviors.

Frontier proprietary models achieve very high pass@1 accuracy on recent math benchmarks, with several exceeding 95% on AIME 2026 and the best reaching 96.7% on HMMT Feb. 2026. The Ring-2.5-1T-Zero model, trained purely through RL without human data, reaches 84.2% on AIME 2026 after first-stage training, showing that large language models can self-discover complex reasoning. Multi-stage refinement and adaptive inference modes further improve performance while allowing flexible compute trade-offs. Claude Opus 4.8 leads on HMMT Feb. 2026 with 96.7%, and Qwen3.7-Plus scores 97.0% on AIME 2026, setting top marks among compared proprietary models. Starting from a pretrained base model with no human-annotated reasoning traces, Ring-2.5-1T-Zero’s first RL stage reaches 84.2% on AIME 2026, confirming emergent chain-of-thought from properly regularized exploration. Self-distillation followed by sample-level and tier-based RL yields consistent gains, while adaptive inference (Low, Medium, High) lets users trade off between reasoning depth and compute cost, though joint training across lengths introduces slight negative transfer on the deepest mode.

Frontier proprietary models now achieve near-perfect accuracy on advanced math benchmarks, with top scores exceeding 96% on both AIME 2026 and HMMT Feb. 2026. Training Ring-2.5-1T-Zero purely via reinforcement learning from a base model without any human reasoning traces yields 84.2% on AIME 2026 after one stage, confirming that large language models can spontaneously develop complex chain-of-thought reasoning through properly regularized exploration. Subsequent self-distillation and multi‑stage refinement provide consistent gains, while adaptive inference modes let users trade reasoning depth for compute savings, though joint training across depths introduces minor negative transfer on the deepest setting.