One-sentence Summary

NVIDIA researchers demonstrate that large-scale reinforcement learning with a simple math-only then code-only RL curriculum significantly enhances reasoning in compact 7B and 14B models, surpassing distillation-based baselines with gains of $+14.6\%$+14.6%/$+17.2\%$+17.2% on AIME 2025 and $+6.8\%$+6.8%/$+5.8\%$+5.8% on LiveCodeBench, while relying on robust data curation and on-policy parameter updates for stability.

Key Contributions

• Demonstrates that large-scale reinforcement learning significantly enhances the reasoning capabilities of strong, small- and mid-sized supervised fine-tuned models, achieving results that surpass state-of-the-art distillation-based approaches, with gains of +14.6%/+17.2% on AIME 2025 for the 7B/14B models.
• Proposes a simple two-stage RL recipe that trains first on math-only prompts, then on code-only prompts, where math-only RL also improves code reasoning performance (+6.8%/+5.8% on LiveCodeBench for the 7B/14B models), and subsequent code-only RL further boosts code benchmarks with minimal or no math degradation.
• Develops a robust data curation pipeline that collects challenging prompts with high-quality, verifiable answers and test cases to enable verification-based RL across math and code, and identifies key insights including curriculum learning with progressively increasing response lengths, on-policy parameter updates for training stabilization, and evidence that RL both elicits foundational reasoning acquired during pretraining and supervised fine-tuning and extends the model's ability to solve previously unsolvable problems.

Introduction

The authors address the growing demand for building reasoning models through large-scale reinforcement learning (RL). Since the release of OpenAI o1 and the open-sourcing of DeepSeek-R1, RL-driven reasoning has attracted significant attention, yet the original DeepSeek-R1 report omitted critical details on data curation and the RL training recipe, leaving the community without a reproducible path. Prior studies have also concluded that distillation outperforms RL for small and mid-sized models, recommending RL only for the largest systems such as DeepSeek-V3-671B or Llama-3.1-Nemotron-Ultra-253B. The authors challenge this assumption by showing that large-scale RL can substantially enhance strong small and mid-sized SFT models (DeepSeek-R1-Qwen-Distilled-7B and 14B), achieving results competitive with distillation-based approaches at 7B and surpassing them at 14B. Their main contributions include conducting math-only RL before code-only RL, where they observe that math RL also boosts code reasoning performance and later code RL does not degrade math abilities; a systematic data curation pipeline for verifiable math problems and coding test cases; and detailed ablations revealing that curriculum learning with progressively increasing response length improves training efficiency, on-policy updates stabilize the RL process, and RL both elicits existing reasoning capabilities and expands the model's ability to solve previously unsolvable problems.

Dataset

The authors build two separate RL training datasets, one for math and one for code, each with its own curation pipeline to ensure data quality and reward signal reliability.

Math-only RL dataset

• Sources and composition: The dataset combines DeepScaler and NuminaMath, covering algebra, combinatorics, number theory, and geometry.
• Contamination filtering: A 9-gram filter removes overlap with common math benchmarks. Additional rules exclude questions with multiple sub-questions, multiple-choice or true/false formats, overly long or complex answers, proof-based questions, non-English content, references to figures, and excessively brief prompts.
• Noise reduction: Because NuminaMath is largely scraped from online sources and processed with OCR and parsing tools, the authors use DeepSeek-R1 to attempt each question up to eight times. They keep only questions where a rule-based verifier confirms a majority-voted correct answer. Questions that DeepSeek-R1 cannot solve consistently are discarded, as human inspection often reveals ambiguity or OCR errors.
• Difficulty balancing: Questions requiring fewer than 2,000 R1 response tokens are filtered out, since they are considered solvable without extensive reasoning. Problems with responses between 2,000 and 4,000 tokens are downsampled to balance the dataset by response length.
• Final size: The rigorously verified dataset contains approximately 49,000 high-quality math problems for RL training.

Code-only RL dataset

• Sources and composition: Curated from modern competitive programming platforms, the dataset includes both function-calling and standard input/output (stdin/stdout) formats, covering algorithmic topics such as graph theory, data structures, number theory, and greedy algorithms.
• Reward stability filtering: Problems incompatible with standard output comparison (such as multi-solution or interactive problems requiring special judges) and those needing platform-specific templates are removed, minimizing false negative rewards.
• Testcase curation: Strong testcases are designed to cover tricky edge cases and extreme input limits, ensuring that incorrect solutions fail and eliminating false positive rewards.
• Difficulty scoring: Each problem is evaluated with DeepSeek-R1-671B using 8 rollouts, producing a difficulty score from 0 to 8. Problems where the model fails all 8 attempts (level 8) are excluded.
• Decontamination and deduplication: The authors perform benchmark decontamination and cross-platform problem deduplication using n-gram context analysis and original URL matching.
• Final size: After aggressive filtering, 8,520 problems remain as the final training set.

Both datasets are used as standalone RL training corpora for the math-only and code-only training stages, with the filtering choices directly motivated by the need to keep reward signals clean and unambiguous throughout reinforcement learning.

Method

The authors leverage the Group Relative Policy Optimization (GRPO) algorithm to train the reasoning models. This approach is selected for its simplicity, as it eliminates the need for a separate value function model. For each question-answer pair, the policy model generates a group of rollouts. A reward score is assigned to each response based on a rule-based verifier. The training utilizes a token-level policy gradient loss variant of GRPO, where the advantage for each token is uniformly assigned the normalized reward score of the entire response group.

The general objective function is defined as:

where $r_{i,t}(\theta)$ri,t​(θ) is the token-level importance weight and $\hat{A}_{i,t}$A^i,t​ is the normalized advantage. To ensure stability and prevent entropy collapse, the authors enforce a strict on-policy constraint by performing only a single gradient update after generating a group of rollouts. Consequently, the importance weight $r_{i,t}(\theta)$ri,t​(θ) equals 1, and the KL divergence term is removed by setting $\beta = 0$β=0. This simplifies the objective to a REINFORCE objective with group-normalized rewards:

The reward mechanism relies on distinct verifiers for different domains. For mathematical problems, a rule-based Python verification function utilizing sympy is employed to check symbolic equivalence of the extracted answer. For coding problems, a local sandbox verifier executes the generated code against a full set of test cases, granting a positive reward only if all tests pass within the time limit.

The training pipeline is structured into a specific curriculum to handle the computational intensity of long Chain-of-Thought reasoning. The process begins with math-only reinforcement learning, gradually extending the maximum response length from 8K to 24K tokens. This is followed by code-only reinforcement learning, which extends the length further to 32K. Finally, a second round of math-only reinforcement learning is conducted at the 32K length.

To enhance efficiency and model capability, the training incorporates three key strategies. First, the strict on-policy approach with a single update per rollout group maintains stable entropy. Second, a stage-wise length extension strategy is used, progressing from 8K to 16K, then 24K, and finally 32K tokens, rather than starting directly at higher lengths. Third, curriculum learning is applied by filtering out easier prompts (those with a pass rate greater than 6/16) during the later stages to introduce harder problems that push the model's limits.

The framework demonstrates significant cross-domain generalization. While code-only RL targets programming tasks, the initial math-only RL stage also enhances coding performance across various topics.

As illustrated in the topic-wise accuracy analysis, the math-only RL stage provides substantial gains in algorithmic and math-related coding areas, while the subsequent code-only RL stage further improves performance on topics relying heavily on implementation and data structures, such as simulation and string manipulation.

Experiment

The evaluation covers math benchmarks (AIME2024/2025, MATH500, HMMT2025, BRUMO2025) and code benchmarks (LiveCodeBench v5/v6, Codeforces ELO, EvalPlus), starting from DeepSeek-R1-Distill-Qwen-7B/14B SFT baselines. Main results show that RL training substantially boosts both math and code accuracy over the SFT models, with the 14B variant surpassing larger distilled models. Analyses reveal that extending the response length limit, training on hard prompts, and staging Math-RL before Code-RL all yield meaningful gains, while false positive or false negative rewards in code RL cause convergence to suboptimal policies. RL improves both pass@1 and pass@k across all sample counts, and it unlocks a long tail of hard problems the SFT models cannot solve.

Math-only reinforcement learning improves code reasoning, outperforming both math-only supervised fine-tuning and distilled baselines. RL-trained models consistently achieve higher scores on code benchmarks across 7B and 14B sizes, while math-only SFT can result in notably poor code performance. Math-only RL boosts LiveCodeBench scores by 6.8 points for 7B and 5.8 points for 14B over distilled SFT baselines. Math-only SFT at 14B yields much lower code performance (19.3) compared to RL (58.9), showing RL's cross-domain benefit. RL models also surpass distilled SFT on AIME math benchmarks for both sizes, indicating broad reasoning gains.

The evaluation results indicate that RL-based training significantly improves math and code reasoning over the initial SFT models, with the 14B variant achieving superior performance even compared to larger distilled models. The models also show competitive or superior results against state-of-the-art RL-based and frontier reasoning models, with consistent improvements in pass@k metrics. RL training yields large gains over SFT on math and coding tasks, including notable improvements on AIME and LiveCodeBench. AceReason-Nemotron-14B outperforms larger SFT distilled models on both math and code benchmarks. The models remain competitive with frontier reasoning models such as QwQ-32B and o3-mini on several tasks.

At the 24K RL stage, filtering training prompts by difficulty improves benchmark performance, with the hardest prompt set yielding the best AIME24 and AIME25 scores despite containing far fewer prompts than the full dataset. Easier and medium difficulty sets also outperform the full unfiltered set, indicating that curriculum filtering by pass rate is effective. The Hard prompt set (2.2K prompts) achieves the highest AIME24 and AIME25 scores, outperforming the full 49K prompt set. Filtering out prompts solved more than 6 out of 16 attempts yields a 2.6% AIME24 gain over using the full unfiltered data. Both Easy and Medium filtered sets also surpass the unfiltered full set, showing that removing well-solved prompts consistently helps.

Math-only RL significantly improves performance on both math and coding benchmarks for both 7B and 14B models. Subsequent code-only RL further boosts coding scores while causing only negligible changes in math performance, indicating that sequential RL training is effective for developing strong reasoning across domains. Math-only RL yields substantial gains on both AIME and LiveCodeBench for both model sizes. Code-only RL after math RL provides additional coding improvements with minimal math degradation. The sequential training pattern is consistent across 7B and 14B models, showing a robust interplay.

The experiments demonstrate that reinforcement learning, particularly math-only RL, substantially improves both mathematical and code reasoning across model sizes, outperforming supervised fine-tuning and distilled baselines. Filtering RL training prompts by difficulty, especially selecting the hardest prompts, yields better benchmark performance. Additionally, sequential RL training, where math RL is followed by code RL, provides further coding gains with negligible impact on math skills, indicating robust cross-domain benefits.