One-sentence Summary

Researchers from Nanyang Technological University and Skywork AI identify that post-trained generators underperform when coupled with general reward models due to distribution mismatch, and they propose a unified reinforcement learning self-verification framework that enables LLMs to assess their own answers, improving post-training performance and enabling test-time scaling without external verifiers, as demonstrated on $\text{Qwen2.5-Math 7B}$Qwen2.5-Math 7B and $\text{DeepSeek-R1-Distill-Qwen-1.5B}$DeepSeek-R1-Distill-Qwen-1.5B across multiple mathematical reasoning benchmarks.

Key Contributions

• The paper introduces a self-verification framework that unifies answer generation and verification within a single reinforcement learning (RL) process, marking the first unified RL approach to jointly train LLMs for both problem-solving and verification.

• The framework removes reliance on external reward models by training models to generatively assess the correctness of their own solutions, directly addressing the distribution shift between post-trained generators and general reward models that limits test-time scaling gains.

• Experiments across multiple mathematical reasoning benchmarks, using Qwen2.5-Math 7B and DeepSeek-R1-Distill-Qwen-1.5B, show that the self-verification models improve post-training performance and enable effective test-time scaling through self-verification.

Introduction

Large language models have shown strong performance on verifiable reasoning tasks such as mathematics and logic puzzles, where ground-truth answers are available. Researchers have pursued two complementary scaling strategies: post-training through reinforcement learning (RL) that rewards correct answers, and test-time scaling that uses additional compute, ensembles, or reward models (RMs) as verifiers to improve accuracy during inference. However, the authors observe that combining these two approaches yields limited synergy. Post-trained models tend to produce a narrow generation distribution because they are tuned on specific training data, and external verifiers trained on regular data do not recognize this shift. As a result, common test-time scaling methods such as best-of-N and beam search with external RMs offer only marginal gains over simple self-consistency via majority voting.

To address this, the authors propose a self-verification framework that unifies problem-solving and verification within a single model. They train the LLM not only to generate answers but also to verify the correctness of its own solutions, using reinforcement learning with reward signals derived from ground-truth answer correctness. To stabilize the joint training of generation and verification, they introduce an online policy-aligned buffer that keeps the verifier input distribution aligned with the latest model outputs, and a dynamic verification reward that uses multiple GRPO rollouts to adjust reward signals automatically. At inference time, the same post-trained model handles both generation and verification, and the authors apply weighted answer aggregation based on verification scores. This design avoids reliance on external RMs and integrates easily into existing inference engines. Experiments on mathematical reasoning benchmarks show that adding self-verification to post-training improves model performance and enables effective test-time scaling as generation budgets increase.

Method

Method: Self-Verification Framework

To address the limited generalization of conventional test-time scaling methods for post-trained models, the authors introduce a self-verification framework that unifies answer generation and verification within a single reinforcement learning process. The framework enables efficient multi-task RL during training and robust test-time scaling at inference, all while requiring only a single LLM deployment.

RL with Self-Verification

The authors adopt the GRPO algorithm to train large language models (LLMs) on math reasoning tasks, using a rule-based correctness reward $r_c(x, y)$rc​(x,y) that reflects whether a generated solution $y$y is correct for problem $x$x. To extend the model's verification capability, an additional verification reward is introduced. Since ground-truth correctness is available for each generated solution, the model's predicted verification outcome can be directly compared with the ground truth:

Here, the token $y_v$yv​ is selected from Yes or No, representing the model's judgment on the verification task. Thus, $r_v$rv​ serves as a direct training signal for the verifier. Notably, the verification task depends not only on the original math problem but also on the model's generated response, so the authors use an online data buffer to store real-time model responses during training.

Policy-Aligned Buffer. The authors leverage solutions generated during math reasoning training as data to train the verifier. However, retaining all historically generated solutions would force the verifier to learn from potentially meaningless or unrefined solutions produced during early training stages. To address this issue, the verifier is designed to focus primarily on solutions representative of the current model's capabilities. Therefore, the authors implement a policy-aligned buffer $B$B that stores only the most recent solutions, mitigating the risks associated with off-policy data. Specifically, only solutions generated within the last $T_b$Tb​ training steps are retained, ensuring the verification dataset is aligned with the evolving policy.

Dynamic Verification Reward. When training the verifier through RL, a simple binary reward as defined in the equation above proves ineffective. As the model improves, correct solutions quickly dominate the verification dataset, creating an imbalanced input. To keep the verifier focused on difficult verification problems, the authors introduce a dynamic verification reward that identifies challenging cases and adjusts rewards according to their difficulty. Leveraging GRPO, a group of generated responses $y_1, \ldots, y_G$y1​,…,yG​ is obtained for each problem $x$x. Denoting $G_c$Gc​ and $G_i$Gi​ as the number of correct and incorrect generations in the group, the dynamic verification reward $\hat{r}_v$r^v​ is defined as:

This reward design provides higher incentives when the verifier successfully identifies a correct solution to a difficult problem or an incorrect solution for an easy problem. The expected reward for correct verification is maintained at 1, ensuring it remains comparable in scale to the problem-solving reward $r_c$rc​.

The algorithm integrates self-verification into GRPO training by maintaining an online buffer of recent model generations. For each training step, the original dataset and the buffer are combined to sample problems. When processing dataset problems, the model generates multiple solutions, computes their correctness rewards, and adds them to the buffer. For problems from the buffer, dynamic verification rewards are computed based on the difficulty of verification. The final reward combines both correctness and verification components, ensuring the model learns to generate correct solutions while simultaneously developing verification capabilities.

Self-Verification for Test-time Scaling

At inference time, the post-trained model possesses a strong built-in verifier, allowing it to scale performance by verifying its own solutions. Given a generation budget of $N$N, the model first samples $N$N candidate responses $y_1, \ldots, y_N$y1​,…,yN​. For each response, a verification score $s(x, y_i)$s(x,yi​) is computed as the likelihood of the Yes token given the problem, the solution, and the verification prompt template. The authors then extract the set of unique answers $\mathcal{A} = \{\mathcal{E}(y_1), \ldots, \mathcal{E}(y_N)\}$A={E(y1​),…,E(yN​)}, where $\mathcal{E}$E extracts the final answer from a response. The verification scores are aggregated for each candidate answer to determine the final prediction $\hat{a}$a^:

where $\alpha$α is a hyperparameter that controls the scale of verification scores. When $\alpha = 0$α=0, this reduces to standard majority voting; as $\alpha \to \infty$α→∞, the answer with the highest verification score dominates. By choosing a moderate value for $\alpha$α, the authors achieve a balance between consensus and confidence, leading to more robust and reliable predictions. A key advantage of this approach is its efficiency: it requires only a single LLM deployment at inference time, eliminating the need for external verifier models. This makes the framework highly practical and easily compatible with modern LLM inference engines such as vLLM and SGLang.

Experiment

Experiments on the self-verification framework used two models, Qwen2.5-Math-7B and DeepSeek-R1-Distill-Qwen-1.5B, trained with reinforcement learning on mathematical reasoning data and evaluated on benchmarks like MATH500, AIME, AMC, and OlympiadBench. During post-training, the self-verification models outperformed standard GRPO baselines in greedy-decoding accuracy, and their verification ability surpassed similarly sized open-source models while matching or exceeding GPT-4o on their own responses despite fewer parameters. For test-time scaling, self-verification achieved the best performance over self-consistency, best-of-N, beam search, and DVTS, with external verifier methods underperforming due to distribution shift. The framework proved efficient, using 24-35% fewer tokens for verification than solving, and scaled faster in inference time than best-of-N and process-based methods, though slightly slower than majority voting.

Self-verification post-training improves math reasoning performance across multiple benchmarks. The proposed models outperform their base counterparts and standard GRPO models, indicating that integrating verification enhances problem-solving abilities. Self-verification models consistently outperform their base models by a significant margin. Both 7B and 1.5B self-verification models achieve strong results on AIME and AMC benchmarks. Incorporating self-verification into reinforcement learning yields higher scores than standard GRPO, suggesting a synergistic effect.

The self-verification model outperforms all open-source baselines on solution verification, with large gains over its base model and Llama. It also surpasses GPT-4o in accuracy and F1, though it trails Claude-3.7-Sonnet and DeepSeek-v3. Self-Verification-Qwen-7B improves accuracy by 14 points over its base model. The model exceeds GPT-4o on both accuracy and F1, but remains behind Claude-3.7-Sonnet and DeepSeek-v3.

The self-verification model outperforms similarly sized open-source baselines by a large margin on verifying AIME24 solutions, achieving a higher F1 score than GPT-4o while maintaining competitive accuracy. Its performance is on par with leading proprietary systems, indicating that self-verification enhances both problem-solving and verification capabilities. Self-Verification-R1-1.5B attains the best F1 score among open-source models and exceeds GPT-4o's F1. The model shows a substantial accuracy improvement over its base model, demonstrating the benefit of self-verification in post-training. Its verification performance is competitive with proprietary models like Claude-3.7-Sonnet and DeepSeek-v3.

The proposed self-verification method achieves the highest average scores across most math reasoning benchmarks compared to other test-time scaling approaches, including self-consistency, best-of-N, beam search, and DVTS. Interestingly, simple self-consistency with majority voting is competitive, while methods relying on external verifiers underperform, likely due to distribution shift and step-wise generation issues. Self-verification consistently outperforms all other test-time scaling methods on most benchmarks. Simple self-consistency via majority voting is surprisingly competitive, often beating external verifier based methods. External verifier methods like best-of-N, beam search, and DVTS perform worse, with process-based approaches amplifying distribution shift.

Self-verification post-training improves both math reasoning and solution verification, with models consistently outperforming their base counterparts and standard GRPO across benchmarks. In verification tasks, the approach surpasses open-source baselines and GPT-4o in accuracy and F1, though it remains behind Claude-3.7-Sonnet and DeepSeek-v3. Compared to other test-time scaling strategies, self-verification achieves the highest average scores on most math benchmarks, while simple self-consistency with majority voting remains competitive and external verifier based methods underperform due to distribution shift.