One-sentence Summary

Researchers from NAVER AI Lab propose On-Policy Delta Distillation (OPD^2), a new on-policy distillation method that uses the delta signal—the difference between a teacher model and its pre-instruction-tuned base model—to directly transfer reasoning capabilities, significantly outperforming conventional on-policy distillation across math, science, and code benchmarks with only short post-training.

Key Contributions

• The delta signal, defined as the difference between a reasoning-tuned teacher and its pre-tuning base model, is introduced as a new distillation reward that captures the knowledge gained from reasoning tuning.
• On-Policy Delta Distillation (OPD²) uses this delta signal as the primary training signal for on-policy distillation, focusing learning on reasoning-relevant tokens while preserving the student's original capabilities.
• Experiments across math, science, and code reasoning benchmarks (14 benchmarks) with models from 1.7B to 8B parameters (Qwen3 and Gemma4) show that OPD² consistently outperforms conventional on-policy distillation and achieves strong reasoning performance after a short post-training period.

Introduction

The authors investigate on-policy distillation (OPD) for post-training large language models, where a student generates its own outputs and receives token-level rewards from a teacher model. OPD sidesteps reinforcement learning’s reward design problems, but its standard reward — the log probability difference between teacher and student — does not explicitly isolate the reasoning capabilities the teacher gained during instruction tuning. The paper proposes On-Policy Delta Distillation (OPD²), which instead uses the difference between the reasoning-tuned teacher and its base model as the main reward signal. With centering and joint conditioning, this delta signal focuses the reward on reasoning-specific knowledge, and the authors show that OPD² consistently outperforms conventional on-policy distillation across math, science, and code benchmarks for models ranging from 1.7B to 8B parameters.

Method

The authors begin by establishing the foundation of On-Policy Distillation (OPD), a post-training method where a student model learns to mimic a teacher model on its own rollout samples. The objective is to minimize the Kullback-Leibler (KL) divergence between the student and teacher distributions. In an RL-like framework, this is achieved by defining a token-level reward for the $t$t-th token $y_t$yt​ given context $y_{<t}$y<t​ and question $x$x: $R_t = \log \pi^*(y_t \mid x, y_{<t}) - \log \pi_\theta(y_t \mid x, y_{<t})$Rt​=logπ∗(yt​∣x,y<t​)−logπθ​(yt​∣x,y<t​) where $\pi^*$π∗ is the teacher and $\pi_\theta$πθ​ is the student. The student is trained to maximize these rewards via gradient computation, effectively minimizing the KL divergence on on-policy data.

To improve upon standard OPD, the authors propose leveraging the learning trajectory of the teacher model rather than just its final output distribution. They introduce a delta signal that captures the difference between the reasoning-tuned teacher and its pre-training base model $\pi^*_{base}$πbase∗​. The delta signal is defined as: $R_t^\Delta = \log \pi^*(y_t \mid x, y_{<t}) - \log \pi^*_{base}(y_t \mid x, y_{<t})$RtΔ​=logπ∗(yt​∣x,y<t​)−logπbase∗​(yt​∣x,y<t​) This signal represents the knowledge acquired during the teacher's post-training phase, specifically targeting reasoning capabilities beyond simple next-token prediction. Refer to the framework diagram comparing the standard approach with the proposed method:

Analysis of these signals reveals that the delta formulation emphasizes reasoning-connective words (e.g., "hence", "however") while suppressing exploratory terms (e.g., "try", "verify"). Furthermore, when examining token-level rewards on incorrect reasoning traces, the delta signal demonstrates greater robustness. As shown in the figure below regarding token-level distillation signals on simple reasoning examples:

The delta signal more consistently suppresses tokens associated with incorrect reasoning steps compared to standard OPD, which can produce positive signals for wrong reasoning due to magnitude differences between teacher and student.

However, using $R^\Delta$RΔ directly poses a convergence issue because it ignores the student's current distribution $\pi_\theta$πθ​, potentially leading to training instability. To address this, the authors design On-Policy Delta Distillation (OPD$^2$2). They first formulate bias-subtracted advantage functions to ensure training stability. The advantage for the delta signal is computed as: $A_t^\Delta = R_t^\Delta - \mathbb{E}_{\tilde{y}_t \sim \pi_\theta(\cdot | x, y_{<t})} \left[ \log \pi^*(\tilde{y}_t \mid x, y_{<t}) - \log \pi^*_{base}(\tilde{y}_t \mid x, y_{<t}) \right]$AtΔ​=RtΔ​−Ey~​t​∼πθ​(⋅∣x,y<t​)​[logπ∗(y~​t​∣x,y<t​)−logπbase∗​(y~​t​∣x,y<t​)] where the expectation is approximated over the top-k tokens of the student policy to save GPU memory.

To ensure the student converges properly to the teacher, OPD$^2$2 introduces a stop condition that aligns the delta advantage with the original OPD advantage $A_t^{OPD}$AtOPD​. The final advantage function $A_t^{D^2}$AtD2​ is defined as: $A_t^{D^2} = \begin{cases} A_t^\Delta & \text{if } A_t^\Delta A_t^{OPD} > 0 \\ 0 & \text{otherwise} \end{cases}$AtD2​={AtΔ​0​if AtΔ​AtOPD​>0otherwise​ This condition restricts updates to directions where the delta signal and the original distillation signal agree, preventing the student from diverging from the teacher's distribution. When the student matches the teacher, the advantage is turned off. The training gradient for OPD$^2$2 is then computed as: $\nabla_\theta J_{OPD^2}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot | x)} \left[ \sum_{t=1}^T A_t^{D^2} \nabla_\theta \log \pi_\theta(y_t \mid x, y_{<t}) \right]$∇θ​JOPD2​(θ)=Ex∼D,y∼πθ​(⋅∣x)​[∑t=1T​AtD2​∇θ​logπθ​(yt​∣x,y<t​)] This formulation allows the model to benefit from the reasoning-focused delta signal while maintaining stable convergence properties inherent to standard on-policy distillation.

Experiment

The experiments evaluate OPD^2 against OPD and ExOPD on small reasoning models (Qwen3 and Gemma4) across multi-domain math, science, and code tasks, testing both non-thinking and thinking modes. OPD^2 consistently achieves the best performance, delivering large gains in non-thinking mode and improving or preserving strong thinking-mode capabilities where other methods often degrade. The delta signal is the primary driver of improvement, and training dynamics show OPD^2 maintains a persistent advantage throughout training. Its computational overhead is modest and generalizes effectively across model families.

The delta signal shifts distillation emphasis toward reasoning-connective words like 'hence' and 'however' while suppressing exploratory terms such as 'see' and 'verify'. These enhanced tokens appear with high occurrence rates in the top-5% signal strength across math, code, and science domains, indicating a consistent bias in the distillation process. The delta signal strongly enhances reasoning-connective words such as 'hence' and 'however' across all three domains, with top-5% strength occurrence rates ranging from 39% to 74%. Exploratory and verification-related words like 'see', 'try', and 'verify' are suppressed, steering the model away from tentative reasoning and toward more decisive outputs.

On-policy distillation methods substantially improve non-thinking math performance for Qwen3 models, with OPD² providing the largest gains. The 1.7B model’s average score nearly doubles, and the 4B OPD² model surpasses the 8B model trained with competing methods, showing that the approach can match or exceed the benefit of scaling model size. OPD² nearly doubles the average math score of Qwen3-1.7B (from 34.8 to 54.6). A 4B model trained with OPD² reaches 70.3, outperforming the 8B model with ExOPD (67.8) and demonstrating a size-equivalent gain.

On-policy distillation methods substantially improve non-thinking performance of Qwen3 models on Code and Science. OPD2 delivers the largest gains across all model sizes, and a 4B model trained with OPD2 already surpasses an 8B model trained with OPD or ExOPD, demonstrating that the method can provide benefits comparable to scaling model size. For Qwen3-1.7B, OPD2 boosts the Code average from 10.5 to 29.4 and the Science average from 30.8 to 38.8, outperforming OPD and ExOPD by margins of 3.6 and 3.2 points respectively. The 4B OPD2 model exceeds the 8B model trained with OPD or ExOPD, indicating that the proposed method can match or exceed gains from larger model sizes.

In thinking-mode math evaluation, OPD² improves the average score of Qwen3-1.7B from 59.2 to 62.7, while OPD and ExOPD degrade performance to 57.1 and 58.4 respectively. The gains are concentrated on contest-level benchmarks like AIME24 and AIME25, with smaller improvements on MATH500 and RGMath. OPD² is the only method that raises the Qwen3-1.7B math average in thinking mode; OPD and ExOPD both fall below the baseline. The largest absolute improvements with OPD² occur on AIME25 (+9.8 points) and AIME24 (+1.5 points), while HMMT25 and AMC23 also see notable gains.

In thinking mode, on-policy distillation methods show divergent effects on Qwen3 models. OPD² substantially improves the 1.7B model across Code and Science, while standard OPD degrades the 4B model's performance. Gains are especially pronounced on challenging coding benchmarks like CodeContests. OPD² boosts the 1.7B model's average Code score from 29.3 to 40.4 and Science score from 40.1 to 43.4, surpassing both OPD and ExOPD. On CodeContests, OPD² nearly doubles the 1.7B model's score from 19.8 to 37.8, representing the largest single-benchmark improvement. Standard OPD reduces the 4B model's average Code score from 48.7 to 46.4 and Science score from 51.3 to 48.4, indicating that thinking-mode performance is sensitive to the distillation approach.

The experiments evaluate on-policy distillation methods (OPD, ExOPD, OPD²) applied to Qwen3 models across math, code, and science benchmarks in both non-thinking and thinking modes. A token-level analysis of the distillation signal reveals a consistent bias toward amplifying reasoning-connective words and suppressing exploratory terms, guiding the model toward more decisive outputs. Performance comparisons show that OPD² consistently yields the largest gains, often enabling a smaller model to outperform a larger model trained with competing methods, while thinking-mode results are more sensitive to the distillation approach, with standard OPD sometimes degrading performance.