One-sentence Summary

Researchers from KAIST AI, Microsoft, University of Toronto, Mila, Université de Montréal, and CIFAR AI Chair propose On-Policy Reverse Distillation (OPRD), which amplifies verifier-supported student policy gradients along the teacher's policy shift direction on student rollouts, preserving optimization stationary points while accelerating weak-to-strong generalization beyond the teacher, achieving higher performance with fewer updates in successive and multi-teacher distillation and remaining closer to verifier-based RL than to weak teachers.

Key Contributions

• Introduces On-Policy Reverse Distillation (OPRD), which transfers a weak teacher’s post-training policy shift by amplifying the component of a stronger student’s verifier-driven policy gradient aligned with that shift, without making the teacher policy an optimization target.
• Preserves the stationary points of the student’s policy objective in logit space by rescaling rather than replacing the gradient, accelerating learning beyond the teacher while avoiding competing objectives found in teacher-matching distillation losses.
• Experiments across successive model transfer, multi-teacher distillation, and strong-to-weak distillation show OPRD reaches teacher-level or higher performance in fewer student updates than existing RL and distillation baselines, with response-style analysis indicating students remain closer to verifier-only RL training than to the weak teacher.

Introduction

Knowledge distillation transfers capabilities from a teacher model to a student, but conventional methods often create a mismatch between training prefixes and those the student encounters at inference. On-policy distillation (OPD) addresses this by training on student-generated responses, yet it still optimizes the student to match the teacher's policy exactly. This becomes problematic in weak-to-strong settings, where a weaker model supervises a stronger one. Standard OPD transfers the teacher's full behavior, including its capacity limitations and preferences, which can suppress the stronger student's potential and cap its performance. Prior attempts to isolate the teacher's policy change or add reinforcement learning still treat the weak policy as an optimization target, limiting the student's ability to surpass its supervisor.

The authors introduce On-Policy Reverse Distillation (OPRD), a method that leverages the policy shift a weak teacher learns during post-training to accelerate a stronger student's own optimization. Instead of matching the teacher, OPRD computes the student's verifier-driven policy gradient on its own rollouts and projects it onto the direction of the teacher's policy shift, amplifying the aligned component while leaving orthogonal parts unchanged. This rescaling preserves the student's optimization stationary points while adding a nonnegative alignment gain, allowing the student to converge faster and move beyond the teacher's capabilities. Evaluations on mathematical and logical reasoning show OPRD reaches teacher-level performance with 33 to 67 percent fewer updates than GRPO and outperforms all four specialized teachers in a multi-teacher consolidation setting, demonstrating that weak models can accelerate stronger students without limiting their final performance.

Method

Method

Preliminary

Reinforcement Learning with Verifiable Rewards (RLVR). RLVR optimizes a language-model policy using rewards computed by programmatic verifiers, such as exact-answer checks or code execution, and has become central to reasoning post-training. For $x \sim \mathcal{D}$x∼D, the student samples $y \sim \pi_{\boldsymbol{\theta}}(\cdot \mid x)$y∼πθ​(⋅∣x) and visits prefixes $s_t = (x, y_{<t})$st​=(x,y<t​). Let $A_t$At​ denote the advantage assigned to token $y_t$yt​ and $\mathbf{z}_t$zt​ the corresponding next-token logits at prefix $s_t$st​. The token-level policy gradient is

OPRD later rescales this gradient while preserving the RLVR objective, so the student's attainable performance is determined by the verifier objective and its own policy class rather than being bounded by the teacher's capacity.

On-Policy Distillation (OPD). OPD reduces the training-inference distribution mismatch by sampling responses from the student and querying the teacher at each visited prefix, thereby providing dense token-level supervision over the student's inference-time state distribution. A common reverse-KL formulation is

Equivalently, OPD can be implemented as token-level policy optimization on student-sampled tokens using the teacher-to-student log-probability ratio as the advantage, with negligible empirical differences from direct reverse-KL optimization. OPD is increasingly used in frontier-model post-training for reasoning and capability integration across domains. Recent methods combine teacher matching with reinforcement learning to pair dense teacher supervision with outcome-based optimization. Even in these hybrid methods, however, teacher matching remains a separate objective, leaving the teacher policy as a direct optimization target.

Weak-to-Strong Generalization. Weak-to-strong generalization studies whether a more capable model can learn from weaker supervisors, such as smaller models or imperfect human feedback, and ultimately outperform them. Prior work has used weak labels, preferences, and fixed reasoning trajectories to supervise stronger students. Refinement methods help the student exploit its own representations and greater capacity, but often recover only part of the gap to strong supervision. OPD instead provides the full next-token distribution $\bar{\pi}_{T}(\cdot \mid s_t)$πˉT​(⋅∣st​) at each student-visited prefix, where $\bar{\pi}_{T}$πˉT​ denotes either the weak teacher or a target policy derived from it. Under realizability, the resulting KL objective has the pointwise minimizer

Alternative teacher-derived targets only change which policy the student matches, while adding reinforcement learning yields a compromise between teacher matching and reward maximization. In both cases, the student remains directly optimized toward a policy defined by the weak teacher. OPRD instead extracts the policy change learned during weak-model post-training and uses it only to rescale the stronger student's own policy gradient.

On-Policy Reverse Distillation

Overview. OPRD transfers the policy change learned during teacher post-training rather than matching the teacher's final policy. At each student-visited prefix, it extracts the local direction of this change relative to the teacher's reference policy and uses its alignment with the verifier-driven student gradient to rescale only the gradient component along that direction. Because the teacher signal only rescales the student's own gradient, it can accelerate verifier-supported optimization without defining an independent optimization target. Positive-alignment scaling is active from the outset to amplify updates supported by both the verifier and the teacher, whereas negative-alignment scaling is gradually increased to reinforce verifier-supported departures beyond the weak teacher.

Teacher Policy Shift. The teacher's final policy reflects the change acquired during RL post-training, preferences inherited from its reference policy, and behavior constrained by the weak model's limited capacity. Directly matching it would therefore make all of these part of the student's distillation target. Let $\pi_{T}$πT​ denote the frozen RL-trained teacher and $\pi_{T}^{\mathrm{ref}}$πTref​ its frozen pre-RL reference policy, and let $\mathbf{z}_{T}(s_t)$zT​(st​) and $\mathbf{z}_{T}^{\mathrm{ref}}(s_t)$zTref​(st​) denote their next-token logit vectors at a student-visited prefix $s_t$st​. To extract the RL-induced policy delta, the authors mean-center the difference between the teacher and reference logits, removing a common offset that does not affect relative token preferences. With $\mathcal{C}(\mathbf{v}) := \mathbf{v} - \frac{1}{|\mathcal{V}|} (\mathbf{1}^{\top} \mathbf{v}) \mathbf{1}$C(v):=v−∣V∣1​(1⊤v)1, they define

Intuitively, $\boldsymbol{\Delta}_t$Δt​ captures the change in the teacher's relative next-token preferences induced by post-training, and the corresponding uncentered log-policy ratio admits an implicit-reward interpretation under KL-regularized policy optimization. However, because this shift is learned within the weak teacher's policy class, it need not improve verifier reward for the stronger student. OPRD therefore uses only its unit direction $\mathbf{d}_t := \boldsymbol{\Delta}_t / \|\boldsymbol{\Delta}_t\|_2$dt​:=Δt​/∥Δt​∥2​ for gradient scaling rather than optimizing toward the shift itself. The student gradient determines whether the resulting correction follows or opposes this direction, independently of its raw magnitude.

Gradient Scaling along the Teacher Direction. At token $t$t, OPRD decomposes the student's policy gradient $\mathbf{g}_t$gt​ (Eq. 2.1) relative to the teacher direction $\mathbf{d}_t$dt​. Let $u_t := \mathbf{d}_t^{\top} \mathbf{g}_t$ut​:=dt⊤​gt​ denote their alignment coefficient, and define the projected and orthogonal components as $\mathrm{Proj}_{\mathbf{d}_t}(\mathbf{g}_t) := u_t \mathbf{d}_t$Projdt​​(gt​):=ut​dt​ and $\mathbf{g}_t^{\perp} := \mathbf{g}_t - \mathrm{Proj}_{\mathbf{d}_t}(\mathbf{g}_t)$gt⊥​:=gt​−Projdt​​(gt​), respectively. At optimization step $k$k, OPRD uses a nonnegative scale $\lambda_t$λt​:

Essentially, OPRD decomposes the student's policy gradient into its projection onto the teacher-informed direction and an orthogonal component, amplifying only the projected component by $1 + \lambda_t$1+λt​, while leaving the orthogonal component unchanged. The authors backpropagate $\widetilde{\mathbf{g}}_t$g​t​ in place of $\mathbf{g}_t$gt​ and use the resulting parameter gradients to update the student.

Learning Beyond the Weak Teacher. Direct teacher matching makes the weak teacher's policy a target of student optimization, even when moving beyond the teacher would yield higher verifier reward. OPRD instead uses the weak teacher only to rescale the student's own policy gradient. At token $t$t, this scaling can be written as the linear map $\widetilde{\mathbf{g}}_t = (\mathbf{I} + \lambda_t \mathbf{d}_t \mathbf{d}_t^{\top}) \mathbf{g}_t$g​t​=(I+λt​dt​dt⊤​)gt​. For $\lambda_t \geq 0$λt​≥0, the map is invertible and satisfies:

Since Eq. 2.6 holds at every token, the scaling preserves the stationary points of the verifier objective for a fixed response. Eq. 2.7 shows that the transformed gradient $\widetilde{\mathbf{g}}_t$g​t​ retains the first-order progress of $\mathbf{g}_t$gt​ and adds the nonnegative alignment gain $\lambda_t u_t^2$λt​ut2​, so greater alignment magnitude $|u_t|$∣ut​∣ yields greater first-order progress. For a fixed response, these token-level gains sum into a nonnegative term in the guaranteed one-step ascent. OPRD can therefore accelerate the student's optimization without introducing a teacher-defined target.

Asymmetric Alignment Scaling. The sign of $u_t$ut​ indicates whether $\mathbf{g}_t$gt​ aligns with or opposes $\mathbf{d}_t$dt​, so scaling reinforces teacher-following updates when $u_t \geq 0$ut​≥0 and verifier-supported departures when $u_t < 0$ut​<0. However, both signals may include reward-irrelevant bias (e.g., $\mathbf{g}_t = \mathbf{g}_t^{\star} + \epsilon_t$gt​=gt⋆​+ϵt​, with $\mathbf{g}_t^{\star}$gt⋆​ denoting the reward-improving signal and $\epsilon_t$ϵt​ aggregating structured bias components). Scaling only one sign can then systematically magnify this bias term, causing it to accumulate over training. Because negative-alignment updates are less reliable on initially weak student rollouts, the authors activate the positive branch immediately and gradually ramp up the negative branch. At optimization step $k$k, they set the token-wise scaling coefficient as

where $\lambda$λ is the scaling strength and $K_{\text{warm}}$Kwarm​ is the warm-up horizon. Since $\lambda_t \geq 0$λt​≥0, Eqs. 2.6 and 2.7 continue to hold under this schedule. The negative-branch warm-up gradually mitigates the initial one-sided amplification caused by positive-only scaling. This preserves immediate teacher-aligned transfer while progressively strengthening verifier-supported departures from the weak teacher.

Discussion of Key Challenges

Vanishing Policy Gradients Limit OPRD's Teacher-Guided Correction. OPRD requires a nonzero verifier-driven policy gradient. In an additional strong-to-weak experiment pairing a Qwen3-8B-Base teacher with a Qwen3-1.7B-Base student, most Knights & Knaves responses are invalid, so most rollout groups receive identical rewards; the resulting group-relative advantages and their contributions to $\mathbf{g}_t$gt​ therefore vanish. For these groups, the projection onto $\mathbf{d}_t$dt​ also vanishes, leaving no component for OPRD to amplify and hence no teacher-guided correction. OPRD still accelerates learning relative to GRPO and KDRL, although all three remain below 20% Pass@1, whereas OPD reaches 40.5% using dense policy-matching targets that do not depend on verifier rewards. This challenge arises from the student's initial rollout distribution rather than the absence of a useful teacher signal. Such an extreme regime is less likely in the primary weak-to-strong setting, where the student has greater capacity than the teacher, but may still arise on sufficiently difficult tasks. A short task-specific SFT or distillation warm-up could bootstrap valid on-policy behavior before switching to OPRD.

Reference Policy Selection Can Prevent Length Bias from Distorting Teacher Guidance. Both $\mathbf{g}_t$gt​ and $\mathbf{d}_t$dt​ can contain reward-irrelevant components such as $\epsilon_t$ϵt​, which the projection-and-amplification step can magnify. Response length is one example: when it correlates with verifier reward, both signals can encode a preference for longer or shorter responses, even if changing length does not itself improve reasoning quality. The step-0 reference $\pi_{T}^{\mathrm{ref}}$πTref​ produces substantially longer responses than the step-105 teacher $\pi_{T}$πT​ on Color Cube. The resulting shift $\boldsymbol{\Delta}_t$Δt​ therefore contains a strong shortening component. With this reference, OPRD rapidly shortens its responses and achieves strong early gains. It nevertheless plateaus at 52.5% Pass@1, below GRPO and KDRL, suggesting that the teacher-guided correction overemphasizes shortening at the expense of task-relevant reasoning. A simple mitigation is to move the reference to step 30, after the teacher's initial length collapse. This excludes some of the teacher's early gains from $\boldsymbol{\Delta}_t$Δt​ but substantially narrows the reference-teacher length gap and weakens the associated bias. OPRD then avoids the plateau and jumps to 89.5%, discovering a more effective reasoning strategy. The same pattern appears on Binary Matrix, where this reference policy adjustment is likewise effective.

Experiment

OPRD is evaluated on mathematical and logical reasoning tasks across weak-to-strong transfer, multi-teacher consolidation, and strong-to-weak distillation, using Qwen3 models and baselines like GRPO, OPD, and KDRL. In weak-to-strong settings, OPRD accelerates learning beyond the weak teacher and outperforms all baselines, while in multi-teacher settings it consolidates specialists without cross-task tradeoffs. Strong-to-weak results confirm the method works regardless of teacher-student capacity ordering. Analyses show that teacher guidance is most effective when derived from a well-trained weak policy delta, with sufficient directional amplification, and that OPRD lets the student surpass the teacher’s reasoning paths and style, though challenges arise with vanishing policy gradients and reference-induced length bias.

Successive model transfer from a 4B teacher to an 8B student shows that OPRD consistently outperforms both GRPO and OPD on mathematics and reasoning tasks, with average gains of 7.92 points on mathematics and 10.80 points on Reasoning Gym over the strongest baseline. The method accelerates early learning like OPD but continues beyond the teacher's performance, reaching GRPO's end-of-training results much earlier. OPRD also avoids the plateau seen in OPD by not forcing the student to match the teacher's policy directly. OPRD achieves higher average scores than GRPO, OPD, and KDRL on both mathematics and Reasoning Gym benchmarks. OPRD matches OPD's initial acceleration but surpasses the weak teacher and reaches GRPO's end-of-training performance earlier. OPRD stays stylistically closer to the stronger student, unlike OPD which remains closer to the teacher across all response-style categories.

Multi-teacher distillation consolidates four task-specific Qwen3-4B-Base teachers into a single Qwen3-8B-Base student, where OPRD achieves the highest average Pass@1, surpassing both Mix-RL and the specialist average. In strong-to-weak distillation, OPRD also outperforms standard OPD on both AIME'24 and Knights & Knaves, showing effectiveness regardless of teacher-student capacity ordering. OPRD improves the average score by 11.09 points over Mix-RL and 14.12 points over the specialist average in multi-teacher distillation. OPRD exceeds the corresponding specialist on all four tasks, indicating joint improvement without cross-task tradeoffs. In strong-to-weak distillation, OPRD outperforms OPD by 3.79 points on AIME'24 and 29.20 points on Knights & Knaves.

OPRD consistently outperforms weak-to-strong baselines that rely on off-policy teacher trajectories or direct optimization of the weak policy delta, by keeping training on-policy and using the teacher shift only to rescale the verifier gradient. Better-trained weak teacher checkpoints accelerate student learning, but the teacher's absolute performance does not cap the student's final performance. The guidance direction from the weak policy delta is most effective, and its influence diminishes as training progresses. OPRD surpasses baselines like W2SR-P, S2L-PO, Direct-OPD, and W2S-OPD on average across math and reasoning tasks. Later, better-performing teacher checkpoints lead to faster student learning under OPRD, but even a weak teacher checkpoint can enable the student to far exceed the teacher's own score. The weak policy delta as a guidance direction yields faster and more sustained gains than alternative directions like teacher-matching or self-distillation gradients. Increasing the guidance strength lambda improves final performance and early learning speed, with gains saturating beyond a moderate value. Early in training, the guidance direction aligns strongly with the student's policy gradient, but this alignment fades toward orthogonality as training proceeds.

OPRD consistently outperforms GRPO, OPD, and other baselines across mathematics and reasoning tasks in successive model transfer, multi-teacher distillation, and strong-to-weak distillation settings, with notable gains in average scores and earlier convergence to strong baselines' end-of-training performance. It avoids the plateau seen in OPD by not directly matching the teacher's policy, and it maintains stylistic closeness to the stronger student. In multi-teacher distillation, OPRD surpasses both Mix-RL and the specialist average on all tasks without cross-task tradeoffs, while in strong-to-weak distillation it also beats standard OPD. OPRD keeps training on-policy, using the teacher shift only to rescale the verifier gradient, which allows students to exceed weak teacher performance and benefits from better teacher checkpoints, with guidance strength saturating at moderate values.