One-sentence Summary

Researchers from the University of Science and Technology of China, Peking University, and other institutions demonstrate that on-policy distillation transfers a teacher's reasoning behavior rather than specific answers, with same-origin teacher-student pairs enabling broad cross-domain generalization while cross-origin pairs mainly fit the training distribution, and that multi-teacher distillation produces a mixture-dependent seesaw effect among capabilities, thereby clarifying when OPD generalizes.

Key Contributions

• A controlled study systematically varies generalization factors in on-policy distillation (OPD) across math, code, science, and instruction-following domains, isolating in-domain shifts in problem difficulty, language, and reasoning horizon, cross-domain transfer, and multi-teacher settings.
• OPD transfers a teacher’s reasoning behavior rather than its answers to specific problems: training problem difficulty barely matters, and even problems the teacher never solves are useful.
• Generalization depends strongly on the teacher-student origin relationship: same-origin pairs achieve broad transfer across languages, reasoning horizons, and even other domains, while cross-origin pairs mostly fit the training distribution; in multi-teacher OPD, routing cannot confine each teacher’s influence, yielding a mixture-dependent seesaw among their capabilities.

Introduction

The authors investigate on-policy distillation (OPD), a technique that transfers capabilities from strong teacher models to smaller student models by supervising the student on its own generated trajectories. While prior work demonstrates OPD improves performance on target tasks, it fails to distinguish whether the student merely fits the training distribution or acquires broader reasoning patterns that generalize to new settings. This gap is critical for understanding how OPD works and for designing effective multi-teacher integration strategies. The authors conduct a controlled study that systematically varies one generalization factor at a time, revealing that OPD transfers the teacher’s reasoning behavior rather than specific problem solutions, that same-origin teacher-student pairs transfer broadly across languages, horizons, and even domains, and that this broad transfer becomes a double-edged sword in multi-teacher OPD, where prompt routing cannot isolate each teacher’s influence, leading to a mixture-dependent seesaw among capabilities.

Experiment

This work evaluates on-policy distillation with single and multiple teachers across math, code, science, and instruction following. It finds that training-problem difficulty has little effect, while dynamically discarding problems the student already solves gives small consistent gains. Same-origin teacher-student pairs transfer capabilities across language, reasoning horizon, and domains much more effectively than cross-origin pairs, which fit mainly the trained distribution. Because a teacher's influence reaches beyond its assigned domain, multi-teacher routing cannot isolate domain experts; changing teacher mixture ratios creates a seesaw effect, and same-origin teachers exert a stronger pull by aligning the student policy as a whole.

Dynamically discarding only the problems the student already solves (pass-rate in [0,1)) yields a small but consistent average improvement across six math benchmarks, while filtering to only unsolved or only solved problems does not help. This suggests that removing already-mastered problems prevents redundant realignment without harming generalization. For Polaris-7B → DS-distill-1.5B, discarding fully solved problems raised average accuracy from 41.4% to 42.0% (+0.6 pp). Restricting to only unsolved (pass-rate=0) or only solved (pass-rate=1) problems resulted in average accuracy equal to or below the no-filtering baseline. The benefit of discarding solved problems was consistent across both teacher-student pairs, with gains on most individual benchmarks.

Varying the prompt mixture ratio between a math-specialist teacher and a science/IF-specialist teacher in multi-teacher online policy distillation shifts the student's math accuracy toward the teacher that receives more prompts. When the math teacher JustRL-1.5B dominates, average math accuracy rises slightly; when the science/IF teacher Nemotron-1.5B dominates, math accuracy falls. This demonstrates the cross-domain seesaw effect, where a teacher's influence pulls performance in domains beyond its assigned teaching area. Increasing the share of prompts given to the math teacher improves the student's math accuracy, while increasing the science/IF teacher's share degrades it. The highest average math accuracy (27.1%) is achieved when the math teacher receives the largest prompt share (J/N=25/8), and the lowest (25.1%) when the science/IF teacher dominates (J/N=2/25). Relative to an equal mixture, a math-teacher-heavy ratio boosts math accuracy by 0.8 percentage points, whereas a science-teacher-heavy ratio reduces it by 1.2 points.

Filtering out problems the student already solves during online policy distillation yields a small but consistent accuracy improvement by preventing redundant realignment, while restricting to only unsolved or only solved problems does not help. Varying the prompt mixture between a math-specialist teacher and a science/instruction-following teacher demonstrates a cross-domain seesaw effect, where the student's math performance shifts toward the teacher that receives more prompts, even beyond that teacher's assigned domain. These results validate that selective problem filtering and teacher prompt allocation are effective levers for guiding student performance in multi-teacher distillation.