One-sentence Summary

Dual-Anchored Policy Distillation (DAPD), proposed by researchers from Shanghai Jiao Tong University, Shanghai Artificial Intelligence Laboratory, and University of Science and Technology of China, resolves the privilege illusion in on-policy self-distillation by aligning reference and rollout behavior via Dual-Path and Dual-Source Anchoring, achieving average gains of $+2.69$+2.69 at 4B and $+2.78$+2.78 at 32B on Qwen3 models.

Key Contributions

• This paper identifies information asymmetry between the privileged teacher and the student at inference as the root cause of privilege illusion in on-policy self-distillation, and validates this by introducing a self-conditioned student distribution that reduces late-stage wrong claims by 45% and improves mean Avg@12 by +6.22 points.
• It proposes Dual-Anchored Policy Distillation (DAPD), whose Dual-Path Anchoring aligns reference and rollout behavior under matched information via a self-conditioned bridge and whose Dual-Source Anchoring combines correctness-oriented reference guidance with student-reachable rollout guidance in both directions.
• Experiments on Qwen3 models from 1.7B to 32B show that DAPD alleviates privilege illusion and outperforms on-policy self-distillation, with an average gain of +2.00 points on Qwen3-4B and consistent scaling gains of +2.69 at 4B and +2.78 at 32B.

Introduction

Post-training techniques like on-policy self-distillation (OPSD) turn pretrained language models into reasoning models by distilling dense token-level supervision from a teacher conditioned on privileged information (such as a reference completion). Because the student cannot access this privileged information at inference, it often develops a "privilege illusion": it makes unsupported claims and hallucinates steps as if the hidden reference were available, which degrades task performance. Prior attempts to mitigate this illusion either use the privileged teacher's full supervision directly or selectively reweight it, but both approaches modify the teacher signal without aligning the information available to the teacher and student. They also rely solely on the reference as a guidance source, ignoring complementary learning signals from on-policy rollouts.

The authors identify information asymmetry as the root cause: the teacher sees privileged context that the student lacks, leading to an entangled distillation that mixes reproducible guidance with privilege-dependent behavior. To solve this, they propose Dual-Anchored Policy Distillation (DAPD). The framework introduces two modules: Dual-Path Anchoring, which aligns reference and rollout behavior under matched information availability (both with and without privileged information) using a self-conditioned student distribution as a bridge; and Dual-Source Anchoring, which balances correctness-oriented reference guidance with student-reachable rollout guidance by applying the alignment paths in both directions. Experiments across reasoning, coding, and instruction-following benchmarks show DAPD consistently reduces privilege illusion and outperforms OPSD, with stable gains across model scales (1.7B to 32B parameters) where OPSD's benefits largely diminish.

Method

The authors identify a critical flaw in On-Policy Self-Distillation (OPSD) known as privilege illusion, where the student policy learns to rely on privileged information, such as a ground-truth reference, that is available to the teacher during training but inaccessible at inference. This information asymmetry causes the student to make unsupported claims and degrades reasoning performance over time. As shown in the figure below, standard OPSD training leads to a significant increase in wrong claims and a drop in reasoning accuracy, whereas matching the information available to both teacher and student mitigates these issues.

To resolve this structural limitation, the authors propose Dual-Anchored Policy Distillation (DAPD), a framework designed to correct information asymmetry by introducing a self-conditioned distribution as a trainable bridge. The framework operates on three distinct distributions at each autoregressive token step: the None distribution, which predicts without privileged information; the Cross distribution, which conditions on the alternative completion; and the Self distribution, which conditions on the full completion being predicted. By leveraging the Self distribution, DAPD constructs anchors that align reference and rollout behavior under matched information conditions.

The core of DAPD consists of two main components: Dual-Path Anchoring (DPA) and Dual-Source Anchoring (DSA). The overall architecture and information flow are illustrated in the figure below.

Dual-Path Anchoring (DPA) matches information availability during distillation by composing three directed objectives into two complementary alignment paths for a given completion direction. The first objective, Entangled Distillation, moves the None distribution toward the Cross distribution to preserve privileged guidance transfer. The second objective, Inference Anchor, trains the Self distribution toward the None distribution to align behavior without privileged information. The third objective, Privileged Anchor, trains the Self distribution toward the Cross distribution to align behavior when both receive privileged information.

DPA combines these objectives into an unconditioned path and a privileged path. The unconditioned path targets inference-relevant distributions by combining Entangled Distillation on the current completion with the Inference Anchor on the alternative completion. This joint update implicitly aligns the two None distributions through the Self distribution acting as a proxy bridge. The privileged path directly applies the Privileged Anchor to align the Self and Cross distributions when both are conditioned on a full completion.

Dual-Source Anchoring (DSA) balances reference-guided and rollout-guided supervision by applying DPA in both completion directions. In the Rollout-to-Reference direction, the rollout serves as the completion and the reference supplies guidance, leveraging the reliability of the ground truth. Conversely, in the Reference-to-Rollout direction, the reference becomes the completion and the rollout supplies guidance, capturing useful on-policy reasoning signals that are reachable by the student. The final DAPD objective balances these two directional paths using a weighting parameter $\lambda$λ, ensuring the model benefits from both correctness-oriented reference guidance and student-reachable rollout guidance.

Experiment

The experiments evaluate DAPD across Qwen3 model scales on reasoning, coding, and instruction tasks, comparing against on-policy self-distillation baselines that suffer from privilege illusion. DAPD consistently outperforms these methods by aligning teacher and student information conditions, with gains that persist at larger scales where baseline improvements vanish. Ablations confirm that both the inference and privileged anchor paths are necessary, and that combining reference and rollout guidance provides complementary supervision. The approach remains effective even when replacing curated references with dual on-policy rollouts and further improves with a verification signal, demonstrating robustness without external references.

DAPD achieves the highest overall average across the six benchmarks, surpassing standard on-policy self-distillation (OPSD) by a clear margin. It also outperforms methods that filter or reroute the privileged teacher signal, showing that matching the information conditions of teacher and student is more effective than correcting an asymmetric signal. Gains are consistent across reasoning, coding, and instruction tasks. DAPD improves the six-task average by over two points compared to OPSD, the strongest on-policy baseline. Matched-information anchoring provides larger gains than modifying the privileged signal, with DAPD outperforming Purified OPSD and DOPD by notable amounts.

After training on reasoning data, DAPD transfers best to unseen coding and instruction benchmarks, achieving the highest three-task average. It outperforms both standard on-policy distillation and methods that modify the privileged signal, showing that matching teacher–student information conditions improves out-of-distribution generalization. DAPD obtains the top LCB v5 score (54.14) and the best overall average (49.64), leading the next-best Purified OPSD by 0.86 points on average. DAPD surpasses standard on-policy distillation (OPSD) by 1.37 average points, while purification (Purified OPSD) and routing (DOPD) strategies yield smaller gains or lower performance.

Ablation studies with Qwen3 models show that both the dual-path structure within each guidance source and the dual-source combination improve reasoning scores. The best result of 65.28 Avg@12 is achieved by combining reference and rollout guidance, while the optimal guidance balance shifts with scale, and gains persist when using only on-policy rollouts with a correctness verifier. Combining reference and rollout guidance reaches 65.28 Avg@12, outperforming each source alone (63.89 and 65.09). Larger models prefer a lower reference-guidance weight (λ down to 0.2) and adjusted privileged anchor weights, showing reduced dependence on static references. Using dual independent rollouts without curated references improves over the on-policy baseline, and verifying one rollout further lifts scores by up to 1.11 points.

DAPD matches teacher and student information conditions, outperforming on-policy self-distillation and methods that filter or reroute the privileged signal across six reasoning, coding, and instruction benchmarks, with gains transferring to unseen tasks. Ablation studies with Qwen3 models confirm the value of dual-path and dual-source guidance, as combining reference and rollout signals yields the highest performance, while larger models benefit from reduced reliance on static references and pure on-policy rollouts with a verifier also improve over baselines.