One-sentence Summary

Researchers from University of Maryland, College Park, University of California, San Diego, Duke University, and MBZUAI propose Visual Contrastive Self-Distillation (VCSD), an on-policy self-distillation method that generates a stronger teacher signal purely via input conditioning by computing token-wise log-probability differences between the EMA teacher’s original-image and content-erased-image distributions to sharpen the target, and using the ViRL39K dataset VCSD improves the seven-benchmark aggregate for Qwen3-VL from 62.27% to 67.04% at 2B, from 71.30% to 73.16% at 4B, and from 72.51% to 76.26% at 8B, while requiring no external teacher, privileged answers, visual evidence signals, reasoning traces, or additional inference-time cost.

Key Contributions

• Matched input conditioning provides the asymmetry required for on-policy self-distillation, removing the need for privileged answers or visual evidence, by contrasting the model’s predictions under the original image and a content-erased version.
• Visual Contrastive Self-Distillation (VCSD) constructs a sharpened full-distribution target from the token-wise log-probability difference between the teacher’s next-token distributions under the original image and a content-erased control, and distills that target into the student.
• Across seven vision-language benchmarks, VCSD consistently improves Qwen3-VL (2B, 4B, 8B) and Qwen3.5 models over both base models and matched on-policy self-distillation baselines, raising aggregate scores from 62.27% to 67.04% (2B), 71.30% to 73.16% (4B), 72.51% to 76.26% (8B), and yielding 2.9% to 4.3% gains on Qwen3.5.

Introduction

On-policy distillation trains a vision-language model (VLM) on its own generated prefixes, but self-distillation that removes the external teacher struggles because the teacher and student see identical information. Prior methods create the necessary teacher–student asymmetry through auxiliary signals such as privileged answers or cropped visual evidence, which are task-specific, require extra pipelines, or rely on external models. The authors leverage paired conditioning: they feed the same prefix to an EMA teacher under the original image and a content-erased control, then contrast the two token-level log-probability distributions. This visual contrast sharpens the original-image distribution within its plausible support, producing a detached, full-distribution target for on-policy self-distillation. The approach, called Visual Contrastive Self-Distillation (VCSD), needs no external teacher, annotations, or privileged information and consistently improves Qwen3-VL and Qwen3.5 models from 2B to 9B across seven vision-language benchmarks.

Method

The authors propose VCSD, a method that constructs the target asymmetry required for on-policy self-distillation from matched visual conditioning. In this setup, a trainable student model $\pi_{\theta}$πθ​ and its exponential moving average teacher $\pi_{\phi}$πϕ​ are utilized. The teacher acts as a stop-gradient target model and does not receive direct gradient updates. For each prompt-image pair $(P, J)$(P,J) from the dataset, the student samples an on-policy response $y \sim \pi_{\theta}(\cdot \mid P, J)$y∼πθ​(⋅∣P,J). At each generation step $t$t, both models are evaluated along the same student-generated prefix $y_{<t}$y<t​.

Refer to the framework diagram

To construct a more informative teacher target, the authors introduce a visual conditioning contrast. They generate a content-erased control $J_{\mathrm{ctrl}} = \mathcal{C}(J)$Jctrl​=C(J), which is a same-size black RGB image that preserves the resolution, multimodal input interface, and visual-token count while removing instance-specific content. At each fixed prefix $y_{<t}$y<t​, the EMA teacher produces two next-token distributions: one conditioned on the original image $p_{\phi, t}^J(v) = \pi_{\phi}(v \mid P, J, y_{<t})$pϕ,tJ​(v)=πϕ​(v∣P,J,y<t​) and one on the control image $p_{\phi, t}^0(v) = \pi_{\phi}(v \mid P, J_{\mathrm{ctrl}}, y_{<t})$pϕ,t0​(v)=πϕ​(v∣P,Jctrl​,y<t​). The vocabulary-level log-probability contrast is then measured as:

$$\Delta_t(v) = \log p_{\phi, t}^J(v) - \log p_{\phi, t}^0(v)$$Δt​(v)=logpϕ,tJ​(v)−logpϕ,t0​(v)

A positive $\Delta_t(v)$Δt​(v) indicates that token $v$v receives greater support under the original image. This contrast describes how the teacher's next-token preferences change in response to instance-specific visual content.

However, a large relative change does not guarantee a token is likely under the original image. To address this, the authors apply contrastive shaping only to candidates considered sufficiently likely by the original-image teacher distribution. They define a relative plausibility support set:

$$\mathcal{S}_t(\beta) = \left\{v \in \mathcal{V}: p_{\phi, t}^J(v) \geq \beta \max_{u \in \mathcal{V}} p_{\phi, t}^J(u) \right\}$$St​(β)={v∈V:pϕ,tJ​(v)≥βu∈Vmax​pϕ,tJ​(u)}

where $\beta$β controls the support threshold. The contrast-shaped teacher target is constructed as:

$$q_t^\star(v) = \frac{\mathbf{1}[v \in \mathcal{S}_t(\beta)] p_{\phi, t}^J(v) \exp(\alpha \widetilde{\Delta}_t(v))}{\sum_{u \in \mathcal{S}_t(\beta)} p_{\phi, t}^J(u) \exp(\alpha \widetilde{\Delta}_t(u))}$$qt⋆​(v)=∑u∈St​(β)​pϕ,tJ​(u)exp(αΔt​(u))1[v∈St​(β)]pϕ,tJ​(v)exp(αΔt​(v))​

Here, $\alpha \geq 0$α≥0 controls the shaping strength, and $\widetilde{\Delta}_t$Δt​ equals $\Delta_t$Δt​ except that sequence-termination tokens have their contrast values set to zero for stability. The original-image distribution determines the admissible candidate set and initial probability, while $\widetilde{\Delta}_t$Δt​ adjusts relative probabilities based on visual evidence.

The contrast-shaped target is distilled into the student at every position of the student-generated response using full-distribution forward KL divergence:

$$\mathcal{L}_{\mathrm{VCSD}} = T_{\mathrm{KD}}^2 \mathbb{E}_{\substack{(P,J)\sim \mathcal{D} \\ y\sim \pi_{\theta}(\cdot |P,J)}} \left[ \frac{1}{|y|} \sum_{t=1}^{|y|} D_{\mathrm{KL}}\left(\operatorname{sg}[q_t^\star] \parallel p_{\theta,t}\right) \right]$$LVCSD​=TKD2​E(P,J)∼Dy∼πθ​(⋅∣P,J)​​​∣y∣1​t=1∑∣y∣​DKL​(sg[qt⋆​]∥pθ,t​)​

where $\operatorname{sg}[\cdot]$sg[⋅] denotes stop-gradient and $T_{\mathrm{KD}}$TKD​ is the distillation temperature. Gradients pass only through the student distribution. After each student update, the teacher parameters are updated via $\phi \leftarrow \mu \phi + (1 - \mu) \theta$ϕ←μϕ+(1−μ)θ.

From a theoretical perspective, the conditional log-ratio $\Delta_t(v)$Δt​(v) serves as an implicit visual-evidence reward. The contrast-shaped target corresponds to the unique solution of a one-step KL-regularized policy update, where the original-image prediction acts as the reference policy. Furthermore, $\Delta_t(v)$Δt​(v) approximates the conditional pointwise mutual information between a candidate token and the observed image. Maximizing this objective encourages the student to preserve image-dependent evidence in its next-token distribution, promoting stronger dependence on instance-specific visual evidence while retaining the fluency encoded by the teacher.

Experiment

The experiments evaluate VCSD on Qwen3-VL and Qwen3.5 across 2B to 9B scales, comparing against base models and answer-hint OPSD on seven benchmarks spanning general perception, math, high-resolution, and hallucination tasks. VCSD consistently outperforms OPSD, demonstrating that input-conditioned contrastive targets are more effective than privileged answer hints; the method robustly improves visual grounding by shifting probability toward image-dependent tokens. Ablations confirm that plausibility support prevents recursive target distortion, contrastive strength is insensitive around α=1, forward KL distillation is optimal, and control-image construction is largely interchangeable. The original-image anchor reduces language drift without harming accuracy, and training dynamics show VCSD resists late-stage degradation better than OPSD.

VCSD consistently outperforms both the base model and OPSD across all tested Qwen3-VL and Qwen3.5 model scales, improving average accuracy by +1.86% to +4.77% over the base models. Gains span general visual perception, mathematical reasoning, high-resolution tasks, and hallucination benchmarks, indicating broad visual grounding rather than isolated improvements. The method also shows higher robustness to continued training with less late-stage degradation. On Qwen3-VL-2B, VCSD improves every benchmark over OPSD, with HallusionBench gaining +4.32% and MMStar +3.00%. On Qwen3.5 models, OPSD provides no consistent gain over the base model, whereas VCSD improves all tested scales and achieves the highest aggregate accuracy across all six configurations.

Forward KL divergence consistently yields the highest aggregate accuracy when distilling a contrast-shaped target, outperforming both Jensen-Shannon divergence and reverse KL. It ranks first on six out of seven benchmarks, suggesting that a mode-covering objective better preserves the full target distribution for this visually conditioned distillation task. Forward KL achieves the best overall accuracy, exceeding JSD by 0.79 points and reverse KL by 2.27 points. Forward KL places first on six of the seven evaluated benchmarks, while the alternatives lag behind on most tasks.

Different control-image constructions (black, Gaussian noise, Gaussian blur, and no image) yield highly similar average accuracy across seven benchmarks, all within a narrow 0.9-point range. The contrastive signal is robust to the specific degradation method, and the shared requirement is the removal of instance-specific visual content rather than any particular pattern. Aggregate accuracy for all four control-image variants falls between 66.24 and 67.14, a spread of less than one point. Content removal, whether via a black image, noise, blur, or absent image, is sufficient for effective contrastive shaping. The contrastive target is insensitive to the exact control construction, as qualitatively different inputs produce comparable results.

Removing the original-image anchor from VCSD keeps aggregate accuracy nearly unchanged on Qwen3-VL-2B, with a 0.26-point difference across the seven benchmarks. The per-benchmark shifts are small and inconsistent, indicating that the contrastive shaping component, not the anchor, drives the capability gains. The anchor instead regularizes language consistency during training, as shown by reduced language drift. Without the anchor, the seven-benchmark average accuracy is 66.78, almost identical to 67.04 with the full VCSD method. Removing the anchor lowers BLINK accuracy by 1.42 points and MathVista by 0.90 points, but improves V* accuracy by 2.09 points. The paper interprets the result as evidence that capability gains come mainly from contrastive shaping, while the anchor improves language stability.

Experiments on Qwen3-VL and Qwen3.5 models across multiple benchmarks show that VCSD provides broad gains over baselines, with forward KL divergence being the best distillation objective. The contrastive signal is insensitive to control-image type, and the anchor primarily preserves language stability rather than boosting accuracy.