One-sentence Summary

A team from Shanghai Jiao Tong University, Xiaohongshu Inc., The Chinese University of Hong Kong, Zhejiang University, and Southeast University introduces Visual Attribution Distillation (VAD), a counterfactual target-reconstruction method that isolates visual evidence in multimodal on-policy distillation by counterfactually evaluating a fixed teacher to compute a signed proxy $u_t$ut​, then reconstructs a student-anchored target from the proxy-aligned component, and demonstrates that VAD outperforms direct privileged-view distillation and visual-advantage weighting on six fine-grained visual benchmarks at 4B and 9B scales.

Key Contributions

• Visual Attribution Distillation (VAD) estimates the visually attributable part of a teacher correction by contrasting the teacher with evidence present and removed to define a signed proxy u_t, then projects the original correction onto this proxy to reconstruct a student-anchored target.
• VAD discards teacher changes not explained by the controlled evidence, using only the intervention-aligned component to avoid the source-mixed supervision of prior on-policy distillation methods.
• Across six fine-grained visual benchmarks at 4B and 9B scales, VAD outperforms direct privileged-view distillation and visual-advantage weighting, while token-level and controlled-target analyses show the proxy-aligned component enriches task-relevant visual content and strengthens correct-token promotion and wrong-token suppression.

Introduction

Multimodal large language models (MLLMs) frequently fail because a subtle misperception (a missed word, confused attribute, or misread spatial relation) cascades into a fluent but visually ungrounded response. On-policy distillation from a stronger teacher can mitigate these errors by supervising the student on its own generated rollouts, but existing privileged-view approaches either distill the teacher’s complete next-token distribution, which mixes visual and non-visual corrections, or apply visual-advantage weighting that still leaves source-mixed signals and cannot properly capture evidence-based refutation of mistaken tokens. The authors introduce Visual Attribution Distillation (VAD) to isolate the visual correction: for each student prefix, they query the same teacher with and without the fine-grained evidence view, derive a signed proxy direction that indicates which candidate tokens the evidence supports or refutes, and then reconstruct a student-anchored target using only the proxy-aligned component while retaining a weak regularizer. VAD explicitly separates visual support from refutation, improving over direct teacher matching and visual-advantage weighting across six fine-grained visual benchmarks for both 4B and 9B students.

Method

The authors propose Visual Attribution Distillation (VAD), a method that learns from on-policy student prefixes while utilizing counterfactual teacher views exclusively to construct supervision targets. The conceptual pipeline of this approach is illustrated below.

The process begins with the generation of on-policy counterfactual views. Let $x^0$x0 denote the full image available to the student. The student policy $\pi_\theta$πθ​ samples a response $y \sim \pi_\theta(\cdot | x^0)$y∼πθ​(⋅∣x0), and all teacher queries reuse the student prefix $y_{<t}$y<t​. A fixed copy $\pi_{\bar{\theta}}$πθˉ​ of the initial model serves as the teacher. The authors construct two training-only views: an evidence-present crop $x^+$x+ and an evidence-removed or degraded crop $x^-$x−. At position $t$t, the three next-token distributions are computed as:

$$\begin{array}{r} p_S^0 = \pi_\theta(\cdot | x^0, y_{<t}), \\ p_T^+ = \pi_{\bar{\theta}}(\cdot | x^+, y_{<t}), \\ p_T^- = \pi_{\bar{\theta}}(\cdot | x^-, y_{<t}). \end{array}$$pS0​=πθ​(⋅∣x0,y<t​),pT+​=πθˉ​(⋅∣x+,y<t​),pT−​=πθˉ​(⋅∣x−,y<t​).​

Because the teacher parameters and text prefix remain identical, the contrast $p_T^+ - p_T^-$pT+​−pT−​ isolates the teacher's distributional response when the relevant visual evidence is made visible. The training implementation evaluates all three distributions on a compact shared coordinate set $V_t$Vt​ constructed from the student's top-K candidates. Each restricted distribution is mapped to a centered log-probability vector in $\mathbb{R}^{|V_t|}$R∣Vt​∣:

$$\phi_t(p) = \log(p[V_t] + \epsilon) - \text{mean}(\log(p[V_t] + \epsilon)).$$ϕt​(p)=log(p[Vt​]+ϵ)−mean(log(p[Vt​]+ϵ)).

Centering removes a common logit offset while preserving pairwise log-odds on the shared support.

Next, the authors attribute the teacher correction to the visual evidence. The privileged-teacher correction and its response to the visual intervention are defined as:

$$r_t = \phi_t(p_T^+) - \phi_t(p_S^0), \qquad u_t = \phi_t(p_T^+) - \phi_t(p_T^-).$$rt​=ϕt​(pT+​)−ϕt​(pS0​),ut​=ϕt​(pT+​)−ϕt​(pT−​).

Here, $r_t$rt​ represents the complete correction prescribed by the privileged teacher. The vector $u_t$ut​ serves as a proxy for the core visual information available at position $t$t. The part of $r_t$rt​ that agrees with this visual response is retained through a one-sided projection:

$$\beta_t = \frac{[\langle r_t, u_t \rangle]_+}{\|u_t\|_2^2 + \zeta}, \qquad r_t^{\text{vis}} = \beta_t u_t, \qquad r_t^{\text{res}} = r_t - r_t^{\text{vis}},$$βt​=∥ut​∥22​+ζ[⟨rt​,ut​⟩]+​​,rtvis​=βt​ut​,rtres​=rt​−rtvis​,

where $[a]_+ = \max(a, 0)$[a]+​=max(a,0) and $\zeta$ζ stabilizes the projection. If $r_t$rt​ and $u_t$ut​ do not agree, $\beta_t = 0$βt​=0 and no visual shift is applied. This yields a one-sided supervision target:

$$q_{T,t}^{\text{one}} = \text{softmax}(\phi_t(p_S^0) + \text{clip}(r_t^{\text{vis}}, -c, c)).$$qT,tone​=softmax(ϕt​(pS0​)+clip(rtvis​,−c,c)).

Unlike direct teacher matching, this target starts from the student's current distribution and changes only the token odds attributed to the visual intervention.

To further refine the target, a lightweight budgeted support and refutation mechanism decouples the support and refutation coordinates. The authors define:

$$\begin{array}{c} u_t^+ = [u_t]_+, \qquad u_t^- = [u_t]_-, \qquad s_t^\pm = [\langle r_t, u_t^\pm \rangle]_+, \\ Z_t = s_t^+ + s_t^- + \epsilon, \qquad \omega_t^+ = \min\left(\frac{s_t^+}{Z_t}, \tau_+\right), \qquad \omega_t^- = \frac{s_t^-}{Z_t}, \end{array}$$ut+​=[ut​]+​,ut−​=[ut​]−​,st±​=[⟨rt​,ut±​⟩]+​,Zt​=st+​+st−​+ϵ,ωt+​=min(Zt​st+​​,τ+​),ωt−​=Zt​st−​​,​

where $[u_t]_+ = \max(u_t, 0)$[ut​]+​=max(ut​,0) and $[u_t]_- = \min(u_t, 0)$[ut​]−​=min(ut​,0). The agreement scores allocate the visual-correction budget $B_t = \|r_t^{\text{vis}}\|_2$Bt​=∥rtvis​∥2​ between the two signed branches. The resulting correction and target are:

$$\begin{array}{rl} r_t^{\text{VAD}} &= B_t \left(\omega_t^+ \frac{u_t^+}{\|u_t^+\|_2 + \epsilon} + \omega_t^- \frac{u_t^-}{\|u_t^-\|_2 + \epsilon}\right), \\ q_{T,t}^{\text{VAD}} &= \text{softmax}(\phi_t(p_S^0) + \text{clip}(r_t^{\text{VAD}}, -c, c)). \end{array}$$rtVAD​qT,tVAD​​=Bt​(ωt+​∥ut+​∥2​+ϵut+​​+ωt−​∥ut−​∥2​+ϵut−​​),=softmax(ϕt​(pS0​)+clip(rtVAD​,−c,c)).​

Finally, the training objective and inference procedure are established. Given the reconstructed target $q$q and student distribution $p$p, let $m = (q + p) / 2$m=(q+p)/2. The primary supervision is the standard token-level Jensen-Shannon objective:

$$\mathcal{L}_{\text{vis}} = \frac{1}{|\mathcal{T}|} \sum_{t \in \mathcal{T}} D_{\text{JS}}(\text{stopgrad}(q_{T,t}^{\text{VAD}}), p_S^0),$$Lvis​=∣T∣1​t∈T∑​DJS​(stopgrad(qT,tVAD​),pS0​),

where $\mathcal{T}$T is the set of valid response positions. Using $\mathcal{L}_{\text{vis}}$Lvis​ alone can cause language and output drift. To mitigate this, the authors add a weak privileged-teacher regularizer. Its token weight is larger when a smaller fraction of the complete correction is visually attributed:

$$\begin{array}{rl} \rho_t &= \frac{\|r_t^{\text{VAD}}\|_2}{\|r_t\|_2 + \epsilon}, \qquad a_t = \text{stopgrad}(\text{clip}(1 - \rho_t, 0, 1)), \\ \mathcal{L}_{\text{reg}} &= \frac{1}{|\mathcal{T}|} \sum_{t \in \mathcal{T}} a_t D_{\text{JS}}(\text{stopgrad}(p_T^+), p_S^0), \qquad \mathcal{L} = \mathcal{L}_{\text{vis}} + \lambda \mathcal{L}_{\text{reg}}. \end{array}$$ρt​Lreg​​=∥rt​∥2​+ϵ∥rtVAD​∥2​​,at​=stopgrad(clip(1−ρt​,0,1)),=∣T∣1​∑t∈T​at​DJS​(stopgrad(pT+​),pS0​),L=Lvis​+λLreg​.​

The regularizer weakly anchors semantics, formatting, response length, and stopping while leaving the reconstructed visual target as the primary signal. All target-side quantities are detached during optimization. At inference time, VAD operates as a standard full-image student policy and introduces no additional model calls or visual views.

Experiment

Experiments on Qwen3.5 models at 4B and 9B scales evaluate Visual Attribution Distillation (VAD), a post-training method that uses counterfactual image crops to attribute teacher corrections to visual evidence and then reconstructs the resulting student target. Under matched data and compute, VAD consistently outperforms alternative objectives and even larger models on a suite of fine-grained visual benchmarks, while also preserving base-level generalization on held-out tasks. Analysis shows that the attributed correction selectively concentrates visual attributes and decision semantics, and the approach is robust to hyperparameter choices, with Jensen-Shannon divergence providing the most balanced results across tasks.

VAD achieves the highest fine-grained perception average among all compared models, surpassing the closed-source Gemini 3 Flash by 1.0 point at 4B and 2.6 points at 9B. Under matched data and post-training budgets, VAD consistently leads other Qwen3.5-based methods, with gains over regularized one‑sided projection and its target‑only variant highlighting the benefit of joint attribution reconstruction and weak regularization. VAD reaches Avg₆ of 78.32 at 4B and 79.93 at 9B, exceeding the best closed-source baseline Gemini 3 Flash (77.32) and the strongest thinking‑with‑images agent SenseNova‑MARS (72.10). Within the controlled Qwen3.5 family, VAD outperforms the next‑best method by 0.80 points at 4B and its own target‑only ablation by 0.26 points, demonstrating the value of the full distillation objective.

Refining the distillation target from the direct teacher distribution to scalar-shrunk, one-sided, and finally the full VAD objective yields steady multi-benchmark accuracy gains, with full VAD attaining the highest average. The weak teacher anchor provides consistent incremental improvement for both one-sided and VAD targets, and the VAD direction drives the largest favorable shift in token-level support and suppression. Full VAD reaches 78.32% average accuracy, outperforming the regularized one-sided projection by 0.80 points and the unregularized VAD target by 0.26 points. Adding a weak regularizer lifts the one-sided target from 77.06% to 77.52% and VAD from 78.06% to 78.32%, showing consistent small gains. The VAD target produces the largest combined improvement in correct-token support (+7.89 points) and wrong-token suppression (+6.61 points) relative to the direct teacher distribution.

Among the post-trained visual models evaluated, VAD is the only method that maintains or slightly exceeds the base model’s held-out generalization at both tested scales, while all other methods degrade average performance to varying degrees. Decomposed OPD comes close at the smaller scale but suffers a notable drop at the larger one, and simpler post-training variants show more pronounced declines. VAD achieves positive gains over the scale-matched base model at both 4B and 9B, making it the sole post-trained method with a net improvement on held-out benchmarks. Decomposed OPD performs almost on par with the base model at 4B but falls significantly below it at 9B. Other post-training approaches, such as GRPO and VA-OPD, consistently reduce held-out average scores below the base level.

Symmetric Jensen-Shannon divergence yields the highest aggregate accuracy across six fine-grained benchmarks for both 4B and 9B model scales. JSD leads on the majority of individual tasks, though Reverse KL shows competitive results on HRBench-8K and MME-EN at the 9B scale. Forward KL consistently underperforms the other two divergences in overall average. JSD achieved the best average score at both scales, leading five out of six benchmarks at 4B and four at 9B. At 4B, Forward KL slightly outperformed JSD on V*, but JSD was clearly ahead on all other tasks. At 9B, Reverse KL surpassed JSD on HRBench-8K and MME-EN, narrowing the gap but not overcoming JSD's aggregate advantage. Forward KL produced the lowest overall averages at both model sizes, trailing both symmetric and reverse KL variants.

the paper evaluate VAD, a visual post-training approach that jointly reconstructs token attributions and applies a weak regularizer, on six fine-grained perception benchmarks and held-out generalization tasks. VAD surpasses all compared models, including closed-source Gemini 3 Flash and other Qwen3.5-based methods, and is the only post-trained technique that maintains or improves over the base model on held-out data. Ablation studies confirm that the full VAD objective with Jensen-Shannon divergence provides the best balance of correct-token support and wrong-token suppression, leading to the highest average accuracy.