One-sentence Summary

Researchers from Tsinghua University, Zhejiang University, and Alibaba Group introduce Self-OPD, a teacher-free on-policy distillation framework for flow matching that turns the student's self-exploration into step-wise supervision by branching next-state predictions into $K$K stochastic SDE candidates and applying an all-branch pull-push objective with reward-level fusion, outperforming prior RL and OPD methods on single and mixed reward benchmarks without task-specific teachers.

Key Contributions

• The paper presents Self-OPD, a teacher-free on-policy distillation framework for flow matching models that converts reward-guided self-exploration and a self-reference baseline into dense step-wise supervision, and shows through experiments on single and mixed reward benchmarks that it outperforms prior RL and teacher-based OPD methods without task-specific teachers.
• It introduces an all-branch pull-push distillation objective with SDE-variance normalization and direction-aware attenuation, which uses both high- and low-advantage branches to optimize the velocity field while maintaining stable training.
• It develops a reward-level fusion strategy for multi-objective alignment that combines normalized scalar scores to rank sampled branches, enabling black-box composition without field-level teacher routing or gradient conflicts.

Introduction

Flow matching models have become a dominant backbone for high-quality visual generation, but aligning them with downstream objectives such as text rendering, compositional correctness, and human preference remains challenging. Prior reinforcement learning approaches assign credit from terminal rewards over many denoising steps, leading to high-variance gradients and fragile multi-objective alignment. On-policy distillation methods offer stable per-step supervision by regressing toward a pretrained teacher, yet they require task-specific teachers, are upper-bounded by teacher quality, and struggle to fuse multiple teachers without conflicting updates. The authors propose Self-OPD, a teacher-free on-policy distillation framework that replaces teacher-provided velocity targets with reward-weighted targets discovered through the student’s own local exploration. By branching trajectories, evaluating them with task rewards, and using a deterministic self-reference baseline, Self-OPD provides dense step-wise supervision without any external teacher and fuses multiple objectives at the reward level, enabling a single image to simultaneously satisfy diverse criteria.

Method

The authors propose Self-OPD, a teacher-free on-policy distillation framework that transforms self-generated rewards into dense, step-wise supervision. As illustrated in the framework diagram,

the pipeline begins by exploring local trajectories around the student’s on-policy path via stochastic SDE branching, evaluating these candidates against a deterministic self-reference baseline, and finally distilling the collective feedback through an advantage-weighted pull-push objective.

Flow Matching and SDE Formulation The foundation of the method relies on Flow Matching, which learns a continuous velocity field $v_\theta$vθ​ to transport a noise distribution to the data distribution. During inference, samples are generated by integrating the reverse-time ordinary differential equation backward. To enable stochastic exploration, the authors augment the reverse ODE into a reverse-time stochastic differential equation. Under the Euler-Maruyama discretization scheme, each transition step decomposes into a deterministic next-state prediction and an isotropic stochastic perturbation. By substituting the score function into the drift formulation, the deterministic prediction $x_{t_{j+1}, \theta}$xtj+1​,θ​ exhibits an elegant affine relationship with the velocity field:

where $c_{t_j} = (1 + \frac{\eta^2}{2}) \Delta t_j$ctj​​=(1+2η2​)Δtj​. This affine equivalence guarantees that any distillation objective formulated in transition space can be minimized with identical convergence properties in velocity space.

Self-Referenced Evaluation and Self-Exploration To retain dense supervision without a pretrained teacher, the framework allows the student to evaluate its own local alternatives. At timestep $t_j$tj​, the method computes the deterministic next-state prediction with a single forward pass and instantiates $K$K candidate branches by drawing independent Gaussian perturbations. All branches share the same base prediction, meaning the expensive transformer evaluation is performed only once.

Each branch is completed to a clean latent via a deterministic ODE rollout, decoded to pixel space, and scored by a task-specific reward model. To obtain a variance-reducing baseline, the authors also run a fully deterministic ODE trajectory directly from the parent state, denoting its reward as $r^{\text{ode}}$rode. The branch advantage is then normalized relative to this self-reference:

Positive advantages indicate local directions that outperform the student’s default trajectory, while negative advantages identify directions to avoid.

All-Branch Pull-Push Distillation Instead of regressing only to the best branch, Self-OPD utilizes the full exploration neighborhood. Positive-advantage branches pull the deterministic prediction toward locally better trajectories, while negative-advantage branches push it away from poor directions. To prevent destructive interference when a low-reward branch points in nearly the same direction as the best branch, the authors introduce a direction-aware attenuation coefficient $d_k$dk​. This coefficient keeps full repulsion for negative branches pointing opposite to the best direction and suppresses repulsion when aligned.

Because the deterministic prediction is affine in $v_\theta$vθ​, every branch can be associated with an effective branch velocity. The step-wise Self-OPD objective is formulated as:

The first term pulls $v_\theta$vθ​ toward high-reward branch velocities, while the second term pushes it away from low-reward velocities after direction-aware attenuation. This normalization admits a KL interpretation, where the transition-variance normalization acts as the precision making every per-step regression an unbiased estimate of the KL gradient. The total training objective aggregates these step-wise losses across the entire denoising trajectory, prioritizing early-to-mid timesteps that establish the global semantic layout.

Reward-Level Fusion for Multi-Objective Alignment Simultaneous multi-objective alignment often triggers destructive gradient interference in the shared parameter space. The authors bypass this by fusing objectives at the non-differentiable reward level, shifting the Pareto-front search from the parameter space to the trajectory space. The joint target collapses into a single tilt of the transition kernel by the composite advantage. This is realized by fusing the normalized scores into a composite reward for each branch:

Crucially, the composite score enters training only through the branch ranking that defines the advantages and is never differentiated. The training target remains a single, concrete trajectory velocity that already scores well across all metrics. This reward-level paradigm obviates inter-objective gradient conflicts, accommodates black-box scorers, and allows runtime composability by adjusting trade-offs without model retraining.

Experiment

The evaluation uses SD3.5-Medium with LoRA across text rendering, compositional generation, and aesthetic alignment tasks, comparing Self-OPD against teacher-free RL baselines and teacher-based OPD methods. Self-OPD consistently outperforms all single- and mixed-reward competitors by enabling step-wise self-distillation through local SDE branching and self-referenced advantage evaluation, eliminating the need for a pretrained teacher. Qualitative results show superior text fidelity, spatial reasoning, and counting accuracy, while ablation studies confirm that all-branch advantage weighting, bounded repulsion, and early-timestep loss scaling are crucial for stable training and faster convergence. Overall, the method demonstrates that reward-level fusion of multiple objectives directly in the exploration phase yields better joint alignment than field-level teacher blending, and it achieves competitive performance with substantially reduced training time.

Self-OPD is a teacher-free alignment paradigm that provides dense per-step credit assignment and local exploration, adapting to the student policy. Ablation studies show that using all branches with advantage-weighted distillation and step-aligned KL normalization stabilizes training, while bounded repulsion and early-timestep weighting accelerate convergence without sacrificing final performance. Compared to teacher-based OPD, Self-OPD eliminates the teacher-training bottleneck, reaching comparable or higher performance in roughly half the wall-clock time. Self-OPD is teacher-free and assigns dense per-step credit, unlike Flow-GRPO which uses terminal credit assignment. Distilling from all branches with advantage weighting and KL normalization stabilizes training, whereas selecting only the top branch leads to instability from high gradient variance. Bounding the repulsion term prevents gradient domination and training collapse, ensuring stable monotonic improvement. Weighting losses toward early timesteps accelerates convergence, but oversampling early steps degrades performance, indicating that exposure to all trajectory stages is necessary. Self-OPD from scratch reaches teacher-based OPD performance faster, and a warm-start strategy is roughly twice as fast while achieving higher final scores.

Self-OPD, a teacher-free method, achieves the highest scores on all evaluation dimensions in single-reward training, outperforming specialized Flow-GRPO models that were each fine-tuned with a different individual reward. While those single-reward models show strong performance only on the metric aligned with their training reward and often degrade on others, Self-OPD simultaneously improves compositional generation, text rendering, and human preference scores, demonstrating effective multi-capability alignment through step-wise self-distillation. Self-OPD surpasses all single-reward Flow-GRPO variants on GenEval, OCR, PickScore, and HPSv2, securing the top result among trained methods. Flow-GRPO models exhibit narrow specialization: optimizing for GenEval reward yields high GenEval but low OCR and aesthetic scores, while optimizing for OCR or PickScore reward similarly sacrifices performance on other axes. The base model without alignment fine-tuning shows limited capability, and Self-OPD lifts all metrics substantially above this baseline, unlike single-reward fine-tuning which creates trade-offs. Self-OPD's consistent gains across all tasks highlight the advantage of step-wise self-distillation over terminal policy gradient optimization for multi-faceted alignment.

In mixed-reward training, the teacher-free Self-OPD model achieves the highest GenEval strict score (0.95) and OCR accuracy (96.0%), and on the same test images protocol it outperforms the teacher-based DiffusionOPD in both PickScore (23.87 vs. 22.72) and HPSv2 (0.3214 vs. 0.2676). This demonstrates that reward-level fusion selects generations inside a joint high-reward region, enabling a single model to satisfy multiple criteria simultaneously, unlike field-level fusion where preference quality degrades on task prompts. Self-OPD attains the top GenEval strict score of 0.95 and the best OCR accuracy of 96.0% among mixed-reward methods, without any teacher supervision. Under the same test images protocol, Self-OPD surpasses the teacher-based DiffusionOPD by 1.15 points in PickScore and by 0.0538 in HPSv2, confirming the advantage of reward-level fusion over field-level fusion. Reward-level fusion keeps preference scores stable across prompt populations (PickScore shift of 0.48 for Self-OPD), while field-level fusion causes a large drop (PickScore shift of 1.23 for DiffusionOPD) when evaluated on task prompts.

The experiments evaluate Self-OPD, a teacher-free alignment method that uses step-wise self-distillation with dense credit assignment. Ablation studies confirm that distilling from all branches with advantage weighting and KL normalization stabilizes training, while bounded repulsion and early-timestep weighting accelerate convergence. In single-reward training, Self-OPD simultaneously improves compositional generation, text rendering, and human preference scores, avoiding the narrow specialization seen in Flow-GRPO models. In mixed-reward training, reward-level fusion enables a single model to satisfy multiple criteria without the preference degradation observed with field-level fusion, and Self-OPD achieves comparable or higher performance than teacher-based methods in roughly half the time.