One-sentence Summary

Researchers at the Chinese Academy of Sciences, the University of Chinese Academy of Sciences, and Meituan propose On-Policy direction-constrained Supervised Fine-Tuning (OPSFT), which constrains supervised fine-tuning (SFT) updates to the cumulative parameter update directions identified by on-policy paradigms, transferring on-policy generalization to efficient SFT while enabling the use of high-quality trajectories.

Key Contributions

• The paper identifies a distinguishing parameter-update behavior: supervised fine-tuning updates parameters along consistent directions, whereas on-policy post-training continuously adjusts update directions during training.
• The paper proposes On-Policy direction-constrained Supervised Fine-Tuning (OPSFT), which constrains SFT updates to the cumulative direction identified by on-policy training; OPSFT achieves performance comparable to the corresponding on-policy paradigm, indicating that the on-policy generalization advantage transfers through update direction.
• The paper demonstrates two practical benefits: a few on-policy training steps identify directions that support strong generalization for efficient OPSFT training, and OPSFT uses newly acquired high-quality trajectories to improve a post-trained model without disrupting previously learned on-policy behavior.

Introduction

On-policy post-training has become an important approach for improving large language model reasoning because it optimizes models on their own generated responses rather than fixed supervised fine-tuning (SFT) targets, and it often generalizes more strongly across tasks. Prior work has examined isolated components such as reverse KL divergence, negative samples, and sparse update locations, but these studies generally treat on-policy update behavior as a byproduct instead of testing whether it can become an optimization principle. That leaves a gap between explaining on-policy generalization and translating those explanations into practical improvements for SFT. The authors bridge this gap by analyzing parameter update direction and find that on-policy training keeps adjusting update directions while SFT stays directionally consistent. They propose On-Policy direction-constrained SFT (OPSFT), which retains SFT but restricts each parameter update to the direction identified by on-policy training. Experiments show that OPSFT substantially outperforms vanilla SFT and reaches performance comparable to the on-policy paradigm, indicating that on-policy update direction can transfer generalization benefits to more efficient SFT.

Method

The authors explore whether an on-policy parameter update behavior can transfer the generalization advantage of on-policy paradigms to Supervised Fine-Tuning (SFT). They investigate the direction, specifically the sign, of parameter updates and its evolution throughout training.

To understand the different behaviors in parameter update direction, the authors compare the gradient formulations of on-policy paradigms and SFT. Given an input prompt $x$x, a response trajectory $\tau$τ, and the current policy $\pi_{\boldsymbol{\theta}}$πθ​ with parameters $\boldsymbol{\theta} \in \mathbb{R}^d$θ∈Rd, the gradient of the trajectory log-probability with respect to parameters $\boldsymbol{\theta}$θ is computed as:

$$s_{\boldsymbol{\theta}}(x, \tau) = \nabla_{\boldsymbol{\theta}} \log \pi_{\boldsymbol{\theta}}(\tau \mid x)$$sθ​(x,τ)=∇θ​logπθ​(τ∣x)

For every input prompt $x$x, $s_{\boldsymbol{\theta}}(x, \tau)$sθ​(x,τ) has zero conditional expectation:

$$\mathbb{E}_{\tau \sim \pi_{\boldsymbol{\theta}}(\cdot | x)} \left[ s_{\boldsymbol{\theta}}(x, \tau) \right] = \sum_{\tau} \pi_{\theta}(\tau \mid x) \nabla_{\theta} \log \pi_{\theta}(\tau \mid x) = \nabla_{\theta} \sum_{\tau} \pi_{\theta}(\tau \mid x) = \mathbf{0}_{d}$$Eτ∼πθ​(⋅∣x)​[sθ​(x,τ)]=τ∑​πθ​(τ∣x)∇θ​logπθ​(τ∣x)=∇θ​τ∑​πθ​(τ∣x)=0d​

where $\mathbf{0}_{d} \in \mathbb{R}^d$0d​∈Rd denotes the zero vector. The policy gradient of on-policy paradigms is represented as:

$$g_{\mathrm{on}}(\boldsymbol{\theta}) = \mathbb{E}_{x, \tau \sim \pi_{\boldsymbol{\theta}}(\cdot | x)} \left[ A_{\boldsymbol{\theta}}(x, \tau) s_{\boldsymbol{\theta}}(x, \tau) \right]$$gon​(θ)=Ex,τ∼πθ​(⋅∣x)​[Aθ​(x,τ)sθ​(x,τ)]

where $A_{\boldsymbol{\theta}}(x, \tau) \in \mathbb{R}$Aθ​(x,τ)∈R is the advantage of trajectory $\tau$τ for prompt $x$x. The projection of $g_{\mathrm{on}}(\boldsymbol{\theta})$gon​(θ) onto an arbitrary sign vector $v \in \{-1, 0, 1\}^d$v∈{−1,0,1}d can be formulated as the projection onto $v$v of the covariance between $A_{\boldsymbol{\theta}}(x, \tau)$Aθ​(x,τ) and $s_{\boldsymbol{\theta}}(x, \tau)$sθ​(x,τ):

$$\boldsymbol{v}^{\top} g_{\mathrm{on}}(\boldsymbol{\theta}) = \mathbb{E}_{x} \big[ \boldsymbol{v}^{\top} \operatorname{Cov}_{\tau \sim \pi_{\boldsymbol{\theta}}(\cdot | x)} \left[ A_{\boldsymbol{\theta}}(x, \tau), s_{\boldsymbol{\theta}}(x, \tau) \right] \big]$$v⊤gon​(θ)=Ex​[v⊤Covτ∼πθ​(⋅∣x)​[Aθ​(x,τ),sθ​(x,τ)]]

For SFT, the projection of their gradients along $v$v is:

$$\boldsymbol{v}^{\top} g_{\mathrm{sft}}(\boldsymbol{\theta}) = - \mathbb{E}_{x} \left[ \boldsymbol{v}^{\top} \mathbb{E}_{\tau \sim \pi_{\mathrm{teacher}}(\cdot | x)} \left[ s_{\boldsymbol{\theta}}(x, \tau) \right] \right]$$v⊤gsft​(θ)=−Ex​[v⊤Eτ∼πteacher​(⋅∣x)​[sθ​(x,τ)]]

By comparing these equations, the authors find that SFT updates parameters along the direction of $s_{\boldsymbol{\theta}}(x, \tau)$sθ​(x,τ) where trajectories are typically sampled from a fixed distribution $\pi_{\mathrm{teacher}}$πteacher​. In contrast, on-policy paradigms update parameters toward the direction of the covariance matrix between the advantages and the trajectory gradients. Since trajectories are sampled from the current policy $\pi_{\boldsymbol{\theta}}$πθ​ that evolves during training, the distribution of advantages changes with $\pi_{\boldsymbol{\theta}}$πθ​. Consequently, the parameter update direction of on-policy paradigms changes with the evolving advantage distribution throughout training.

To verify this theoretical analysis, the authors measure the cosine similarity among update directions at different training steps. As shown in the figure below:

For interval updates, SFT optimizes towards positively correlated directions, while on-policy paradigms explore nearly orthogonal directions at different training stages. The accumulation of these interval updates leads to substantially different cumulative update directions. For cumulative updates, SFT exhibits highly similar directions throughout training with cosine similarity close to 1.0, whereas on-policy paradigms exhibit positive yet substantially lower correlation with cosine similarity around 0.5. These analyses suggest that unlike SFT, which updates parameters along nearly consistent directions, on-policy paradigms tend to continuously adjust their cumulative parameter update directions throughout training.

Inspired by the continuous efforts to adjust update directions in on-policy paradigms, the authors investigate whether the resulting cumulative on-policy update direction can serve as an effective optimization principle for transferring their generalization advantage to SFT. Specifically, they constrain parameter updates in SFT to the directions identified by on-policy paradigms. Given the parameters before and after on-policy training $(\theta_{\mathrm{base}}, \theta_{\mathrm{on}})$(θbase​,θon​), they obtain the sign vector $v = \operatorname{sign}(\theta_{\mathrm{on}} - \theta_{\mathrm{base}}) \in \{-1, 0, 1\}^d$v=sign(θon​−θbase​)∈{−1,0,1}d that determines the direction of the cumulative update. They constrain the gradient $\boldsymbol{g} \in \mathbb{R}^d$g∈Rd of SFT according to $v$v as follows:

$$\boldsymbol{g}_{s} = \mathbb{I}\left(\operatorname{sign}(-\boldsymbol{g}) = \boldsymbol{v}\right) \odot \boldsymbol{g}$$gs​=I(sign(−g)=v)⊙g

where $\odot$⊙ denotes the Hadamard product, $\operatorname{sign}(\cdot)$sign(⋅) is the element-wise sign operator, and $\mathbb{I}(\cdot)$I(⋅) represents the indicator function that retains gradient elements whose signs match $v$v while discarding others. The constrained gradient $\boldsymbol{g}_{s}$gs​ is then passed to the optimizer for gradient descent. By constraining the gradient at each training step, the parameter updates consistently follow the directions identified by the on-policy paradigm. In practice, considering that the regularization terms inherent to the optimizer may affect the imposed constraint, the authors further constrain the update direction after each optimizer step. They refer to this On-Policy direction-constrained SFT as OPSFT.

Motivated by the ability of the on-policy update direction to support strong generalization, the authors further leverage this direction to combine the generalization advantage of on-policy paradigms with the advantages of SFT. First, they perform a small number of GRPO steps to identify an update direction that supports strong generalization, and then apply OPSFT to achieve efficient training. Second, given a model post-trained by an on-policy paradigm, they conduct OPSFT along its original update direction, thereby leveraging newly acquired high-quality trajectories to further improve the model without disrupting the capabilities learned during on-policy training.

Experiment

The experiments evaluate whether the generalization benefits of on-policy post-training can be transferred to supervised fine-tuning by sharing parameter update directions, using Qwen and DeepSeek models on math and code reasoning datasets. Compared with vanilla SFT and constraints based only on update locations, constraining SFT updates along on-policy directions achieves performance comparable to or better than GRPO while reducing training time and transferring reasoning capabilities to in-domain and out-of-domain tasks. The identified direction is reusable across datasets within the same domain but not across domains, and it enables further improvement of already post-trained models without the degradation caused by direct SFT. Ablations further show that later-stage directions provide diminishing gains, sparse updates along the constrained direction remain effective, and the resulting reasoning behavior more closely resembles that of on-policy paradigms.

In the out-of-domain evaluations summarized here, OPSFT achieves the highest average performance where it is reported, slightly above GRPO and ahead of vanilla SFT and the base model. Vanilla SFT shows only mixed gains over the base model, and its average can slightly decline at the larger scale, while GRPO improves the average at both shown scales. The paper links these improvements to using the on-policy update direction during SFT. OPSFT leads the Qwen3-4B mean out-of-domain score, with GRPO close behind and vanilla SFT lower. Gains are concentrated in benchmarks like IFEval and HaluEval, while results on ARC, Hellaswag, Winogrande, and PIQA are smaller or mixed.

Across DeepMath benchmarks, OPSFT tends to outperform SFT and DFT in mean accuracy while requiring less training time. It also reduces training time compared with GRPO, often by more than half, while achieving better or comparable generalization. These trends hold across different model scales and architectures reported in the experiments. On Qwen3-1.7B, OPSFT achieves a mean accuracy of 15.11 in 2.3h, improving over SFT, DFT, and GRPO while using the least training time. On DeepSeek-R1-Distill-LLaMA-8B, OPSFT reaches 26.56 mean accuracy in 8.1h, substantially outperforming DFT at 19.90 in 14.1h. Compared with GRPO on Qwen3-8B, OPSFT cuts training time from 19.3h to 8.9h and raises mean accuracy from 40.31 to 41.67.

For Qwen3-1.7B and Qwen3-4B post-trained by GRPO, OPSFT improves mean benchmark accuracy over both the post-trained baseline and vanilla SFT on math reasoning tasks. Vanilla SFT does not reliably preserve post-trained capabilities and can reduce accuracy, as seen with Qwen3-4B, while OPSFT further improves the same model. Results on code tasks are reported as consistent with this trend. OPSFT achieves higher mean accuracy than vanilla SFT across the reported AIME and HMMT reasoning benchmarks. Vanilla SFT can fall below the post-trained baseline, whereas OPSFT improves over it, with the pattern extending to code tasks.

On code tasks with Qwen3-4B and the Eurus dataset, OPSFT achieves the highest mean accuracy while also having the shortest training time among GRPO, SFT, and OPSFT. It improves over SFT on all reported code benchmarks and slightly exceeds GRPO on average despite training in less than half the time. OPSFT reaches the best mean score and the lowest training time across the compared methods. Compared with SFT, OPSFT delivers consistent accuracy gains on HumanEval+, MBPP+, and LCBv6 with slightly shorter training time.

This ablation examines how BF16 and FP32 parameter precision affect update sparsity and accuracy for SFT and OPSFT. OPSFT produces much sparser updates than vanilla SFT under BF16, yet it still improves mean accuracy over vanilla SFT. With FP32 precision, updates become denser and OPSFT achieves the highest mean accuracy, slightly surpassing GRPO while updating a comparable fraction of parameters. Under BF16, OPSFT updates 0.408% of parameters compared with 2.702% for vanilla SFT, but its mean accuracy is higher and substantially above the base model. Under FP32, vanilla SFT becomes dense with about 90% of parameters updated, while OPSFT updates about 9.45%, similar to GRPO's update proportion. FP32 OPSFT attains the highest mean accuracy and edges out GRPO, suggesting that the constrained update direction remains effective at higher precision.

Across out-of-domain, math reasoning, and code evaluations, OPSFT consistently improves over vanilla SFT and the base model while matching or slightly exceeding GRPO, often with substantially shorter training time and sparser updates. Gains are strongest on benchmarks such as IFEval, HaluEval, and math or code tasks, while effects on standard knowledge benchmarks are smaller or mixed. Precision ablations show OPSFT remains effective under BF16 and FP32, updating far fewer parameters than vanilla SFT while achieving the highest mean accuracy and a comparable update density to GRPO under FP32.