One-sentence Summary

Researchers from Shanghai AI Laboratory, Fudan University, and other institutions propose Proxy-guided Update Signal Transfer (PUST), a modular post-training paradigm that decouples update-signal exploration from distribution alignment by employing a lightweight proxy model to discover high-reward behaviors, extract the relative improvement signal between its initial and optimized states, and transfer that directional update to the primary LLM, thereby enabling asynchronous, reusable, cost-efficient weak-to-strong enhancement across math and code tasks.

Key Contributions

• Post-training is reinterpreted as distinct policy exploration and policy alignment stages, and the tight coupling in existing methods is shown to inflate exploration costs and limit cross-model adaptability.
• The Proxy-guided Update Signal Transfer (PUST) framework decouples update-signal exploration from alignment: a lightweight proxy model discovers high-reward behaviors, and the relative improvement signal is extracted and transferred as a directional update to the primary model.
• Systematic evaluations on Qwen3-family models across math and code domains demonstrate that update signals from substantially weaker proxies robustly improve stronger primary models with adjustable transfer intensity. The signals can be asynchronously cached and reused, yielding a modular, cost-efficient post-training paradigm.

Introduction

Post-training large language models (LLMs) for domain-specific tasks is critical but increasingly expensive as models grow and need to handle multiple domains. Existing approaches like PPO and GRPO rely on sequential on-policy rollouts, making cross-domain adaptation costly. While distribution matching methods such as OPD enable parallelization by aligning a student with an expert teacher, they still tightly couple exploration and alignment within the same model, preventing the asynchronous reuse and cross-model transfer of discovered improvement signals. The authors propose Proxy-guided Update Signal Transfer (PUST), a framework that decouples these stages by using a lightweight proxy model for low-cost exploration, extracting the relative improvement signal between the proxy’s initial and optimized states, and transferring that signal to guide updates in the primary model. This enables reusable, scalable, and cross-model post-training, including weak-to-strong transfer, and transforms the process into a modular pipeline of signal exploration, extraction, and alignment.

Method

The authors begin by analyzing the fundamental differences between reward optimization and distribution matching in post-training. Reward optimization actively seeks high-reward behaviors, while distribution matching aligns a model to a predetermined distribution. To evaluate their training efficiencies, the authors compare GRPO and OPD using Qwen3-1.7B.

The comparative analysis reveals that while distribution matching variants achieve rapid convergence, they lack intrinsic reward awareness and indiscriminately align the student distribution with the teacher. This indicates that genuine directional signals for policy improvement stem from active exploration. To break the dependency on a pre-existing strong target policy, the authors reformulate post-training around the concept of the update signal. They utilize a proxy model to explore reward-induced update signals, which are then extracted and transferred to the primary model, decoupling the process into three stages: proxy exploration, update-signal extraction, and signal transfer.

In the first stage, Proxy Exploration, the authors deploy a lightweight proxy model to serve as an efficient testbed for trial-and-error exploration. Instead of directly updating the primary model, which incurs prohibitive sampling costs, they optimize the proxy policy using standard reward optimization to maximize a given reward function. This decoupled design allows for highly efficient policy exploration and enables the parallel deployment of multiple proxy models to explore diverse high-reward behaviors.

In the second stage, Update-Signal Extraction, the authors extract the relative improvement signal rather than performing absolute distillation. For each state and token, the proxy-induced token-level update direction is defined as: $\Delta_{\phi} (a \mid s_t) = \log \pi_{\phi}^+ (a \mid s_t) - \log \pi_{\phi} (a \mid s_t) = \log \frac{\pi_{\phi}^+ (a \mid s_t)}{\pi_{\phi} (a \mid s_t)}$Δϕ​(a∣st​)=logπϕ+​(a∣st​)−logπϕ​(a∣st​)=logπϕ​(a∣st​)πϕ+​(a∣st​)​ This relative signal encapsulates the reward-induced direction, indicating whether the proxy's tendency to select a token was encouraged or suppressed. This generalized directional improvement is highly transferable regardless of the proxy's absolute capability and can be cached for reuse.

In the final stage, Signal Transfer, the authors apply the extracted relative improvements to the primary model. To prevent severe over-updating from repeatedly transferring a static signal, they introduce Anchor-Based Update Calibration. They define the primary-anchor log-ratio to dynamically measure the update already absorbed by the primary model: $\Delta_{\theta} (a \mid s_t) = \log \frac{\pi_{\theta} (a \mid s_t)}{\pi_{\mathrm{ref}} (a \mid s_t)}$Δθ​(a∣st​)=logπref​(a∣st​)πθ​(a∣st​)​ To prevent over-updating, this term is used as a penalty to define the token-level transferred utility: $r_{\lambda} (a \mid s_t) = \Delta_{\phi} (a \mid s_t) - \lambda \Delta_{\theta} (a \mid s_t)$rλ​(a∣st​)=Δϕ​(a∣st​)−λΔθ​(a∣st​) where the calibration coefficient dictates the alignment conservatism.

Finally, the authors formulate the Signal-Guided Objective. Given a set of token-level states, they optimize the primary policy by maximizing the expected transferred utility, which is equivalent to minimizing the following objective: $\mathcal{L}_{\text{proxy}} (\theta) = - \mathbb{E}_{s_t \sim \mathcal{D}} \left[ \sum_{a \in \mathcal{V}} \pi_{\theta} (a \mid s_t) \left( \log \frac{\pi_{\phi}^+ (a \mid s_t)}{\pi_{\phi} (a \mid s_t)} - \lambda \log \frac{\pi_{\theta} (a \mid s_t)}{\pi_{\text{ref}} (a \mid s_t)} \right) \right]$Lproxy​(θ)=−Est​∼D​[∑a∈V​πθ​(a∣st​)(logπϕ​(a∣st​)πϕ+​(a∣st​)​−λlogπref​(a∣st​)πθ​(a∣st​)​)] The proxy policies and the anchor policy remain strictly frozen, with gradients applied exclusively to the primary policy. This mechanism acts as an efficient distribution-matching algorithm that aligns to a dynamic target constructed by transferring the proxy-explored update signal onto the primary model anchor space.

Experiment

The experiments use GRPO-trained proxy models in Math and Code domains to transfer update signals to primary models, evaluating transfer effectiveness, reusability, transitivity, calibration sensitivity, and exploration quality. Proxy signals consistently improve primary models, often surpassing the proxy's own gains, and can be reused across different models or transitively transferred with minor drift; optimal calibration coefficients are signal-dependent and typically exceed one. Although proxy exploration is slightly weaker than direct exploration, larger proxies and longer training narrow the gap, and the decoupling of signal exploration from the primary model enables flexible, cost-efficient post-training.

Small proxy models trained with reinforcement learning on a math dataset show large benchmark improvements, with the 4B model achieving a 30.5-point average gain. When these update signals are transferred to an 8B primary model, they produce consistent gains across all evaluated benchmarks, sometimes exceeding the proxy’s own improvements. This demonstrates that proxy-explored signals can be effectively reused, decoupling exploration from the primary model. The Qwen3-4B proxy model achieved a 30.5-point average gain across math benchmarks after reinforcement learning, outperforming the 1.7B proxy’s 16.6-point gain. Update signals transferred from these proxy models to the Qwen3-8B primary model led to consistent improvements, exceeding the proxy’s own gains on AIME2024 and HMMT25(NOV).

Transferring update signals from a 4B code proxy model to an 8B primary model via PUST improves average code benchmark performance by +4.6 points, exceeding the proxy's own training gain of +2.9. The largest benefit appears on the LCB benchmark, where the primary model's gain (+8.3) dramatically outstrips the proxy's gain (+1.0), indicating that transferred signals can be amplified by a more capable model. The 8B primary model with PUST achieves a 60.5 average score, a +4.6 improvement over its base, while the 4B proxy's own RL training yields only a +2.9 average gain. On LCB, the primary model's score jumps from 16.7 to 25.0 (+8.3) with PUST, compared to the proxy's modest increase from 18.0 to 19.0 (+1.0). PUST enables the 8B model to outperform the 4B proxy on all benchmarks, with the largest relative improvement on LCB.

A proxy model trained with GRPO can capture effective update signals that transfer to various primary models in far fewer steps. The transferred signals yield substantial gains across math benchmarks, with the largest primary model achieving the highest absolute scores even though the gain is nearly identical to the proxy’s own improvement. Reusing the proxy’s update signal requires only 50 training steps, a tenfold reduction compared to the 500 steps used to explore the signal originally. When transferred to a larger primary model, the same update signal produces a higher average benchmark score than the proxy model itself, showing that the signal scales effectively with model size.

Proxy-explored update signals can be transferred to a primary model, substantially improving math benchmark scores. Direct transfer from a larger proxy model yields the highest gains, while transitive transfer through an intermediate model still provides strong improvement but with a slight reduction compared to the direct path. The smallest proxy model produces the weakest gains, indicating that proxy model scale influences the quality of the transferred signals. Direct transfer from the 4B proxy model achieves the largest average improvement over the base Qwen3-8B. The transitive chain 4B→1.7B→8B delivers strong gains but underperforms the direct 4B→8B transfer, suggesting signal drift across multiple hops. The 1.7B proxy transfer yields the smallest gains, illustrating that larger proxy models produce more effective update signals.

Training small proxy models with reinforcement learning on math and code tasks yields effective update signals that, when transferred to larger primary models via PUST, generate consistent benchmark improvements, often surpassing the proxy’s own gains. The transferred signals are efficiently reusable, decoupling exploration from the primary model and requiring far fewer training steps while scaling with model size. Larger proxy models produce stronger signals, and direct transfer outperforms transitive chains, indicating that both proxy quality and transfer path influence the overall benefit.