One-sentence Summary

Researchers from Shanghai Artificial Intelligence Laboratory, Soochow University, Zhejiang University, and Fudan University propose Iterative Value Refinement, an evolutionary framework that narrows the distributional gap in value-guided decoding by combining Value Exploration with Iterative Self-Refinement to produce more accurate value functions for efficient alignment in text summarization, multi-turn dialogue, and instruction following without retraining base models.

Key Contributions

• Iterative Value Refinement (IVR) narrows the distributional gap in value-guided decoding by training value functions beyond fixed trajectories sampled from the base policy.

• IVR combines Value Exploration, which broadens the value function’s training signal across the output space, with Iterative Self-Refinement, which uses an improved value function to generate higher-quality data for later iterations.

• IVR improves alignment on text summarization, multi-turn dialogue, and instruction following while avoiding full model retraining and reducing computational costs relative to actor-critic reinforcement learning approaches.

Introduction

Reinforcement Learning from Human Feedback (RLHF) is important for aligning large language models with human preferences and task requirements, but traditional approaches such as Proximal Policy Optimization can be expensive and unstable because they repeatedly update both the policy and value function. Guided decoding is a practical alternative because it can steer generation without retraining the base model, and value guided decoding is promising because it uses a learned value function to choose partial outputs that lead to higher reward. Prior value guided methods are limited because they typically train the value function on trajectories sampled only from the base policy, creating a distributional gap that hides the better sequences an improved policy would actually produce. The authors introduce Iterative Value Refinement, or IVR, a self bootstrapping framework that uses Value Exploration to generate broader training signals and Iterative Self Refinement to progressively improve the value function while avoiding full policy retraining.

Method

The method centers on learning a value function that can guide autoregressive decoding toward higher-reward completions while progressively reducing the distributional gap between the data used for training and the states visited by the optimal policy. The authors leverage a token-level Markov decision process formulation, where the model must choose a next token based on the prompt $\mathbf{x}$x and the generated prefix $y_{\leq t}$y≤t​, and the ultimate goal is to generate a full response $\mathbf{y}$y with high reward. The optimal decoding policy is therefore characterized by the maximum expected reward achievable from each prefix state.

The core difficulty is that the optimal value function is not directly available. It represents the expected reward of continuing optimally from a given prefix:

Because $\pi^{*}$π∗ is unknown, existing approaches often approximate the required trajectories using a fixed base policy $\pi_{\text{base}}$πbase​. This creates a distributional mismatch: the value function is trained primarily on states and continuations that are typical under the base policy, but guided decoding must evaluate states that may arise only under a stronger, reward-aligned policy. As a result, the learned value function can be poorly calibrated on the very trajectories that matter most for decoding-time alignment.

To address this limitation, the framework introduces Iterative Value Refinement. This design combines two modules. The first module, Value Exploration, broadens the training signal by sampling multiple candidate responses for each prompt rather than relying on a single rollout. The second module, Iterative Self-Refinement, uses the improved value function to steer future data generation toward higher-quality trajectories. Together, these modules create a closed-loop procedure: the value function improves the sampled data, and the improved data further refines the value function.

At inference time, the learned value function can guide decoding through several search strategies. A practical variant is value-guided top-$k$k sampling. Computing value estimates for every possible next token is expensive, so the system only evaluates tokens in the top-$k$k set under the base policy. The sampling distribution is reweighted by the estimated value:

This approach preserves most of the base model’s language modeling behavior while biasing generation toward higher-value continuations. In some implementations, the perturbation can be applied at the block level rather than token by token, trading fine-grained control for faster inference.

A more deliberate decoding strategy is value-guided blockwise beam search. At each decoding step, for each of $B$B active prefix candidates, the system samples $B$B continuation blocks. All $B^2$B2 extended sequences are ranked according to the value estimate:

and the top $B$B are retained for the next step. After generation finishes, the final sequence with the highest value score is returned. This procedure uses the value function directly as a sequence evaluator without requiring a full KL-regularized training update for the language model at every step.

The training process is structured as an iterative refinement loop. In the initial phase, the current policy, often the base policy, samples multiple responses for each prompt. A reward model $\mathbf{R}$R then scores the completed responses. Each completed response provides a target value for all prefixes along that trajectory:

A parameterized value function $V_{\theta}(\mathbf{x}, y_{\leq t})$Vθ​(x,y≤t​) is then trained to regress toward these target values using an $L_2$L2​ objective:

This is the Value Exploration stage. Multi-trajectory sampling exposes the value function to a wider set of prefixes and completions, producing a more robust estimate than training on a single rollout.

After the value function is updated, the framework moves to Iterative Self-Refinement. Instead of continuing to sample only from the base policy, the method constructs a value-guided policy:

Responses are sampled from this policy:

Because this policy is already influenced by the current value function, its trajectories are more likely to visit high-reward regions of the output space. Each sampled trajectory is again scored by the reward model, and the resulting target values are used to retrain $V_{\theta}$Vθ​:

The process can then be repeated. Each iteration produces data that is more aligned with high-reward behavior, and each updated value function can guide the next round of sampling and decoding. This progressive loop is the central design principle of the method.

The authors frame the benefit of this procedure through the notion of visitation coverage. A policy’s visitation measure describes how often different state-action pairs are encountered when following that policy. For offline or approximate alignment, the error in value estimation is closely tied to how well the training data covers states that the optimal policy would visit. If training data is collected only from the base policy, important high-reward states may be underrepresented, causing the value function to misjudge promising trajectories.

Value Exploration addresses this by broadening the support of the data-collecting distribution through multiple stochastic rollouts. Iterative Self-Refinement addresses it by gradually shifting the data-collecting policy toward a value-guided policy, whose visitation pattern is already biased toward higher-reward states. As a result, the gap between the data distribution and the optimal policy’s visitation distribution is progressively reduced, improving the accuracy of the learned value function.

Compared with conventional online RLHF, this framework does not continuously retrain the entire policy model on newly collected preference data. Instead, it focuses the learning process on the value function and uses value-guided decoding as the alignment mechanism at inference time. This makes the overall pipeline more computationally targeted, since the language model itself does not require repeated policy-level optimization. The final system therefore relies on a trained value function to evaluate partial sequences, a reward model to construct training targets, and guided decoding strategies to convert the learned value estimates into improved responses.

Experiment

The authors evaluate IVR on summarization, multi-turn dialogue, and instruction following, comparing it against decoding-time alignment baselines and preference-tuning methods using reward and policy deviation trade offs, value-guided beam search rewards, and judge-based win rates. Across tasks, IVR consistently provides stronger alignment than the baselines and remains stable under larger policy deviations, while ablations show that value exploration with multiple trajectories and iterative refinement are important for performance. Further analyses indicate that the learned value functions transfer across model sizes and can improve safety against jailbreak attacks, suggesting broad practical usefulness.

On the multi-turn dialogue evaluation, IVR has the highest win-rate against the base policy among the compared methods. It also improves computational efficiency relative to the training-based baselines, while inference-only methods generally require fewer FLOPs but achieve lower win-rates. DPO is the strongest baseline but uses substantially more estimated FLOPs than IVR. IVR obtains the top judged preference win-rate among the compared methods. IVR uses fewer estimated FLOPs than DPO and IPO while still outperforming them. DPO is competitive but more compute intensive, while IPO trails the other training-based method.

Value-guided blockwise beam search improves instruction-following performance over the base policy for all tested model sizes. The improvement remains strong when transferring a value function trained on the 3B model to smaller and larger models, indicating broad applicability across scales. Value-guided search yields consistent gains for 1B, 3B, and 8B models. A single value function trained on 3B data transfers effectively to other model sizes. The largest relative improvement is observed for the 8B model.

Applying value-guided blockwise beam search improves the safety rates of both aligned and unaligned Llama-3 models against jailbreak-style harmful prompts. The aligned model already has strong safety performance, while the unaligned model shows a larger relative gain. In both cases, the unaligned value function provides stronger safety benefits than the aligned value function. Value-guided blockwise beam search raises safety for an already safety strong aligned Llama-3 model. The same approach gives the unaligned model a much larger safety increase. The unaligned value function consistently outperforms the aligned value function in these safety tests.

The experiments evaluate iterative value-guided refinement against dialogue policies, instruction-following performance across model scales, and safety on harmful prompts. In multi-turn dialogue preference, IVR achieves the best win-rate while using fewer estimated FLOPs than strong training baselines such as DPO and IPO. Value-guided blockwise beam search consistently improves instruction-following across 1B, 3B, and 8B models, and a single value function trained on the 3B model transfers effectively to other sizes. The same method also improves safety for aligned and unaligned Llama-3 models, with larger gains on the unaligned model and stronger safety benefits from the unaligned value function.