One-sentence Summary

Researchers propose a framework for learning planning-based reasoning via Direct Preference Optimization on trajectories ranked by synthesized process rewards, which circumvents the high latency of planning-based search and the cost of human annotation, enabling a 7B model to surpass GPT-3.5-Turbo on challenging logical reasoning benchmarks.

Key Contributions

• The paper introduces a framework that learns planning-based reasoning in large language models through Direct Preference Optimization on trajectories ranked by synthesized process rewards, reducing online search latency and human process supervision cost.
• The method estimates process rewards by sampling intermediate reasoning states from seed trajectories, collecting retry completions, and using successful outcome counts as expected returns to train a process reward model for annotating trajectory preferences.
• Experiments on logical and mathematical reasoning benchmarks show improvements over robust baselines, and the reported 7B model surpasses strong counterparts such as GPT-3.5-Turbo on logical reasoning while producing more reliable and concise rationales.

Introduction

Natural language reasoning is central to AI applications such as planning and decision making, and large language models can generate step-by-step rationales, but they still produce misleading reasoning chains in complex tasks. Prior approaches rely on human step-level supervision or online search methods such as reasoning-as-planning with Monte Carlo Tree Search, yet human annotations are costly and online search can introduce large latency and computational overhead. The authors leverage offline simulation, sampling intermediate reasoning states from collected solutions and estimating process rewards from how often follow-up reasoning reaches correct outcomes. They use these estimated rewards to train a process reward model and optimize the language model with direct preference optimization, improving logical and mathematical reasoning while reducing reliance on human annotations.

Dataset

• The paper focuses on two reasoning domains: logical reasoning and mathematical reasoning.

• Logical reasoning uses ReClor and LogiQA-v2 as evaluation benchmarks. Both are multiple-choice question answering datasets and are described as challenging and widely used. Their exact statistics are referenced in Table 3, but not given in the provided text.

• Mathematical reasoning uses the test sets of GSM8K and MATH for evaluation. No training split or mixture ratio is specified in the provided text.

• For reward and preference data construction, the authors build data from complete reasoning trajectories, represented as sequences of states and actions. Each trajectory is scored by a process reward model that evaluates intermediate steps.

• The trajectory-level reward is constructed by accumulating or multiplying process rewards across step-level prefixes of the trajectory. This creates a single score for a complete reasoning path from multiple step-level signals.

• The data construction includes a filtering rule controlled by a hyperparameter, C. This parameter sets the minimum number of successful simulations required before the authors consider a state capable of leading to a correct reasoning process. This reduces the influence of potentially hallucinated rationales from the original language models.

• The resulting reward and preference signals are used to optimize the policy model with Direct Preference Optimization instead of PPO.

Method

The authors model natural language reasoning as a Markov Decision Process with an action-state trajectory. A complete reasoning process is written as

where $a_t$at​ is the action generated at step $t$t, $s_{t+1}$st+1​ is the state observed afterward, and $s_t$st​ may be treated as the generated reasoning prefix. In the LLM setting, both actions and states are generated by the same policy model $\pi_\theta$πθ​, so that

where $c_t=(s_0,a_0,\dots,s_t)$ct​=(s0​,a0​,…,st​) is the history trace. The policy is optimized to maximize the expected outcome reward. For tasks with final labels, the terminal reward is defined as

The corresponding training objective is

where $x$x denotes the prompt and question, $y$y is the ground-truth answer, and $\tau'$τ′ is the generated reasoning trajectory.

To address the limitation that an LLM may reach the correct final answer through incorrect or unreliable intermediate reasoning, the authors introduce process supervision through offline simulation. Instead of manually annotating every intermediate step, they estimate the expected value of an intermediate reasoning state by exploring from that point to the terminal answer. For a given input and an intermediate reasoning prefix, they sample $K$K completions from either an action $a_t$at​ or a state $s_t$st​, complete each trajectory, and count how often the completed trajectory leads to the correct final answer. For an action-level prefix $\tau_{t,a}$τt,a​, the estimated process value is formulated as

where $\tau^{k|\tau_{t,a}}$τk∣τt,a​ denotes the $k$k-th completed trajectory starting from the prefix $\tau_{t,a}$τt,a​. The intuition is that an intermediate state or action that frequently leads to correct answers is more likely to contain valid reasoning, useful facts, or decisive evidence.

The simulated process values are then used to train a synthesized process reward model. The motivation is that directly relying on repeated simulation would be expensive and noisy, since the quality of the simulated completions depends on the underlying policy model. A learned process reward model can approximate these simulation-derived values more efficiently and robustly. The reward modeling dataset is constructed as

where $\tau_{j,a}^{(i)}$τj,a(i)​ is an intermediate trajectory prefix for sample $i$i, and $r_j^{(i)}$rj(i)​ is the corresponding simulated expected value. The process reward model $f_{\mathrm{prm}}:\mathcal{X}\times\mathcal{T}\rightarrow\mathbb{R}^{K}$fprm​:X×T→RK assigns a reward estimate to each incomplete reasoning prefix. It is trained as a classification-style regression problem with

where $p_r$pr​ denotes the target probability associated with the estimated reward value.

After the process reward model is trained, complete reasoning trajectories are scored by aggregating intermediate rewards across steps. For a full trajectory $\tau=\langle s_0,a_0,\dots,s_T,a_T\rangle$τ=⟨s0​,a0​,…,sT​,aT​⟩, the trajectory-level reward is defined as

where $\tau_{t,a}=\langle s_0,a_0,\dots,s_t,a_t\rangle$τt,a​=⟨s0​,a0​,…,st​,at​⟩ and $\tau_{t,s}=\langle s_0,a_0,\dots,s_t\rangle$τt,s​=⟨s0​,a0​,…,st​⟩. The hyperparameter $C$C controls the minimum level of simulation confidence required before a prefix is considered to lead reliably toward a correct answer. This threshold helps reduce the influence of hallucinated or fragile intermediate reasoning generated by the original LLM.

The resulting trajectory-level reward is then used to construct preference data for Direct Preference Optimization. A baseline outcome-supervised preference dataset is first formed by sampling multiple trajectories from the policy model. For each input, a trajectory reaching the correct answer is paired as the chosen solution $\tau_w$τw​, while a trajectory with an incorrect prediction is paired as the rejected solution $\tau_l$τl​, yielding

The policy is then optimized with the standard DPO loss

where $\pi_{\mathrm{ref}}$πref​ is the reference policy and $\beta$β controls the KL-divergence-like constraint.

To incorporate process supervision, the authors further construct preferences among trajectories that both produce correct final answers but differ in reasoning quality. If one correct trajectory has a higher accumulated process reward than another by more than a margin $\sigma$σ, it is selected as the chosen response:

Here $\tau_a$τa​ and $\tau_b$τb​ both induce the correct answer $y^{(i)}$y(i), but $\tau_a$τa​ is preferred because its intermediate reasoning receives stronger process-level support. The final optimization uses the combined preference dataset

This design allows the policy to learn not only from final outcome correctness, but also from finer-grained signals about whether intermediate reasoning steps are reliable, informative, and likely to lead to correct conclusions.

Experiment

The experiments evaluate logical reasoning on ReClor and LogiQA-v2 and mathematical reasoning on GSM8K and MATH, comparing pDPO against foundational LLMs, supervised fine-tuning, rejection-sampling methods, DPO, and RL approaches. Overall, pDPO consistently improves in-domain and out-of-domain logical reasoning over outcome-only DPO, reduces training cost relative to PPO and GRPO, and strengthens mathematical reasoning when paired with stronger base models. Additional analyses show that pDPO benefits from sparse outcome annotations, produces more reasonable and concise rationales under GPT-4 evaluation, and yields clearer, less redundant solution traces. These findings indicate that synthesized process supervision enhances reasoning quality and data efficiency, though performance remains sensitive to base-model capability and annotation scale.

Outcome-based DPO substantially improves over the SFT baseline and other non-iterative baselines, with gains in both in-domain and out-of-domain logical reasoning. Adding synthesized process supervision further improves over vanilla DPO and can outperform PPO while requiring less training time, while GRPO gains over PPO by using group-relative reward baselines. DPO serves as a strong baseline and outperforms rejection-sampling baselines on LogiQA-v2. Process-supervised DPO improves both benchmark performance and rationale quality, producing more reasonable, concise, and logically consistent explanations. DPO-based methods achieve competitive or better results than PPO and GRPO with much lower training cost.

The mathematical reasoning results show that process-reward-guided DPO variants improve smaller models on GSM8K relative to the larger Gemma-7B-Instruct baseline, with pDPO generally strongest. Gains on MATH are more limited for Gemma-2B, while pDPO shows a clearer advantage over DPO when paired with DeepSeekMath-7B-Instruct. Synthesized process rewards can let 2B Gemma models beat the 7B Gemma-Instruct reference on GSM8K. pDPO yields the strongest GSM8K result among Gemma-2B variants and a MATH gain over DPO with DeepSeekMath. The MATH dataset is more sensitive to base-model quality, with limited improvement for Gemma-2B.

The experiments evaluate logical and mathematical reasoning by comparing outcome-based and process-supervised DPO variants against SFT, rejection-sampling, PPO, GRPO, and model-scale baselines on benchmarks including LogiQA-v2, GSM8K, and MATH. The results show that outcome-based DPO is a strong low-cost baseline, and adding synthesized process supervision further improves reasoning performance and explanation quality, often matching or exceeding PPO and GRPO with less training cost. On math tasks, process-reward-guided DPO variants enable smaller models to surpass larger references on GSM8K, with pDPO generally strongest, although improvements on MATH are more sensitive to base-model quality.