One-sentence Summary

Researchers at Apodex US, Inc. and Princeton University introduce Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent (PMD) that reformulates PMD via Bellman equations into a trajectory-level objective for autoregressive generation with terminal rewards, avoiding state-value estimation and using a smoothed complementary-token-probability mismatch-correction weight, with demonstrated effectiveness on mathematical reasoning benchmarks.

Key Contributions

• Bellman Policy Optimization (BPO) is introduced as a critic-free reinforcement learning method derived from Policy Mirror Descent, using Bellman equations to formulate a trajectory-level objective that avoids estimating intermediate state values.
• The method proves that this trajectory-level objective has the same unique optimal solution as the original PMD objective on states reachable under the rollout policy, and derives a practical token-level loss with a smoothed mismatch-correction weight based on complementary token probabilities.
• On mathematical reasoning benchmarks, BPO achieves a peak average accuracy of 50.5% across AIME 2024 to 2026 with Qwen3-30B-A3B-Base, outperforming GRPO-ClipHigher, GSPO, CISPO, and DPPO by 3.1 to 11.0 percentage points, while ablations on Qwen3-4B-Base show similar performance across smoothing and truncation settings.

Introduction

Reinforcement learning with verifiable rewards (RLVR) is an important approach for improving reasoning in large language models, using outcome-level rewards from task-specific verifiers to guide training. Widely used methods such as GRPO normalize rewards within sampled groups and avoid a value model, but they rely on token-level importance-sampling ratios in PPO-style clipped objectives. Directly applying Policy Mirror Descent would require value estimates at intermediate states, and training a separate value model adds memory and computational costs while risking inaccurate estimates on reasoning tasks. The authors introduce Bellman Policy Optimization (BPO), a critic-free reformulation of Policy Mirror Descent that uses Bellman equations and policy likelihood ratios to convert terminal-reward advantages into a trajectory-level objective, thereby avoiding intermediate value or advantage estimation. The resulting practical loss replaces GRPO's importance-sampling ratio with a smoothed mismatch-correction weight, and BPO outperforms several baselines on mathematical reasoning benchmarks.

Method

The authors formulate Bellman Policy Optimization (BPO) as a critic-free policy optimization method for reinforcement learning with verifiable rewards. The method builds on group-based advantage estimation and policy mirror descent, replacing the importance-sampling ratio used in GRPO with a mismatch-correction weight derived from the PMD optimality condition.

For each prompt $x$x, a group of $G$G responses $\{y^i\}_{i=1}^{G}${yi}i=1G​ is sampled from a rollout policy $\mu$μ. Given a terminal reward $R_i = R(x, y^i)$Ri​=R(x,yi), BPO uses the same group-normalized advantage as GRPO:

$$\hat A^i = \frac{R_i - \operatorname{mean}(\{R_j\}_{j=1}^{G})}{\operatorname{std}(\{R_j\}_{j=1}^{G})}.$$A^i=std({Rj​}j=1G​)Ri​−mean({Rj​}j=1G​)​.

For a token $y_t^i$yti​ in response $y^i$yi, the BPO per-token loss is defined as

$$\mathcal{L}^{\mathrm{BPO}}_{i,t}(\pi) = -\hat A^i M_t^i \min\left\{\mathrm{sg}\left(\omega_t^i\right), C\right\} \log \pi(y_t^i \mid x, y_{<t}^i),$$Li,tBPO​(π)=−A^iMti​min{sg(ωti​),C}logπ(yti​∣x,y<ti​),

where $C$C is a constant cap for stability, and $M_t^i$Mti​ is a clipping mask. The main difference from GRPO is the weight $\omega_t^i$ωti​, which replaces the importance-sampling ratio $r_t^i$rti​. This mismatch-correction weight is defined as

$$\omega_t^i = \frac{1+\epsilon-\mu(y_t^i \mid x, y_{<t}^i)} {1+\epsilon-\pi(y_t^i \mid x, y_{<t}^i)},$$ωti​=1+ϵ−π(yti​∣x,y<ti​)1+ϵ−μ(yti​∣x,y<ti​)​,

with additive smoothing controlled by $\epsilon$ϵ. The mask $M_t^i$Mti​ follows the same clipping rule as GRPO, but uses $\omega_t^i$ωti​ instead of the importance ratio:

$$M_t^i= \begin{cases} 0, & \hat A^i > 0 \text{ and } \omega_t^i > 1+\epsilon_{\mathrm{high}},\\ 0, & \hat A^i < 0 \text{ and } \omega_t^i < 1-\epsilon_{\mathrm{low}},\\ 1, & \text{otherwise}. \end{cases}$$Mti​=⎩⎨⎧​0,0,1,​A^i>0 and ωti​>1+ϵhigh​,A^i<0 and ωti​<1−ϵlow​,otherwise.​

Thus, BPO retains GRPO's per-token weighted log-likelihood form, while replacing the ratio $r_t^i$rti​ by the truncated mismatch-correction weight $\min\{\mathrm{sg}(\omega_t^i), C\}$min{sg(ωti​),C}.

The derivation of BPO starts from advantage-based policy mirror descent. For each state $s_t=(x,y_{<t})$st​=(x,y<t​), the PMD update is formulated as

$$\max_{\pi(\cdot \mid s_t)\in\Delta(\mathcal{V})} \mathbb{E}_{y_t\sim\pi(\cdot \mid s_t)} \left[A^\mu(s_t,y_t)\right] - \frac{1}{\eta} D_{\mathrm{KL}}\left(\pi(\cdot \mid s_t) \,\|\, \mu(\cdot \mid s_t)\right),$$π(⋅∣st​)∈Δ(V)max​Eyt​∼π(⋅∣st​)​[Aμ(st​,yt​)]−η1​DKL​(π(⋅∣st​)∥μ(⋅∣st​)),

where $A^\mu(s_t,y_t)$Aμ(st​,yt​) is the rollout-policy advantage function. The unique optimal policy has the form

$$\pi^{+}(y_t \mid s_t) = \frac{\mu(y_t \mid s_t)\exp\left(\eta A^\mu(s_t,y_t)\right)} {Z_\mu(s_t)}.$$π+(yt​∣st​)=Zμ​(st​)μ(yt​∣st​)exp(ηAμ(st​,yt​))​.

Direct implementation of this update would require estimating advantages at intermediate states, typically through a learned critic. BPO avoids this by deriving a critic-free reformulation.

The authors show that PMD can be equivalently expressed through a squared trajectory-level residual objective. For a prompt-dependent positive weight $\phi(x)$ϕ(x), the critic-free objective is

$$\min_{\pi \in \Pi_\mu} \mathcal{L}(\pi) = \mathbb{E}_{x\sim\mathcal{D},\, y\sim\mathbb{P}_\mu(\cdot \mid x)} \left[\phi(x)\cdot\frac{\delta(x,y;\pi,\mu)^2}{2\eta}\right],$$π∈Πμ​min​L(π)=Ex∼D,y∼Pμ​(⋅∣x)​[ϕ(x)⋅2ηδ(x,y;π,μ)2​],

where the trajectory-level residual is

$$\delta(x,y;\pi,\mu) = \eta\left(R(x,y)-V^\mu(x)\right) - \sum_{t=1}^{|y|} \left( \log\frac{\pi(y_t \mid s_t)}{\mu(y_t \mid s_t)} + D_{\mathrm{KL}}\left(\mu(\cdot \mid s_t) \,\|\, \pi(\cdot \mid s_t)\right) \right).$$δ(x,y;π,μ)=η(R(x,y)−Vμ(x))−t=1∑∣y∣​(logμ(yt​∣st​)π(yt​∣st​)​+DKL​(μ(⋅∣st​)∥π(⋅∣st​))).

This objective depends only on terminal rewards and the initial value function $V^\mu(x)$Vμ(x), not on intermediate state values. The equivalence between this critic-free objective and the original PMD problem is established by rewriting the PMD optimality condition, combining it with Bellman equations over the trajectory, and showing that the PMD solution zeroes the residual $\delta$δ.

The practical BPO loss is obtained from this reformulation in four steps. First, the squared-residual objective is linearized around $\pi=\mu$π=μ. Second, $V^\mu(x)$Vμ(x) is estimated by the group mean reward, and $\phi(x)$ϕ(x) is replaced by the inverse empirical reward standard deviation, yielding the normalized advantage $\hat A^i$A^i. Third, the full reverse KL divergence is approximated by a binary KL divergence, which produces the token-level ratio

$$\nabla\left( \log \pi(y_t^i \mid s_t^i) + D_{\mathrm{KL}}^{\mathrm{bin}}\left(\mu(\cdot \mid s_t^i) \,\|\, \pi(\cdot \mid s_t^i); y_t^i\right) \right) = \frac{1-\mu(y_t^i \mid s_t^i)} {1-\pi(y_t^i \mid s_t^i)} \nabla \log \pi(y_t^i \mid s_t^i).$$∇(logπ(yti​∣sti​)+DKLbin​(μ(⋅∣sti​)∥π(⋅∣sti​);yti​))=1−π(yti​∣sti​)1−μ(yti​∣sti​)​∇logπ(yti​∣sti​).

Additive smoothing is then applied to this ratio to obtain $\omega_t^i$ωti​. Finally, GRPO-style masking and the cap $C$C are applied, yielding the BPO gradient

$$\nabla \mathcal{L}^{\mathrm{BPO}}_{i,t}(\pi) = -\hat A^i M_t^i \min\left\{\omega_t^i, C\right\} \nabla \log \pi(y_t^i \mid s_t^i),$$∇Li,tBPO​(π)=−A^iMti​min{ωti​,C}∇logπ(yti​∣sti​),

which corresponds to the BPO per-token loss in Equation (14).

Experiment

The experiments evaluate BPO for mathematical reasoning using Qwen3-30B-A3B-Base trained on the English subset of DAPO-Math-17k, with all methods sharing the same rollout and optimization settings except for the policy loss. Evaluation on AIME24, AIME25, and AIME26 using Pass@1 estimated through Avg@32 shows that BPO consistently achieves the highest accuracy across all benchmarks and at the end of training. The results indicate that BPO outperforms strong baselines including CISPO and DPPO, validating it as an effective policy optimization objective for improving mathematical reasoning.

BPO achieves the highest average accuracy and the highest score on each AIME benchmark among all compared policy loss methods. The strongest baseline at the best checkpoint is CISPO, and BPO outperforms it by 3.1 percentage points on average. At the end of training, BPO also exceeds the strongest final baseline, DPPO, in average accuracy. BPO leads all methods on AIME24, AIME25, and AIME26, as well as on the overall average. BPO improves over the strongest best-checkpoint baseline by 3.1 percentage points and over the strongest final baseline at the end of training.

The evaluation compares policy loss methods on AIME benchmarks, considering both the best checkpoint and end-of-training performance. BPO achieves the highest average accuracy and the highest score on every AIME benchmark, outperforming the strongest best-checkpoint baseline, CISPO, and the strongest final baseline, DPPO. It leads all compared methods on AIME24, AIME25, AIME26, and the overall average, demonstrating consistently superior results.