One-sentence Summary

Tsinghua University, Zhejiang University, and Meituan researchers propose AgentOPSD, a critic-free recursive turn-level credit assignment method that aggregates token-level teacher-student log-probability gaps into turn-level evidence, recursively updates a Bayesian belief state in log-odds space to reweight sparse outcome supervision into turn-level credit signals, identifies pivotal turns, and achieves $89.1\%$89.1% success on ALFWorld with Qwen2.5-7B, outperforming GRPO and self-distillation baselines.

Key Contributions

• AgentOPSD aggregates token-level teacher–student log-probability gaps into turn-level evidence and recursively updates a Bayesian belief state in log-odds space, converting sparse trajectory-level rewards into turn-level credit signals.
• The method identifies pivotal turns by the marginal revision between consecutive belief states and redistributes the trajectory-level advantage without requiring additional rollouts or a learned critic.
• Experiments on ALFWorld, WebShop, and Search-QA with two Qwen model scales show that AgentOPSD consistently outperforms GRPO and strong self-distillation baselines, reaching 89.1% success on ALFWorld with Qwen2.5-7B, and ablations attribute the gains to turn-boundary aggregation and history-dependent recursive belief updates.

Introduction

In agentic reinforcement learning, large language models must make decisions across many turns in interactive environments, but training signals often arrive only as sparse terminal rewards. Standard group-relative policy optimization (GRPO) broadcasts a uniform advantage across the entire trajectory, failing to differentiate pivotal actions from routine steps, especially over long horizons. On-policy self-distillation (OPSD) offers denser, token-level supervision, yet its signals are misaligned with environment turn boundaries and lack history-dependent context. The authors propose AgentOPSD, which aggregates token-level teacher–student log-probability gaps into turn-level evidence and recursively updates a Bayesian belief state of eventual success in log-odds space. This approach redistributes the trajectory-level advantage without requiring extra rollouts or a learned critic, providing precise turn-level credit assignment grounded in history-dependent belief revision.

Method

The authors introduce AgentOPSD, a method that enriches the GRPO (Group Relative Policy Optimization) advantage with turn-level credit derived from a Bayesian evidence perspective. The overall pipeline augments the standard sequence-level reward signal by incorporating a self-distillation contrast that estimates how much each action turn supports eventual success, followed by a recursive belief update that accounts for the temporal context. The modified advantage is then fed into a PPO-style clipped objective.

The agent interacts over a K-turn episode, starting from task description $x$x and initial observation $o_0$o0​. At each turn $k$k, the policy $\pi_\theta$πθ​ generates an action $a_k$ak​ token by token:

$$a_k = (y_{k,1}, \dots, y_{k,L_k}) \sim \pi_\theta(\cdot \mid s_k),$$ak​=(yk,1​,…,yk,Lk​​)∼πθ​(⋅∣sk​),

where $s_k$sk​ is the interaction history before turn $k$k. The episode $\tau$τ receives a binary outcome reward $R(\tau)$R(τ). GRPO samples $G$G trajectories for a task and computes a normalized sequence-level advantage for each trajectory $i$i:

$$A_{\mathrm{seq}}^{(i)} = \frac{R^{(i)} - \bar{R}}{\widehat{\sigma}_R + \epsilon_0},$$Aseq(i)​=σR​+ϵ0​R(i)−Rˉ​,

with $\bar{R}$Rˉ and $\widehat{\sigma}_R$σR​ being the mean and standard deviation of rewards in the group. This scalar advantage is assigned uniformly to all tokens of the trajectory, leaving turn-level credit unresolved.

To decompose the outcome into per-turn importance, the authors propose a tractable hindsight approximation to the ideal Bayesian turn evidence. The ideal evidence would be the log Bayes factor between the success-conditional and failure-conditional likelihoods of $a_k$ak​:

$$\log \frac{p(a_k \mid s_k, C)}{p(a_k \mid s_k, \neg C)} = \log \frac{p(C \mid s_k, a_k)}{p(C \mid s_k)},$$logp(ak​∣sk​,¬C)p(ak​∣sk​,C)​=logp(C∣sk​)p(C∣sk​,ak​)​,

where $C$C denotes eventual success. Because these conditional distributions are unavailable, a self-distillation contrast is used. For each turn $k$k, the policy is evaluated in two modes: the standard student branch with context $h_{k,t} = (s_k, y_{k,<t})$hk,t​=(sk​,yk,<t​), and a teacher branch conditioned on a retrieved skill description $c^+$c+ that summarizes useful subgoals and action patterns: $h_{k,t}^+ = (s_k, c^+, y_{k,<t})$hk,t+​=(sk​,c+,yk,<t​). The token-level likelihood contrast is

$$\delta_{k,t} = \log \pi_\theta(y_{k,t} \mid h_{k,t}^+) - \log \pi_\theta(y_{k,t} \mid h_{k,t}).$$δk,t​=logπθ​(yk,t​∣hk,t+​)−logπθ​(yk,t​∣hk,t​).

The turn-level evidence $e_k$ek​ is the sum over tokens:

$$e_k = \sum_{t=1}^{L_k} \delta_{k,t} = \log \frac{\pi_\theta(a_k \mid s_k, c^+)}{\pi_\theta(a_k \mid s_k)}.$$ek​=t=1∑Lk​​δk,t​=logπθ​(ak​∣sk​)πθ​(ak​∣sk​,c+)​.

This $e_k$ek​ approximates the Bayesian evidence; its sign indicates whether the action increases or decreases the belief in eventual success.

Raw per-turn evidence alone does not capture whether the action is pivotal or redundant given earlier turns. Hence, a decaying evidence accumulator is maintained. Starting from a clipped prior belief $B_0 = \operatorname{clip}(\bar{R}, \epsilon_0, 1-\epsilon_0)$B0​=clip(Rˉ,ϵ0​,1−ϵ0​) (with $\bar{R}$Rˉ being the group success rate), the log-odds are updated recursively:

$$c_0 = 0,\quad c_k = \gamma c_{k-1} + e_k,\quad \ell_k = \operatorname{logit}(B_0) + c_k,$$c0​=0,ck​=γck−1​+ek​,ℓk​=logit(B0​)+ck​,

where $\gamma \in (0,1]$γ∈(0,1] is a decay factor that down-weights older turns, and $B_k = \sigma(\ell_k)$Bk​=σ(ℓk​). The importance of turn $k$k is its marginal revision:

$$\Delta B_k = B_k - B_{k-1} = \sigma(\ell_k) - \sigma(\ell_{k-1}).$$ΔBk​=Bk​−Bk−1​=σ(ℓk​)−σ(ℓk−1​).

This revision is then aligned with the sequence-level outcome signal to obtain a signed credit:

$$q_k = \operatorname{sign}(A_{\mathrm{seq}}) \, \Delta B_k.$$qk​=sign(Aseq​)ΔBk​.

The magnitude $|\Delta B_k|$∣ΔBk​∣ reflects how much the belief shifts, while the sign preserves the direction of the verifier’s outcome signal.

To prevent large swings and to make the credit relative within a trajectory, the raw $q_k$qk​ values are standardized inside each episode. A bounded multiplier is then applied:

$$\begin{aligned} \mu_q^{(i)} &= \frac{1}{K_i}\sum_{j=1}^{K_i} q_j^{(i)}, \quad \sigma_q^{(i)} = \sqrt{\frac{1}{K_i}\sum_{j=1}^{K_i} (q_j^{(i)} - \mu_q^{(i)})^2},\\[4pt] z_k^{(i)} &= \frac{q_k^{(i)} - \mu_q^{(i)}}{\sigma_q^{(i)} + \epsilon_0}, \quad w_k^{(i)} = \operatorname{clip}\!\Big(1 + b\,z_k^{(i)},\, 1-b,\, 1+b\Big). \end{aligned}$$μq(i)​zk(i)​​=Ki​1​j=1∑Ki​​qj(i)​,σq(i)​=Ki​1​j=1∑Ki​​(qj(i)​−μq(i)​)2​,=σq(i)​+ϵ0​qk(i)​−μq(i)​​,wk(i)​=clip(1+bzk(i)​,1−b,1+b).​

The reshaped turn-level advantage is a convex combination of the original sequence-level advantage and the weighted version:

$$\widetilde{A}_k^{(i)} = A_{\mathrm{seq}}^{(i)} \big[ (1-\lambda) + \lambda\, w_k^{(i)} \big],$$Ak(i)​=Aseq(i)​[(1−λ)+λwk(i)​],

where $b \in (0,1)$b∈(0,1) controls the clipping range and $\lambda \in [0,1]$λ∈[0,1] sets the strength of the reshaping. Every token in turn $k$k inherits $\widetilde{A}_k^{(i)}$Ak(i)​.

The training objective is the standard PPO clipped loss, but with the reshaped advantage:

$$\mathcal{L}_{\text{AgentOPSD}}(\theta) = -\frac{1}{G} \sum_{i=1}^{G} \frac{1}{\sum_t M_{i,t}} \sum_t M_{i,t} \min\!\Big(r_{i,t} \widetilde{A}_{\kappa_i(t)}^{(i)},\; \operatorname{clip}(r_{i,t}, 1-\varepsilon, 1+\varepsilon) \widetilde{A}_{\kappa_i(t)}^{(i)}\Big) + \beta \mathcal{L}_{\mathrm{KL}},$$LAgentOPSD​(θ)=−G1​i=1∑G​∑t​Mi,t​1​t∑​Mi,t​min(ri,t​Aκi​(t)(i)​,clip(ri,t​,1−ε,1+ε)Aκi​(t)(i)​)+βLKL​,

where $r_{i,t}$ri,t​ is the importance ratio against the rollout policy, $\kappa_i(t)$κi​(t) maps each token to its turn, $M_{i,t}$Mi,t​ is a mask for valid response tokens, and the KL penalty term is added with coefficient $\beta$β. No separate distillation loss is required; the self-teacher evidence influences learning solely through the reshaped advantage.

Experiment

The evaluation spans ALFWorld, Search-QA, and WebShop environments with Qwen2.5-3B/7B models, comparing AgentOPSD against training-free, group-relative RL, and self-distillation baselines all under identical skill access. The main results show that gains stem from turn-level credit construction via belief revision rather than privileged skills, and the advantage grows with trajectory length, where uniform credit degrades faster. Mechanism ablations confirm that turn-level belief tracking, recursive state update, signed outcome alignment, and prior anchoring each contribute, and hyperparameter sweeps indicate the method is robust except to the credit reshaping weight. Overall, the approach localizes pivotal decisions in long-horizon tasks by accumulating a teacher-student gap into a belief state and assigning credit according to belief revision.

GRPO achieves the best overall performance across ALFWorld, Search-QA, and WebShop, outperforming skill‑augmented Skill‑GRPO and vanilla baselines. Removing skills at inference (OPSD) collapses on Search‑QA and WebShop, while the belief prior anchor and the reshaping weight λ are critical for stable credit assignment in long‑horizon tasks. GRPO consistently outperforms Skill‑GRPO, yielding higher success rates on ALFWorld, higher accuracy on Search‑QA, and a higher score on WebShop. Without skills at inference, OPSD fails on Search‑QA and WebShop, with near‑zero scores, but retains some ALFWorld capability. The empirical prior B0 is essential: removing it reduces the success rate to 78.9, indicating that anchoring stabilizes belief revision for long‑horizon tasks. Among hyperparameters, the reshaping weight λ is the most influential; reducing λ from 0.5 degrades performance, while evidence decay and policy clipping have little effect.

Removing any single component from AgentOPSD reduces the success rate from the full 89.1%, with the signed direction and state-prior anchor causing the largest drops. Turn-level tracking and recursive state revision provide smaller but consistent gains, confirming that each mechanism contributes to the method's performance. Dropping the state-prior anchor lowers success to 78.9%, the largest decline, showing its role in stabilizing early belief updates. Removing the signed direction and using only magnitude reduces success to 80.5%, indicating that outcome-aligned sign is critical. Replacing turn-level belief tracking with per-token accumulation reduces success to 85.9%, and replacing recursive revision with the raw local gap drops it to 82.8%.

GRPO consistently outperforms Skill-GRPO and vanilla baselines across ALFWorld, Search-QA, and WebShop, while removing skills at inference (OPSD) results in near-zero scores on Search-QA and WebShop, indicating that skill guidance and the belief prior anchor are crucial for stable credit assignment in long-horizon tasks. The empirical B0 anchor proves essential: dropping it reduces success to 78.9%, and the reshaping weight λ is the most influential hyperparameter, with reductions from 0.5 harming performance. Ablation of AgentOPSD confirms that the state-prior anchor and outcome-aligned signed direction cause the largest drops from the full 89.1% success rate, while turn-level tracking and recursive revision contribute smaller but consistent gains, validating the role of each mechanism.