One-sentence Summary

Researchers from Microsoft Research, Tsinghua University, and Peking University propose Experiential Learning (EL), which transforms an LLM-as-a-Judge into an LLM-as-a-Coach that distills rich textual assessment into transferable experiential knowledge, enabling on-policy context distillation to provide dense, high-bandwidth feedback that outperforms rubric-based reinforcement learning on open-ended tasks, improves generalization, and mitigates reward hacking.

Key Contributions

• Experiential Learning repurposes an LLM-as-a-Judge into an LLM-as-a-Coach, which distills rubric-based assessments into transferable experiential knowledge. This knowledge is internalized through on-policy context distillation, providing a higher-bandwidth feedback signal that preserves fine-grained preferences among high-quality responses.
• Across two policy families and with feedback from the policy itself or a proprietary model, EL consistently outperforms rubric-based reinforcement learning on held-out and unseen open-ended tasks, while also showing stronger generalization beyond the training distribution and mitigating reward hacking.
• Distilled experiential knowledge is more effective than raw critiques or rubrics, and iterative teacher updates and on-policy distillation over general-domain prompts improve task performance and prevent forgetting of out-of-distribution capabilities.

Introduction

The authors tackle the challenge of applying reinforcement learning to open-ended language tasks where quality is multidimensional and no single correct answer exists. Prior work relies on LLM-as-a-Judge evaluations that produce rich textual feedback alongside a scalar reward, but standard RL discards the textual analysis and uses only the final score, creating an information bottleneck that makes otherwise distinct high-quality responses indistinguishable. The authors introduce Experiential Learning (EL), which repurposes the evaluator into an LLM-as-a-Coach that extracts reusable, transferable guidance from rubric-based assessments. EL then internalizes this experiential knowledge via on-policy context distillation, converting natural-language feedback into dense per-token supervision and enabling the policy to improve beyond the limits of scalar rewards on open-ended tasks.

Method

The authors address the challenge of training a language model policy on non-verifiable, open-ended tasks where no ground-truth answer exists and quality is judged by another model. Given a prompt $x$x and a set of rubrics $\mathcal{R}_x$Rx​ that decompose desired quality into checkable criteria, the policy $\pi_\theta$πθ​ generates a response $y$y. A feedback language model $M$M then assesses the response, producing a detailed evaluation $M(x, y, \mathcal{R}_x)$M(x,y,Rx​). The core idea is to transform this assessment from a scalar reward into a richer, text-based coaching signal.

In the standard reinforcement learning baseline, $M$M operates as an LLM-as-a-Judge: only a scalar reward $r = \text{Extract\_Reward}(M(x, y, \mathcal{R}_x))$r=Extract_Reward(M(x,y,Rx​)) is retained, discarding all other textual analysis. The policy is trained to maximize the expected reward over a dataset $\mathcal{D}$D of prompts and rubrics, as formalized in Equation (1). The authors implement this using Group Relative Policy Optimization (GRPO).

Experiential Learning (EL) reuses the same feedback model $M$M but elevates it to an LLM-as-a-Coach. Instead of compressing the assessment into a scalar, the coach analyzes the rubric-based evaluation and distills transferable insights into a textual block of experiential knowledge $e = \text{Extract\_Knowledge}(M(x, y, \mathcal{R}_x))$e=Extract_Knowledge(M(x,y,Rx​)). The policy is then optimized by minimizing the reverse Kullback-Leibler (KL) divergence between the policy’s own distribution and a context-conditioned teacher distribution $\pi_{\text{teacher}}(\cdot \mid e, x)$πteacher​(⋅∣e,x). The loss function, given in Equation (2), is a token-level distillation objective that backpropagates the rich information contained in $e$e through on-policy samples.

The teacher can be either the initial frozen policy checkpoint or an iteratively updated version: the policy checkpoint at the end of each epoch becomes the teacher for the next epoch. This turns EL into an on-policy context distillation procedure that internalizes the coach’s distributional guidance directly into the policy weights.

The fundamental distinction between the judge and coach interfaces lies in the feedback bandwidth exploited by the optimizer. A scalar reward derived from a discrete 1–10 scale carries at most $\log_2(10) \approx 3.3$log2​(10)≈3.3 bits of information per sample; even if a learned projection head outputs a bfloat16 reward, the theoretical ceiling is 16 bits. In contrast, the experiential knowledge $e$e produced by the coach is a textual context of up to $L$L tokens drawn from a vocabulary of size $|\mathcal{V}|$∣V∣, yielding a theoretical upper bound of $L \cdot \log_2(|\mathcal{V}|)$L⋅log2​(∣V∣) bits. With typical values ($L = 1024$L=1024, $|\mathcal{V}| = 150,000$∣V∣=150,000), this bandwidth exceeds $10^4$104 times that of a scalar reward. On non-verifiable tasks that involve multiple quality dimensions, stylistic preferences, and nuanced trade-offs, this vastly wider channel enables the policy to capture fine-grained preferences that would be indistinguishable once the scalar reward saturates. EL thus bypasses the information bottleneck of judge-based RL, allowing a higher-fidelity approximation of the target distribution defined by the feedback model.

Experiment

The experiments compare experiential learning (EL) with standard reinforcement learning (RL) for non-verifiable tasks, using rubrics to evaluate open-ended responses. EL employs an LLM-as-a-Coach that generates rich textual feedback, while RL uses a scalar reward from an LLM-as-a-Judge. Results show that EL consistently outperforms RL on held-out data and transfers better to unseen benchmarks, as the richer feedback channel mitigates reward hacking and promotes more generalizable behaviors. Ablations demonstrate that iterative teacher updates combined with general-prompt distillation help preserve out-of-distribution capabilities, and that extracting experiential knowledge from the coach's critique is the most effective context format. A toy analysis further illustrates that scalar rewards lose fine-grained distinctions, while the higher feedback bandwidth of textual guidance enables faithful distribution matching.

When the same feedback model and rubrics are used, treating it as a judge yields only a scalar score and a low-bandwidth learning signal, while using it as a coach extracts high-bandwidth experiential knowledge that can provide finer-grained guidance. A controlled distribution-matching experiment confirms that scalar rewards lose information when the target requires more than two reward levels, producing quantization artifacts that high-bandwidth feedback avoids. Using an LLM-as-a-Judge collapses rich rubric-based assessment into a single scalar, providing a low-bandwidth learning signal. When the target distribution has more than two reward levels, scalar feedback leads to quantization artifacts that erase fine-grained distinctions, whereas high-bandwidth experiential guidance preserves them.

Experiential learning (EL) outperforms reinforcement learning (RL) on most benchmarks, with pronounced gains on AlpacaEval v2.0 and WildBench. The advantage persists when the feedback model is either the initial frozen policy or GPT-4o, indicating that EL's richer information channel drives the improvement. Using GPT-4o as the feedback model further boosts WildChat scores for both methods. EL surpasses RL on nearly all evaluated benchmarks, showing especially large improvements on AlpacaEval v2.0 and WildBench. The performance gap holds regardless of whether the feedback model is the frozen policy checkpoint or GPT-4o, suggesting the benefit comes from EL's richer feedback signal rather than the specific feedback LLM. Switching the feedback model to GPT-4o improves WildChat scores for both RL and EL compared to using the initial policy checkpoint.

Iteratively updating the teacher during experiential learning improves in-distribution chat quality (WildChat score) but causes a sharp drop in out-of-distribution instruction-following accuracy (IFEval). Adding general-prompt OPD from the initial frozen teacher recovers most of the OOD loss while preserving the chat quality improvement. Iterative teacher updates raise the WildChat score from 80.0 to 80.7 but reduce IFEval accuracy from 83.1 to 74.9. Mixing general prompts with iterative teacher updates restores IFEval accuracy to 79.9 without sacrificing the 80.7 WildChat score.

Ablating the teacher context shows that the default experiential learning (EL) format achieves the best combination of WildChat score and out-of-distribution IFEval accuracy. Providing full critiques or rubrics alone improves in-domain score but reduces IFEval accuracy below the base model, indicating forgetting. The default EL recovers and surpasses all other variants on both metrics. Default EL reaches the highest WildChat score (68.6) and the highest IFEval accuracy (68.8), outperforming all other context configurations. Full Critique and Rubrics Only improve the WildChat score over the base model (64.8) but cause IFEval accuracy to drop to 64.6 and 65.6, respectively, falling below the base model's 68.0. RL alone raises the score from 64.8 to 67.7 while slightly decreasing IFEval accuracy from 68.0 to 67.6. Multiple-Choice context yields the lowest WildChat score among the fine-tuned variants (66.3) but preserves high IFEval accuracy (68.5), close to the base model.

Reinforcement learning provides a scalar reward per sample with constant O(1) bandwidth, which can obscure fine-grained distinctions between tokens that receive the same score. By contrast, on-policy distillation from a stronger teacher scales as O(N) with response length, and experiential learning from a coach's knowledge context scales as O(L), both carrying richer per-token information. A controlled distribution-matching experiment confirms that quantizing feedback into scalar levels causes the policy to collapse token probabilities within reward plateaus, while direct distributional guidance preserves the full target distribution without such artifacts. Reinforcement learning feedback is sequence-level and scalar, with limited bandwidth that does not grow with response or context length. On-policy distillation feedback bandwidth scales with response length, and experiential learning bandwidth scales with the coach's experiential context length, offering richer signals. In a toy setting, scalar rewards caused identical probabilities for all tokens that shared the same maximum reward level, losing fine-grained distinctions that distributional matching retained.

Experiments compare standard reinforcement learning (RL) using scalar, low-bandwidth feedback with experiential learning (EL) that extracts high-bandwidth coach knowledge. Scalar rewards collapse fine-grained distinctions into quantization artifacts when the target distribution exceeds two levels, and across benchmarks EL outperforms RL, especially on AlpacaEval v2.0 and WildBench, with the advantage rooted in the richer feedback channel rather than the feedback model. Iterative teacher updates boost in-distribution chat quality but degrade out-of-distribution instruction following, a trade-off that mixing general prompts recovers, while default EL context achieves the best overall balance of in-domain and out-of-domain performance.