One-sentence Summary

LLM-as-a-Tutor, proposed by researchers from KAIST, Upstage, University of Toronto, and others, is a framework that extends an LLM from judge to tutor by appending atomic constraints to non-challenging prompts, enabling policy-aware prompt adaptation that monotonically raises difficulty with policy capability, yielding self-calibrating rewards and outperforming prior methods on three complex instruction-following benchmarks.

Key Contributions

• The paper identifies that a mismatch between prompt difficulty and policy capability can suppress reward variance, framing this mismatch as a key precondition that limits rubric-equipped judges in non-verifiable RL.
• It introduces LLM-as-a-Tutor, a framework that extends a single LLM from judge to tutor by detecting non-challenging prompts through pairwise comparison and appending atomic constraints, monotonically raising difficulty in step with the policy without external schedules.
• Across three complex instruction-following benchmarks, the method consistently outperforms both policy-unaware baselines and prior policy-adaptive approaches that adapt rubrics or rewrite prompts, establishing prompt adaptation as a complementary axis of policy-awareness in non-verifiable RL.

Introduction

Reinforcement learning for instruction following in large language models has advanced by replacing coarse scalar reward models with LLM judges that score responses against instance-specific rubrics, yielding fine-grained discriminative signals. However, these rubric-equipped judges only work when the prompt elicits rollouts of varying quality; prompts that are trivially easy or impossibly hard for the current policy cause the reward signal to collapse, regardless of judge quality. Existing pipelines draw training prompts from static corpora, leaving the supply of such challenging prompts to chance, and prior policy-adaptive methods either fall back on coarse scalar rewards or use generic, prompt-agnostic rubrics that lack the needed granularity. The authors introduce LLM-as-a-Tutor, which reuses the judge LLM to detect non-challenging prompts via pairwise rollout comparison and then augments them with atomic constraints, creating a self-calibrating curriculum that adapts prompts to the policy’s current capability and consistently outperforms both static and prior adaptive baselines.

Method

The authors introduce a framework that dynamically adapts training prompts to match a policy’s evolving capability during reinforcement learning for non-verifiable instruction-following tasks. The core idea is that a fixed prompt corpus can become non-discriminative as the policy improves, leading to rollouts with little quality variance and thus weak learning signals. To address this, they propose an LLM-as-a-Tutor that iteratively appends atomic constraints to prompts that the current policy has mastered, thereby escalating difficulty in lockstep with the policy’s progress.

The training pipeline rests on a rubric-based reward model. For each prompt $x$x, a base rubric $R(x) = \{(r_k, w_k)\}_{k=1}^K$R(x)={(rk​,wk​)}k=1K​ defines $K$K quality criteria with non-negative weights summing to one. An LLM judge $\mathcal{J}$J scores a response $y$y against each criterion independently, and the overall reward is a weighted sum: $s(x, y) = \sum_{k=1}^{K} w_k \cdot \mathcal{J}(y \mid x, r_k).$s(x,y)=∑k=1K​wk​⋅J(y∣x,rk​). Policy optimization follows the Group Relative Policy Optimization (GRPO) framework. For a given prompt, $G$G rollouts are sampled from the old policy $\pi_{\theta_{\text{old}}}$πθold​​, and the policy $\pi_\theta$πθ​ is updated by maximizing a clipped surrogate objective that includes a KL penalty toward a frozen reference policy. The per-rollout advantage $A^{(i)}$A(i) is computed in a group-relative manner: $A^{(i)} = \frac{s(x, y^{(i)}) - \mu(x)}{\sigma(x)},$A(i)=σ(x)s(x,y(i))−μ(x)​, where $\mu(x)$μ(x) and $\sigma(x)$σ(x) are the mean and standard deviation of the rollout scores within the group. This advantage provides a meaningful learning signal only when the rollouts exhibit sufficient score spread. The authors emphasize that discriminativeness is a property of the prompt–policy pair, not the prompt alone; when the policy has mastered a prompt, the rollouts become indistinguishable in quality, and the advantage collapses.

To overcome this, the LLM-as-a-Tutor operates at the start of every training iteration. It first generates a base rubric $R(x)$R(x) for each prompt. Then, for each prompt, it samples two rollouts from the current policy and makes a binary pairwise judgment: whether the two responses are indistinguishable in quality. This pairwise comparison leverages the observation that LLMs are sharper and more human-aligned in pairwise settings than in pointwise scoring. If the rollouts are deemed indistinguishable, the prompt is declared non-discriminative for the current policy.

When a prompt is non-discriminative, the tutor invokes its second role as a constraint generator. It produces an atomic constraint $c$c, a single requirement along a dimension left unspecified by the original prompt, along with a corresponding rubric $R_c = \{(r_{c,j}, w_{c,j})\}_{j=1}^m$Rc​={(rc,j​,wc,j​)}j=1m​ that evaluates adherence to that constraint. The adapted prompt and rubric set are formed by appending the constraint and merging the rubrics: $\tilde{x} = x \oplus c, \quad \tilde{R}(x) = R(x) \cup R_c,$x~=x⊕c,R~(x)=R(x)∪Rc​, with the weights renormalized to sum to one. Discriminative prompts are left unchanged. The reward and GRPO update then proceed exactly as before but with the adapted $(\tilde{x}, \tilde{R}(x))$(x~,R~(x)), making the adaptation the only point of modification relative to the baseline.

This append-only design ensures that difficulty monotonically increases: any response satisfying $\tilde{x}$x~ must also satisfy $x$x, so the adapted prompt is never easier than the original. Moreover, retaining the original rubrics $R(x)$R(x) in $\tilde{R}(x)$R~(x) keeps the base criteria active, so as constraints accumulate, scoring well on $R(x)$R(x) itself becomes more demanding. The tutor is invoked at every adaptation interval (e.g., every epoch), allowing prompts adapted in earlier iterations to be revisited and further escalated as the policy improves. The sequence of prompts $\{x^{(t)}\}${x(t)} is thus non-decreasing in difficulty, with escalation triggered only when the current policy renders a prompt non-discriminative. This dynamic, policy-driven curriculum realizes a per-prompt difficulty progression that a static corpus cannot provide, effectively shaping the training signal so the student discovers new capabilities through RL without requiring target responses from the tutor.

Experiment

The evaluation trains a Qwen3-1.7B policy with a Qwen3-8B tutor and judge on WildChat prompts, testing on instruction-following benchmarks. Experiments show that policy-adaptive prompt modification via appended atomic constraints outperforms both policy-unaware baselines and rubric-only adaptation, as it restores discriminative reward signals when prompts become too easy for the current policy. Ablations confirm that targeting only non-discriminative prompts and cumulatively adding constraints are essential, while analysis reveals the tutor's decisions track empirical difficulty and scale with policy capability. The results suggest that shaping the prompt distribution through a tutor's discriminative judgments constitutes an effective form of distillation that allows the policy to improve beyond the tutor's own answer quality.

LLM-as-a-Tutor achieves the highest average score across instruction-following benchmarks and outperforms all baselines on five of six metrics, showing that prompt-level adaptation provides a richer training signal than rubric-based methods alone. Policy-adaptive rubrics improve over policy-unaware approaches, but modifying prompts can restore variance in saturated tasks where even well-adapted rubrics fail to discriminate. Distillation without prompt modification degrades performance below the untrained policy, indicating that naive imitation is harmful. Policy-adaptive rubrics outperform static rubrics and WildChecklists, confirming that adapting the reward signal to the current policy yields better instruction following. LLM-as-a-Tutor surpasses all baselines on five of six metrics, with prompt adaptation reintroducing discriminative variance when rubrics cannot differentiate rollout quality.

Adaptive constraint addition, which selects prompts based on the policy's own rollouts, outperforms all non-adaptive variants. Applying constraints to all prompts or a random subset is less effective, and using a larger model's judgments to identify targets also falls short despite modifying more prompts, demonstrating that the current policy's behavior is the key signal. The adaptive method achieves the highest average score, surpassing always-on, random, and off-policy selection strategies. Adding constraints to every prompt or to a random 28% of prompts both yield lower performance, showing that indiscriminate modification is harmful. Using an 8B model's rollouts to select prompts (Wrong) modifies 47% of prompts but still underperforms the adaptive approach, indicating that difficulty alone is insufficient and the policy's own behavior matters.

Appending new constraints while retaining previous ones (Append) outperforms both substituting constraints (Reset) and rewriting the entire prompt (Rewrite) on all evaluated benchmarks. Append monotonically increases prompt difficulty without drifting from the original task distribution, leading to higher scores on FB, AdvIF, and IB, and the highest average performance. Append achieves the highest scores on all three benchmarks (FB, AdvIF, IB) and the highest average. Reset and Rewrite both underperform Append, with Rewrite scoring slightly higher than Reset but still below Append. Append's monotonic difficulty increase and preservation of the seed distribution explain its superior results, while Reset and Rewrite risk reducing difficulty or drifting from the task.

Adding a single atomic constraint to a saturated prompt reintroduces discriminative reward variance. On the original easy prompt, all model rollouts achieved perfect scores, yielding zero standard deviation and no informative signal. After appending a requirement to name the specific company policy, the mean score dropped and standard deviation increased, separating compliant responses from those that deflect without satisfying the constraint. Before the constraint, all rollouts scored 100 with zero standard deviation, making the prompt uninformative for training. After the constraint, a rollout that explicitly cited a policy name scored 100, while one that only advised consulting a supervisor scored 60, demonstrating a clear quality gap.

The evaluation compares LLM-as-a-Tutor, a prompt-level adaptation method, against various baselines for instruction following, validating that adapting prompts based on the policy's own rollouts and appending atomic constraints yields a richer training signal than static rubrics or indiscriminate modifications. Adaptive constraint selection and monotonic difficulty increase via appending outperform alternative strategies like resetting or rewriting prompts, and this approach restores discriminative variance in saturated tasks where even well-adapted rubrics fail. The findings confirm that the policy's behavior is the key signal for effective adaptation, with LLM-as-a-Tutor achieving superior performance by combining policy-adaptive rubrics with targeted prompt modifications.