Command Palette
Search for a command to run...
CoRT: Kontrafaktisches Replay für tokenbasierte, rubrikengesteuerte Policy-Optimierung
CoRT: Kontrafaktisches Replay für tokenbasierte, rubrikengesteuerte Policy-Optimierung
Bo-Wen Zhang Junwei He Wen Wang Song-Lin Lv Wentao Ma Rongyi Lin Shuhan Zhong Lan-Zhe Guo
Zusammenfassung
Rubrikenbasiertes bestärkendes Lernen bereichert das Training von Sprachmodellen, indem es Modellausgaben anhand expliziter Kriterien bewertet. In GRPO-artigen Pipelines werden diese strukturierten Urteile jedoch auf eine skalare Belohnung auf Antwort-Ebene reduziert und in einen Vorteil auf Antwort-Ebene umgewandelt, der gleichmäßig auf alle generierten Tokens übertragen wird. Dies lässt keinen expliziten Mechanismus für die Zuweisung von Anerkennung innerhalb einer Antwort, selbst wenn unterschiedliche Kriterien in verschiedenen Textabschnitten, Formatierungsentscheidungen oder semantischen Entscheidungen verankert sind. Wir schlagen CoRT vor, eine Methode zur tokenbasierten Anerkennungsgewichtung für rubrikenkonditioniertes GRPO. Anstatt ein zusätzliches Token-Bewertungsmodell zu trainieren, verwendet CoRT kontrafaktisches Replay, um dieselbe gesampelte Antwort unter dem ursprünglichen rubrikenkonditionierten Prompt und einem abgestimmten, kriterienfreien Prompt erneut zu bewerten. Die resultierenden tokenweisen Log-Likelihood-Kontraste dienen als Proxy für die Abhängigkeit vom Rubrikenkontext. CoRT bildet diese Kontraste auf beschränkte, antwortnormalisierte Gewichte ab und verwendet sie, um den vorzeichenbehafteten GRPO-Vorteil über die Tokens hinweg neu zu verteilen, ohne einen zusätzlichen Bewerter einzuführen oder die Belohnung auf Antwort-Ebene zu verändern. Experimente mit instruktionsoptimierten Modellen und unterschiedlichen Belohnungsgranularitäten zeigen, dass CoRT in der überwiegenden Mehrheit der Vergleiche das entsprechende GRPO auf Antwort-Ebene übertrifft, mit einem durchschnittlichen Gewinn von 4,4 Prozentpunkten. Die Methode bleibt konkurrenzfähig zu gelernten tokenbasierten Anerkennungs-Baselines, vermeidet jedoch eine separate Relevanz-Lernphase. Diese Ergebnisse deuten darauf hin, dass policy-interne kontrafaktische Likelihood-Kontraste ein effektives Trainingssignal für die anerkennungsbasierte Zuweisung innerhalb von Antworten liefern, während sie die Einfachheit und Stabilität von GRPO beibehalten.
One-sentence Summary
CoRT, proposed by researchers from Nanjing University, ByteDance, and the University of Chinese Academy of Sciences, is a token-level credit weighting method for rubric-conditioned GRPO that uses counterfactual replay to compute tokenwise log-likelihood contrasts between rubric-conditioned and criteria-free prompts, redistributing the signed advantage without an auxiliary scorer and yielding an average 4.4 percentage point improvement while retaining GRPO's simplicity.
Key Contributions
- CoRT is a token-level credit assignment method for rubric-conditioned GRPO that uses counterfactual replay to compute tokenwise log-likelihood contrasts between rubric-conditioned and criteria-free prompts, serving as a proxy for criterion dependence without training an auxiliary scoring model.
- CoRT maps these contrasts to bounded, response-normalized weights and redistributes the signed GRPO advantage across tokens, keeping the original rollout distribution, scalar reward, verifier, and clipped surrogate structure intact.
- Experiments across instruction-tuned models and reward granularities show that CoRT improves over response-level GRPO in most comparisons, with an average gain of 4.4 percentage points, and remains competitive with learned token-level credit baselines while integrating smoothly with alternative policy optimization algorithms like DAPO and GSPO.
Introduction
In rubric-based instruction following, language models are fine-tuned with response-level rewards that evaluate how well the output satisfies a set of criteria. Standard GRPO assigns the same advantage signal to all tokens in a response, ignoring the fact that some tokens contribute more to meeting the criteria than others. This within-response credit mismatch can slow learning and destabilize training. The authors introduce CoRT, a lightweight method that uses counterfactual replay to rescore each response with the criteria removed, then converts the resulting likelihood contrast into token-level credit weights. These weights redistribute the GRPO advantage toward tokens that depend more on the criteria, without requiring a separate model, while keeping the original rollout distribution, verifier, and clipped surrogate objective intact.
Experiment
Within the Qwen3-4B-Instruct block, the CSR reward GRPO variant consistently achieves the highest scores across all instruction-following benchmarks, outperforming Instruct, SFT, and DPO baselines. The largest improvement is on MultiDimIF Accuracy, where GRPO reaches 74.38 compared to the 55–57 range of the other methods, while the other metrics show more modest gains. GRPO with CSR reward yields the best result on every metric, with a particularly large jump on MultiDimIF Accuracy. SFT and DPO trade places as the second-best method depending on the benchmark, with SFT stronger on IFEval and DPO stronger on IFBench Instruction, MultiDimIF Overall, and AdvancedIF Rubric.
The Qwen3-14B model was evaluated with two reward strategies, CSR and AON, both with and without CoRT. CoRT consistently improves MultiDimIF accuracy, especially for AON where instruction following accuracy rises from 37.92 to 41.09, but it slightly reduces IFEval and IFBench scores when applied to the AON strategy. AON without CoRT achieves the highest IFEval and IFBench, while AON with CoRT yields the best overall MultiDimIF accuracy. CoRT boosts MultiDimIF instruction following accuracy for AON reward GRPO from 37.92 to 41.09, while slightly lowering IFEval from 90.13 to 89.80. CSR reward GRPO sees uniform improvements from CoRT across all metrics, with IFEval rising from 87.87 to 89.17.
Adding CoRT to DAPO leads to consistent improvements across all evaluated metrics, including clear gains on instruction-following benchmarks. In contrast, CoRT provides only marginal benefits for GSPO, with a slight drop in MultiDimIF accuracy. DAPO with CoRT achieves the highest MultiDimIF accuracy among the compared configurations. DAPO with CoRT improves IFEval instruction score by 1.3 points and IFBench instruction score by 1.23 points over DAPO alone. GSPO with CoRT yields only a 0.1 point increase on IFEval instruction and a 0.68 point increase on IFBench instruction, while MultiDimIF accuracy decreases by 0.3 points. DAPO+CoRT outperforms GSPO+CoRT on MultiDimIF accuracy by a clear margin, achieving 84.06 versus 81.96.
Across three experiments, the paper evaluate a range of training methods for instruction-following with Qwen3 models. The CSR reward GRPO variant consistently outperforms baselines, yielding especially large gains on MultiDimIF accuracy. The reasoning trace augmentation CoRT improves MultiDimIF accuracy when applied to GRPO with CSR reward or DAPO, but it can slightly reduce standard IFEval/IFBench scores for AON reward GRPO or GSPO, while DAPO+CoRT achieves the best MultiDimIF accuracy among compared RL configurations.