One-sentence Summary

CoRT, proposed by researchers from Nanjing University, ByteDance, and the University of Chinese Academy of Sciences, is a token-level credit weighting method for rubric-conditioned GRPO that uses counterfactual replay to compute tokenwise log-likelihood contrasts between rubric-conditioned and criteria-free prompts, redistributing the signed advantage without an auxiliary scorer and yielding an average 4.4 percentage point improvement while retaining GRPO's simplicity.

Key Contributions

• CoRT is a token-level credit assignment method for rubric-conditioned GRPO that uses counterfactual replay to compute tokenwise log-likelihood contrasts between rubric-conditioned and criteria-free prompts, serving as a proxy for criterion dependence without training an auxiliary scoring model.
• CoRT maps these contrasts to bounded, response-normalized weights and redistributes the signed GRPO advantage across tokens, keeping the original rollout distribution, scalar reward, verifier, and clipped surrogate structure intact.
• Experiments across instruction-tuned models and reward granularities show that CoRT improves over response-level GRPO in most comparisons, with an average gain of 4.4 percentage points, and remains competitive with learned token-level credit baselines while integrating smoothly with alternative policy optimization algorithms like DAPO and GSPO.

Introduction

In rubric-based instruction following, language models are fine-tuned with response-level rewards that evaluate how well the output satisfies a set of criteria. Standard GRPO assigns the same advantage signal to all tokens in a response, ignoring the fact that some tokens contribute more to meeting the criteria than others. This within-response credit mismatch can slow learning and destabilize training. The authors introduce CoRT, a lightweight method that uses counterfactual replay to rescore each response with the criteria removed, then converts the resulting likelihood contrast into token-level credit weights. These weights redistribute the GRPO advantage toward tokens that depend more on the criteria, without requiring a separate model, while keeping the original rollout distribution, verifier, and clipped surrogate objective intact.

Experiment

Within the Qwen3-4B-Instruct block, the CSR reward GRPO variant consistently achieves the highest scores across all instruction-following benchmarks, outperforming Instruct, SFT, and DPO baselines. The largest improvement is on MultiDimIF Accuracy, where GRPO reaches 74.38 compared to the 55–57 range of the other methods, while the other metrics show more modest gains. GRPO with CSR reward yields the best result on every metric, with a particularly large jump on MultiDimIF Accuracy. SFT and DPO trade places as the second-best method depending on the benchmark, with SFT stronger on IFEval and DPO stronger on IFBench Instruction, MultiDimIF Overall, and AdvancedIF Rubric.

The Qwen3-14B model was evaluated with two reward strategies, CSR and AON, both with and without CoRT. CoRT consistently improves MultiDimIF accuracy, especially for AON where instruction following accuracy rises from 37.92 to 41.09, but it slightly reduces IFEval and IFBench scores when applied to the AON strategy. AON without CoRT achieves the highest IFEval and IFBench, while AON with CoRT yields the best overall MultiDimIF accuracy. CoRT boosts MultiDimIF instruction following accuracy for AON reward GRPO from 37.92 to 41.09, while slightly lowering IFEval from 90.13 to 89.80. CSR reward GRPO sees uniform improvements from CoRT across all metrics, with IFEval rising from 87.87 to 89.17.

Adding CoRT to DAPO leads to consistent improvements across all evaluated metrics, including clear gains on instruction-following benchmarks. In contrast, CoRT provides only marginal benefits for GSPO, with a slight drop in MultiDimIF accuracy. DAPO with CoRT achieves the highest MultiDimIF accuracy among the compared configurations. DAPO with CoRT improves IFEval instruction score by 1.3 points and IFBench instruction score by 1.23 points over DAPO alone. GSPO with CoRT yields only a 0.1 point increase on IFEval instruction and a 0.68 point increase on IFBench instruction, while MultiDimIF accuracy decreases by 0.3 points. DAPO+CoRT outperforms GSPO+CoRT on MultiDimIF accuracy by a clear margin, achieving 84.06 versus 81.96.

Across three experiments, the paper evaluate a range of training methods for instruction-following with Qwen3 models. The CSR reward GRPO variant consistently outperforms baselines, yielding especially large gains on MultiDimIF accuracy. The reasoning trace augmentation CoRT improves MultiDimIF accuracy when applied to GRPO with CSR reward or DAPO, but it can slightly reduce standard IFEval/IFBench scores for AON reward GRPO or GSPO, while DAPO+CoRT achieves the best MultiDimIF accuracy among compared RL configurations.