Command Palette
Search for a command to run...
CoRT: トークンレベルのルーブリック誘導型方策最適化のための反実仮想リプレイ
CoRT: トークンレベルのルーブリック誘導型方策最適化のための反実仮想リプレイ
Bo-Wen Zhang Junwei He Wen Wang Song-Lin Lv Wentao Ma Rongyi Lin Shuhan Zhong Lan-Zhe Guo
概要
ルーブリックに基づく強化学習は、明示的な評価基準に対してモデル出力を評価することで、言語モデルの訓練を豊かにする。しかし、GRPOスタイルのパイプラインでは、これらの構造化された判断はスカラーの応答レベル報酬に縮約され、応答レベルのアドバンテージに変換された後、生成されたすべてのトークンに一様にブロードキャストされる。これにより、異なる基準が異なるスパン、フォーマット決定、意味的選択に根ざしている場合でも、応答内でクレジットを割り当てる明示的なメカニズムが存在しない。我々は、ルーブリック条件付きGRPOのためのトークンレベルのクレジット重み付け手法であるCoRTを提案する。CoRTは、補助的なトークンスコアリングモデルを訓練する代わりに、反実仮想リプレイを用いて、同一のサンプル応答を、元のルーブリック条件付きプロンプトと、それに対応する基準なしプロンプトの下で再スコアリングする。結果として得られるトークン単位の対数尤度の対比は、ルーブリックコンテキストへの依存度の代理指標として機能する。CoRTは、これらの対比を有界で応答正規化された重みにマッピングし、補助的なスコアラーを導入したり応答レベルの報酬を変更したりすることなく、符号付きGRPOアドバンテージをトークン間で再分配するために使用する。命令チューニング済みモデルと報酬粒度にわたる実験により、CoRTは、比較の大多数において、対応する応答レベルGRPOを上回り、平均で4.4パーセントポイントの向上を示すことが確認された。本手法は、個別の関連性学習段階を回避しつつ、学習されたトークンレベルクレジットのベースラインと競合する性能を維持する。これらの結果は、方策内部の反実仮想尤度対比が、GRPOの単純さと安定性を保持しながら、応答内クレジット割り当てのための効果的な訓練信号を提供することを示唆している。
One-sentence Summary
CoRT, proposed by researchers from Nanjing University, ByteDance, and the University of Chinese Academy of Sciences, is a token-level credit weighting method for rubric-conditioned GRPO that uses counterfactual replay to compute tokenwise log-likelihood contrasts between rubric-conditioned and criteria-free prompts, redistributing the signed advantage without an auxiliary scorer and yielding an average 4.4 percentage point improvement while retaining GRPO's simplicity.
Key Contributions
- CoRT is a token-level credit assignment method for rubric-conditioned GRPO that uses counterfactual replay to compute tokenwise log-likelihood contrasts between rubric-conditioned and criteria-free prompts, serving as a proxy for criterion dependence without training an auxiliary scoring model.
- CoRT maps these contrasts to bounded, response-normalized weights and redistributes the signed GRPO advantage across tokens, keeping the original rollout distribution, scalar reward, verifier, and clipped surrogate structure intact.
- Experiments across instruction-tuned models and reward granularities show that CoRT improves over response-level GRPO in most comparisons, with an average gain of 4.4 percentage points, and remains competitive with learned token-level credit baselines while integrating smoothly with alternative policy optimization algorithms like DAPO and GSPO.
Introduction
In rubric-based instruction following, language models are fine-tuned with response-level rewards that evaluate how well the output satisfies a set of criteria. Standard GRPO assigns the same advantage signal to all tokens in a response, ignoring the fact that some tokens contribute more to meeting the criteria than others. This within-response credit mismatch can slow learning and destabilize training. The authors introduce CoRT, a lightweight method that uses counterfactual replay to rescore each response with the criteria removed, then converts the resulting likelihood contrast into token-level credit weights. These weights redistribute the GRPO advantage toward tokens that depend more on the criteria, without requiring a separate model, while keeping the original rollout distribution, verifier, and clipped surrogate objective intact.
Experiment
Within the Qwen3-4B-Instruct block, the CSR reward GRPO variant consistently achieves the highest scores across all instruction-following benchmarks, outperforming Instruct, SFT, and DPO baselines. The largest improvement is on MultiDimIF Accuracy, where GRPO reaches 74.38 compared to the 55–57 range of the other methods, while the other metrics show more modest gains. GRPO with CSR reward yields the best result on every metric, with a particularly large jump on MultiDimIF Accuracy. SFT and DPO trade places as the second-best method depending on the benchmark, with SFT stronger on IFEval and DPO stronger on IFBench Instruction, MultiDimIF Overall, and AdvancedIF Rubric.
The Qwen3-14B model was evaluated with two reward strategies, CSR and AON, both with and without CoRT. CoRT consistently improves MultiDimIF accuracy, especially for AON where instruction following accuracy rises from 37.92 to 41.09, but it slightly reduces IFEval and IFBench scores when applied to the AON strategy. AON without CoRT achieves the highest IFEval and IFBench, while AON with CoRT yields the best overall MultiDimIF accuracy. CoRT boosts MultiDimIF instruction following accuracy for AON reward GRPO from 37.92 to 41.09, while slightly lowering IFEval from 90.13 to 89.80. CSR reward GRPO sees uniform improvements from CoRT across all metrics, with IFEval rising from 87.87 to 89.17.
Adding CoRT to DAPO leads to consistent improvements across all evaluated metrics, including clear gains on instruction-following benchmarks. In contrast, CoRT provides only marginal benefits for GSPO, with a slight drop in MultiDimIF accuracy. DAPO with CoRT achieves the highest MultiDimIF accuracy among the compared configurations. DAPO with CoRT improves IFEval instruction score by 1.3 points and IFBench instruction score by 1.23 points over DAPO alone. GSPO with CoRT yields only a 0.1 point increase on IFEval instruction and a 0.68 point increase on IFBench instruction, while MultiDimIF accuracy decreases by 0.3 points. DAPO+CoRT outperforms GSPO+CoRT on MultiDimIF accuracy by a clear margin, achieving 84.06 versus 81.96.
Across three experiments, the paper evaluate a range of training methods for instruction-following with Qwen3 models. The CSR reward GRPO variant consistently outperforms baselines, yielding especially large gains on MultiDimIF accuracy. The reasoning trace augmentation CoRT improves MultiDimIF accuracy when applied to GRPO with CSR reward or DAPO, but it can slightly reduce standard IFEval/IFBench scores for AON reward GRPO or GSPO, while DAPO+CoRT achieves the best MultiDimIF accuracy among compared RL configurations.