Command Palette
Search for a command to run...
CoRT: 토큰 수준 루브릭 기반 정책 최적화를 위한 반사실 재생
CoRT: 토큰 수준 루브릭 기반 정책 최적화를 위한 반사실 재생
Bo-Wen Zhang Junwei He Wen Wang Song-Lin Lv Wentao Ma Rongyi Lin Shuhan Zhong Lan-Zhe Guo
초록
루브릭 기반 강화 학습은 명시적 기준에 따라 모델 출력을 평가함으로써 언어 모델 훈련을 풍부하게 한다. 그러나 GRPO 방식의 파이프라인에서는 이러한 구조화된 판단이 응답 수준의 스칼라 보상으로 축소되고 응답 수준의 어드밴티지로 변환되어, 생성된 모든 토큰에 균일하게 적용된다. 이는 서로 다른 기준이 서로 다른 텍스트 구간, 형식 결정 또는 의미적 선택에 근거하더라도 응답 내에서 크레딧을 할당하는 명시적 메커니즘을 제공하지 못한다. 우리는 루브릭 조건부 GRPO를 위한 토큰 수준 크레딧 가중치 부여 방법인 CoRT를 제안한다. CoRT는 보조 토큰 점수 모델을 훈련하는 대신, 반사실 재생을 사용하여 동일하게 샘플링된 응답을 원래의 루브릭 조건부 프롬프트와 이에 대응하는 기준 없는 프롬프트 하에서 재점수화한다. 그 결과로 얻은 토큰별 로그 우도 대비는 루브릭 맥락에 대한 의존성의 대리 지표 역할을 한다. CoRT는 이러한 대비를 유계이고 응답 정규화된 가중치로 매핑하고, 이를 사용하여 보조 점수기를 도입하거나 응답 수준 보상을 변경하지 않고도 부호 있는 GRPO 어드밴티지를 토큰 전체에 재분배한다. 명령어 튜닝된 모델과 다양한 보상 세분화 수준에 걸친 실험 결과, CoRT는 대다수의 비교에서 대응되는 응답 수준 GRPO보다 성능이 향상되었으며, 평균 4.4% 포인트의 개선을 보였다. 이 방법은 별도의 관련성 학습 단계를 피하면서도 학습된 토큰 수준 크레딧 기준선과 경쟁력 있는 성능을 유지한다. 이러한 결과는 정책 내부의 반사실적 우도 대비가 GRPO의 단순성과 안정성을 유지하면서 응답 내 크레딧 할당을 위한 효과적인 훈련 신호를 제공함을 시사한다.
One-sentence Summary
CoRT, proposed by researchers from Nanjing University, ByteDance, and the University of Chinese Academy of Sciences, is a token-level credit weighting method for rubric-conditioned GRPO that uses counterfactual replay to compute tokenwise log-likelihood contrasts between rubric-conditioned and criteria-free prompts, redistributing the signed advantage without an auxiliary scorer and yielding an average 4.4 percentage point improvement while retaining GRPO's simplicity.
Key Contributions
- CoRT is a token-level credit assignment method for rubric-conditioned GRPO that uses counterfactual replay to compute tokenwise log-likelihood contrasts between rubric-conditioned and criteria-free prompts, serving as a proxy for criterion dependence without training an auxiliary scoring model.
- CoRT maps these contrasts to bounded, response-normalized weights and redistributes the signed GRPO advantage across tokens, keeping the original rollout distribution, scalar reward, verifier, and clipped surrogate structure intact.
- Experiments across instruction-tuned models and reward granularities show that CoRT improves over response-level GRPO in most comparisons, with an average gain of 4.4 percentage points, and remains competitive with learned token-level credit baselines while integrating smoothly with alternative policy optimization algorithms like DAPO and GSPO.
Introduction
In rubric-based instruction following, language models are fine-tuned with response-level rewards that evaluate how well the output satisfies a set of criteria. Standard GRPO assigns the same advantage signal to all tokens in a response, ignoring the fact that some tokens contribute more to meeting the criteria than others. This within-response credit mismatch can slow learning and destabilize training. The authors introduce CoRT, a lightweight method that uses counterfactual replay to rescore each response with the criteria removed, then converts the resulting likelihood contrast into token-level credit weights. These weights redistribute the GRPO advantage toward tokens that depend more on the criteria, without requiring a separate model, while keeping the original rollout distribution, verifier, and clipped surrogate objective intact.
Experiment
Within the Qwen3-4B-Instruct block, the CSR reward GRPO variant consistently achieves the highest scores across all instruction-following benchmarks, outperforming Instruct, SFT, and DPO baselines. The largest improvement is on MultiDimIF Accuracy, where GRPO reaches 74.38 compared to the 55–57 range of the other methods, while the other metrics show more modest gains. GRPO with CSR reward yields the best result on every metric, with a particularly large jump on MultiDimIF Accuracy. SFT and DPO trade places as the second-best method depending on the benchmark, with SFT stronger on IFEval and DPO stronger on IFBench Instruction, MultiDimIF Overall, and AdvancedIF Rubric.
The Qwen3-14B model was evaluated with two reward strategies, CSR and AON, both with and without CoRT. CoRT consistently improves MultiDimIF accuracy, especially for AON where instruction following accuracy rises from 37.92 to 41.09, but it slightly reduces IFEval and IFBench scores when applied to the AON strategy. AON without CoRT achieves the highest IFEval and IFBench, while AON with CoRT yields the best overall MultiDimIF accuracy. CoRT boosts MultiDimIF instruction following accuracy for AON reward GRPO from 37.92 to 41.09, while slightly lowering IFEval from 90.13 to 89.80. CSR reward GRPO sees uniform improvements from CoRT across all metrics, with IFEval rising from 87.87 to 89.17.
Adding CoRT to DAPO leads to consistent improvements across all evaluated metrics, including clear gains on instruction-following benchmarks. In contrast, CoRT provides only marginal benefits for GSPO, with a slight drop in MultiDimIF accuracy. DAPO with CoRT achieves the highest MultiDimIF accuracy among the compared configurations. DAPO with CoRT improves IFEval instruction score by 1.3 points and IFBench instruction score by 1.23 points over DAPO alone. GSPO with CoRT yields only a 0.1 point increase on IFEval instruction and a 0.68 point increase on IFBench instruction, while MultiDimIF accuracy decreases by 0.3 points. DAPO+CoRT outperforms GSPO+CoRT on MultiDimIF accuracy by a clear margin, achieving 84.06 versus 81.96.
Across three experiments, the paper evaluate a range of training methods for instruction-following with Qwen3 models. The CSR reward GRPO variant consistently outperforms baselines, yielding especially large gains on MultiDimIF accuracy. The reasoning trace augmentation CoRT improves MultiDimIF accuracy when applied to GRPO with CSR reward or DAPO, but it can slightly reduce standard IFEval/IFBench scores for AON reward GRPO or GSPO, while DAPO+CoRT achieves the best MultiDimIF accuracy among compared RL configurations.