One-sentence Summary

Researchers from the Chinese Academy of Sciences, The Hong Kong University of Science and Technology, and Tencent propose RewardVerse, a rubric-guided video reward framework that generates dynamic query-adaptive rubrics before scoring and employs two-stage Rubric-Guided Policy Optimization (RGPO) to mitigate scalar drift, achieving state-of-the-art pointwise and pairwise evaluation results for video generation reinforcement learning.

Key Contributions

• RewardVerse is a rubric-based video reward framework that uses a dynamic rubric as an intermediate representation between the evaluation query and the scorer. It first generates explicit evaluation criteria and then performs rubric-guided scoring, providing a stable semantic anchor that mitigates scalar drift.
• Rubric-Guided Policy Optimization (RGPO) is a two-stage training algorithm that first warms up the scorer using self-evolving seed rubrics and then jointly optimizes the rubric generator to produce query-adaptive criteria while continuously aligning the scorer with human ratings.
• Experiments on the 16-dimensional EvalVerse benchmark and external datasets show that RewardVerse mitigates scalar drift and achieves state-of-the-art pointwise and pairwise evaluation performance. RewardVerse attains high pointwise correlation on EvalVerse, strong pairwise agreement on VGRB, and improved downstream video generation while being more robust to reward hacking.

Introduction

Generative video models rely on reward models to align outputs with human preferences during reinforcement learning, but existing video reward models typically map a video directly to a single scalar score. Because video quality is subjective and multidimensional, this unconstrained scoring often produces scalar drift, where scores collapse into a narrow range or shift across prompts and contexts. Prior methods lack explicit evaluation criteria before scoring, making their reward signals unstable for downstream RL. The authors propose RewardVerse, a video evaluation framework that inserts a dynamic rubric as an intermediate representation, decoupling criterion generation from scoring to stabilize pointwise and pairwise rewards. They also introduce Rubric-Guided Policy Optimization, a two-stage training algorithm that jointly learns query-adaptive rubrics and aligns the scorer using only 30 preference pairs per dimension without supervised fine-tuning.

Method

The authors reframe rubrics as a learnable intermediate representation rather than a fixed prompt scaffold. The goal is to generate query-adaptive evaluation criteria while keeping the scorer aligned with human ratings. RewardVerse implements this through a dual-role policy model $\pi_\theta$πθ​ that acts as both a dynamic rubric generator and a scorer. The training procedure, Rubric-Guided Policy Optimization (RGPO), consists of two stages: a scorer warm-up using self-evolving seed rubrics, followed by joint optimization of the generator and scorer.

An evaluation query is defined as $\boldsymbol{q} = (d, p)$q=(d,p), where $d \in \mathcal{D}$d∈D is a target evaluation dimension and $p$p is the prompt used to synthesize the candidate video. Each video is scored independently during training and inference.

The dynamic rubric generator samples a structured rubric from the query:

$$\hat{R}_q \sim \pi_\theta^{\mathrm{gen}}(\cdot \mid q), \quad \hat{R}_q = \{(t_k, w_k, T_k)\}_{k=1}^{K},$$R^q​∼πθgen​(⋅∣q),R^q​={(tk​,wk​,Tk​)}k=1K​,

where $t_k$tk​ is a theme, $w_k \in (0,1)$wk​∈(0,1) is a normalized weight, and $T_k$Tk​ is a set of execution tips. Importantly, the generator sees only the query and does not observe the candidate video. This candidate-independent design prevents the generator from producing biased rubrics that are overly lenient to weak videos or overly strict to strong videos.

The scorer uses a soft-logits readout instead of parsing discrete digits. For each theme, the model reads the logits of the five rating tokens and computes an expected rating:

$$c_k(y \mid R) = \sum_{v=1}^{5} v \cdot P_\theta(v \mid y, t_k, T_k),$$ck​(y∣R)=v=1∑5​v⋅Pθ​(v∣y,tk​,Tk​),

where

$$P_\theta(v \mid y, t_k, T_k) = \frac{\exp(l_v)}{\sum_{v'=1}^{5}\exp(l_{v'})}.$$Pθ​(v∣y,tk​,Tk​)=∑v′=15​exp(lv′​)exp(lv​)​.

The pointwise reward aggregates theme scores by their rubric weights:

$$S(y \mid R) = \sum_{k=1}^{K} w_k \cdot c_k(y \mid R).$$S(y∣R)=k=1∑K​wk​⋅ck​(y∣R).

The generator and scorer share the same backbone and are distinguished by role-specific prompts.

In Stage 1, the authors warm up the scorer with seed rubrics. For each dimension $d$d, a frontier MLLM constructs an initial seed rubric $R_d^{\star}$Rd⋆​ through an offline self-evolving loop. Given preference triples, the MLLM proposes a rubric that explains why $y_w \succ y_l$yw​≻yl​, verifies whether the rubric recovers that preference when used for scoring, and revises the rubric on failure. Verified rubrics are pooled, deduplicated with a two-level Jaccard filter, and reduced to five representative rubrics using an $\mathrm{MCR}^2$MCR2 sampler.

Under each seed rubric, the scorer samples $G$G completion pairs for the preferred and non-preferred videos. The preference reward is:

$$r_{\mathrm{pref}}(a_{w,g}, a_{l,g}) = \sigma\big(S_g(y_w \mid R_d^{\star}) - S_g(y_l \mid R_d^{\star})\big).$$rpref​(aw,g​,al,g​)=σ(Sg​(yw​∣Rd⋆​)−Sg​(yl​∣Rd⋆​)).

The full warm-up reward adds a format reward that checks whether the completions follow the requested scoring format. The joint reward is standardized across the group to produce a trajectory-level advantage, and the scorer is optimized with a clipped GRPO objective plus a KL penalty against a frozen reference policy.

Because preference reward alone does not control the magnitude of score differences, the authors add a human-aligned margin calibration loss:

$$\mathcal{L}_{\mathrm{cal}}(\theta \mid R) = \frac{1}{S_{\max} - S_{\min}} \left| \big(S(y_w \mid R) - S(y_l \mid R)\big) - \Delta\hat{s}_{w,l} \right|.$$Lcal​(θ∣R)=Smax​−Smin​1​​(S(yw​∣R)−S(yl​∣R))−Δs^w,l​​.

This loss calibrates predicted score gaps against human-annotated margins. The Stage 1 objective combines the GRPO loss with this calibration term:

$$\mathcal{L}_{\mathrm{stage1}}(\theta) = \mathcal{L}_{\mathrm{GRPO}}^{(1)}(\theta) + \lambda_{\mathrm{cal}} \mathcal{L}_{\mathrm{cal}}(\theta \mid R_d^{\star}).$$Lstage1​(θ)=LGRPO(1)​(θ)+λcal​Lcal​(θ∣Rd⋆​).

In Stage 2, the generator learns to produce dynamic rubrics tailored to each query. For each preference triple, it samples $G$G rubrics $\{\hat{R}_g\}_{g=1}^{G} \sim \pi_{\theta_{\mathrm{old}}}^{\mathrm{gen}}(\cdot \mid q)${R^g​}g=1G​∼πθold​gen​(⋅∣q). Each rubric is scored according to how well its induced pointwise scores separate the preferred and non-preferred videos:

$$r_{\mathrm{pref}}(\hat{R}_g) = \sigma\big(S_g(y_w \mid \hat{R}_g) - S_g(y_l \mid \hat{R}_g)\big).$$rpref​(R^g​)=σ(Sg​(yw​∣R^g​)−Sg​(yl​∣R^g​)).

The Stage 2 reward also includes a format reward and an alignment reward:

$$\mathcal{R}_2(\hat{R}_g) = r_{\mathrm{pref}}(\hat{R}_g) + \beta r_{\mathrm{fmt}}(\hat{R}_g) + \lambda r_{\mathrm{align}}(\hat{R}_g).$$R2​(R^g​)=rpref​(R^g​)+βrfmt​(R^g​)+λralign​(R^g​).

The alignment reward measures cosine similarity between BGE-M3 embeddings of the generated and seed themes. The composite reward is standardized across the group and broadcast to all tokens in the generated rubric. The generator is then updated with a clipped GRPO objective.

The optimization signals for the two roles are asymmetric. The generator receives the rubric-level policy gradient from $\mathcal{R}_2$R2​, while the scorer receives no direct policy-gradient update from this reward. Instead, the scorer is optimized only by the margin calibration loss, treating each generated rubric as fixed text context:

$$\mathcal{L}_{\mathrm{scorer}}(\theta) = \frac{1}{G}\sum_{g=1}^{G} \mathcal{L}_{\mathrm{cal}}(\theta \mid \hat{R}_g).$$Lscorer​(θ)=G1​g=1∑G​Lcal​(θ∣R^g​).

The full Stage 2 objective is:

$$\mathcal{L}_{\mathrm{stage2}}(\theta) = \mathcal{L}_{\mathrm{GRPO}}^{(2)}(\theta) + \alpha(t)\mathcal{L}_{\mathrm{scorer}}(\theta),$$Lstage2​(θ)=LGRPO(2)​(θ)+α(t)Lscorer​(θ),

where $\alpha(t)$α(t) is a scheduled coefficient that activates scorer calibration after the initial generator warm-up. This joint formulation allows the generator to produce query-adaptive rubrics with varying weights and prompt-specific tips, while the scorer remains calibrated to human score margins.

Experiment

The analytical study shows that unconstrained MLLM video scoring suffers from scalar drift through score collapse and context sensitivity, while rubric guidance anchors the scale and soft-logits decoding further decompresses it, though static rubrics alone are insufficient. Evaluation on EvalVerse and VGRB validates that jointly optimizing rubric generation and scoring improves pointwise human correlation and transfers to pairwise ranking, including unseen dimensions. Ablations confirm the importance of rubric-mediated training, scorer warm-up, and dynamic rubric adaptation under a low-data budget, and downstream RL experiments show that RewardVerse improves target visual quality while preserving other video capabilities.

RewardVerse (Joint) leads pointwise evaluation across EvalVerse dimensions, achieving the top PLCC in 14 of 16 dimensions and outperforming prior reward models most clearly on challenging cognitive and temporal axes such as Logic and Action. External video scorers are often poorly calibrated on these fine-grained dimensions, including negative alignment on Logic. Some non-quality dimensions remain competitive for Q-Scorer, indicating partial overlap with low-level visual quality. RewardVerse (Joint) achieves the strongest overall performance, with the highest PLCC in 14 of 16 EvalVerse dimensions. It shows especially large gains on cognitive and temporal tasks such as Logic and Action compared with the next-best models. External video scorers can be poorly calibrated on fine-grained dimensions, with VideoScore-v1.1 showing negative Logic alignment. Q-Scorer remains competitive on non-quality dimensions like Lighting and Logic, suggesting those axes partly correlate with low-level visual quality.

RewardVerse records the highest pairwise agreement among non-oracle methods on both the unseen text alignment and seen visual quality splits. It improves over the strongest non-oracle baseline on visual quality by a substantial margin while using a limited training budget. The results suggest that reducing scalar drift in pointwise scoring also improves pairwise preference alignment, including on unseen evaluation dimensions. RewardVerse ranks first among non-oracle methods on both the unseen text alignment and seen visual quality splits. Despite limited training pairs, RewardVerse outperforms the strongest non-oracle baseline and closes much of the gap to the oracle upper bound. On the unseen text alignment split, RewardVerse shows stronger zero-shot generalization than all non-oracle baselines.

On EvalVerse, the full RGPO configuration achieves the highest macro-averaged PLCC and SRCC among all tested settings. Omitting Stage-1 scorer warm-up produces the largest drop among model ablations, while skipping Stage-2 joint optimization or using flat rubrics also lowers alignment. Rubric-free training improves only modestly over zero-shot and remains below the rubric-guided full model; downstream, RewardVerse improves target visual quality while better preserving text and quality capabilities than VideoReward. The full RGPO configuration outperforms every ablation and rubric-free baseline on both PLCC and SRCC. Removing Stage-1 scorer warm-up causes the largest correlation drop among ablations, highlighting its role in score calibration. Skipping Stage-2 joint optimization or replacing hierarchical tips with a flat rubric reduces alignment relative to the full model. Direct rubric-free training improves only slightly over zero-shot and remains well below the full rubric-guided model. RewardVerse improves visual quality in downstream generation while preserving text and quality capabilities better than VideoReward.

RewardVerse (Joint) is evaluated across fine-grained EvalVerse dimensions, pairwise preference alignment, and ablation settings, and it consistently provides the strongest pointwise and pairwise agreement among non-oracle models. The experiments show that its advantages are especially clear on challenging cognitive and temporal dimensions, while external scorers can be poorly calibrated and some non-quality axes partly overlap with low-level visual quality. Ablations confirm that scorer warm-up, joint optimization, and hierarchical rubrics all contribute to alignment, with rubric-free training remaining weaker, and RewardVerse also improves downstream visual quality while better preserving text and quality capabilities than prior reward models.