One-sentence Summary

Researchers from The University of Hong Kong, ByteDance Seed, and Peking University propose SpectraReward, a training-free reward model that leverages pretrained MLLMs to measure the average image-conditioned prompt log-likelihood of recovering the original prompt from a generated image, and introduce Self-SpectraReward, a self-improving closed-loop RL framework where the policy's own understanding branch serves as the reward model for its generation branch, outperforming prior MLLM-derived reward methods.

Key Contributions

• SpectraReward is a training-free reward function that uses the average image-conditioned prompt log-likelihood from a pretrained MLLM as a reward signal for image-generation RL, without preference labels or fine-tuning.
• Self-SpectraReward forms a closed-loop self-improvement framework for unified multimodal models, where the model's understanding branch serves as the reward model for its generation branch, eliminating external reward models.
• Extensive experiments across two generator backbones, three RL algorithms, nine MLLM backbones, and five benchmarks demonstrate that SpectraReward and Self-SpectraReward consistently improve generation and outperform prior methods, with analysis revealing that reward-policy alignment is key, as Self-SpectraReward matches or surpasses larger external models.

Introduction

Image generation has shifted from specialized text-to-image models to unified multimodal models (UMMs) that handle both visual understanding and generation, and reinforcement learning (RL) has become a key post-training step to boost compositional fidelity and instruction-following. However, existing reward models either require expensive human preference annotations and fine-tuning, or rely on pretrained multimodal large language models (MLLMs) as zero-shot judges whose scalar outputs can be noisy and whose question-decomposition pipelines add engineering complexity. The authors propose SpectraReward, a training-free reward function that repurposes any frozen MLLM by computing the image-conditioned prompt-token likelihood as a scalar reward, eliminating the need for preference labels, fine-tuning, or explicit judging. They further introduce Self-SpectraReward for UMMs, where the policy’s own understanding branch provides the reward signal, creating a closed-loop self-improving framework. Extensive experiments across multiple generators, RL algorithms, and MLLM backbones show consistent gains and reveal that a well-aligned self-reward can match or outperform much larger external reward models.

Method

The authors introduce SpectraReward, a reward function designed to extract a scalar reward $R_{\mathcal{M}}(x, y)$RM​(x,y) from a frozen pretrained Multimodal Large Language Model (MLLM) $\mathcal{M}$M. This reward measures how well a generated image $y \sim G_{\theta}(\cdot \mid x)$y∼Gθ​(⋅∣x) realizes a text prompt $x = (x_1, \dots, x_T)$x=(x1​,…,xT​) without requiring preference labels or reward-model fine-tuning. The core idea is to evaluate the generated image by asking an inverse question: how well can the image $y$y be translated back into the prompt $x$x from the perspective of $\mathcal{M}$M? To compute this, the image $y$y is fed as a visual condition to $\mathcal{M}$M, followed by a single teacher-forced forward pass over the prompt tokens. The final reward function is defined as the mean image-conditioned prompt log-likelihood:

$R_{\mathcal{M}}(x, y) = \frac{1}{T - 1} \sum_{t = 1}^{T - 1} \log p_{\mathcal{M}}(x_{t + 1} \mid x_{\leq t}, y).$RM​(x,y)=T−11​∑t=1T−1​logpM​(xt+1​∣x≤t​,y).

This approach aggregates the token-wise likelihoods, which form a semantic spectrum describing how the visual evidence in $y$y supports the semantics of $x$x. As shown in the figure below, SpectraReward computes the image-conditioned prompt likelihood through a single forward pass, providing a dense likelihood signal without the training bottlenecks of scalar scoring or VQA decomposition.

To specialize this for unified multimodal models, the authors propose Self-SpectraReward. In this closed-loop self-improving framework, the model uses its own understanding branch to measure how likely the original prompt is under its generated image. This design couples two dual capabilities within the same unified model: the generation branch maps text to images, while the reward function uses the understanding branch to evaluate the reverse translation. Because both branches share the same tokenizer, vision encoder, and pretraining distribution, the reward signal is highly compatible with the policy's own generation knowledge, ensuring strong reward-policy alignment without needing external reward models.

The authors further analyze the discriminative properties of the reward signal. Regarding language-prior cancellation, while the reward contains the MLLM's language prior, this text-only likelihood is shared by all generated images for a fixed prompt. Therefore, it cancels out when computing group-relative advantages in reinforcement learning, making Pointwise Mutual Information normalization unnecessary for optimization within prompt groups.

To examine token-level semantic sensitivity, the authors evaluate positive and negative image pairs for common failures like attribute and object identity mismatches. As shown in the figure below, the likelihood drop is highly concentrated on the mismatched semantic words. For instance, a counting error significantly lowers the likelihood of the word "Two", while replacing a guitar with a chair sharply reduces the likelihood of "guitar". The sequence-level reward remains higher for positive images, confirming that the semantic spectrum reliably captures targeted visual errors.

Finally, regarding reward ranking reliability, the method effectively orders different rollouts from the same prompt. As shown in the figure below, SpectraReward assigns higher rewards to images that better satisfy the prompt and lower rewards to those with missing objects, wrong attributes, or incorrect spatial relations, providing a robust group-wise ranking signal for text-to-image reinforcement learning.

Experiment

SpectraReward, which uses an MLLM's image-conditioned caption likelihood as a reward, consistently improves text-to-image generation across diverse benchmarks and models, outperforming prior decompositional verification methods. Self-SpectraReward leverages the policy model's own understanding branch to achieve performance rivaling much larger external reward models, demonstrating that reward-policy distributional alignment is more critical than scale alone. Ablations show that likelihood-based rewards are more effective than scalar scoring or VQA-based signals, that a moderate-scale pretrained MLLM suffices, and that AWM is the optimal RL algorithm in this setting.

SpectraReward and its self-rewarding variant consistently improve over the BAGEL baseline and AlphaGRPO across all evaluated text-to-image benchmarks. The gains transfer from 512 to 1024 resolution, with Self-SpectraReward achieving leading GenEval and GenEval2 scores and a strong result on the knowledge-grounded WISE benchmark. This indicates that image-conditioned prompt likelihood provides a more effective RL supervision signal than decompositional verification. SpectraReward improves TIIF-Bench overall scores over BAGEL by a large margin, with a bigger boost on short prompts. Self-SpectraReward adds further GenEval gains on top of SpectraReward's TIIF-Bench improvements. Compared to AlphaGRPO, both SpectraReward variants yield consistent TIIF-Bench and GenEval improvements. Self-SpectraReward transfers 512-resolution training benefits to 1024-resolution inference, achieving top GenEval and GenEval2 results. Self-SpectraReward attains a notably higher WISE score than prior methods, demonstrating stronger knowledge-grounded generation.

SpectraReward consistently improves image generation metrics across different base models and reward MLLM backbones, with gains on both GenEval and TIIF-Bench. Using the policy's own understanding branch as the reward model (Self-SpectraReward) matches or exceeds much larger external MLLMs, demonstrating that reward-policy alignment can be more important than scale. Moderate-scale models around 30B parameters are sufficient, and pretraining-stage checkpoints can outperform instruction-tuned ones for this reward function. Gemma3-12B-Pretrain achieves the highest GenEval score (88.7) among all tested reward MLLMs, outperforming its instruction-tuned counterpart on all three benchmarks. Scaling Qwen3-VL from 8B to 30B improves all metrics, but further scaling to 235B leads to a clear performance drop, showing that larger MLLMs do not always yield better rewards.

When trained with Self-SpectraReward, the AWM algorithm achieves the best downstream performance among RL methods, outperforming FlowGRPO and DiffusionNFT. Self-SpectraReward also surpasses earlier reward models when using the same RL algorithm, demonstrating its effectiveness for image generation reinforcement learning. AWM with Self-SpectraReward yields the highest GenEval and TIIF-Bench scores, establishing it as the top RL algorithm in this comparison. Self-SpectraReward consistently outperforms prior reward models like HPSv3, UnifiedReward, VIEScore, and DVReward under the same RL training setup. Among reward models tested with AlphaGRPO, DVReward leads on GenEval (85.0) while VIEScore trails at 81.7, showing a 3.3-point gap. FlowGRPO with Self-SpectraReward improves over the baseline by 1.3 GenEval points and 6.9 TIIF-Short points, but AWM with DVReward delivers larger gains.

The reward function strongly influences reinforcement learning for image generation. A simple scalar rating score degrades all benchmarks, while a VQA-based yes/no probability yields only minor gains on one metric and no improvement on another. The image-conditioned prompt likelihood used by SpectraReward consistently achieves the highest scores, showing likelihood-based rewards are more effective. Scalar scoring (1–5 rating) causes a substantial performance drop, including a 6.3-point decline on GenEval compared to the baseline. VQA-Score (yes/no probability) provides a small boost on TIIF-S but does not improve GenEval. Prompt likelihood reward achieves the best results across all metrics, outperforming both scalar and VQA-based alternatives.

Sequence-level advantage outperforms token-level advantage on GenEval and TIIF-S, while token-level advantage gives a marginal gain on TIIF-L. None of the token-level normalization strategies consistently surpass the sequence-level baseline, and per-token standard deviation notably degrades TIIF-L. Sequence-level reward is preferred for its stability and overall performance. Sequence-level advantage achieves the highest GenEval (89.5) and TIIF-S (85.1) scores among all reward granularity variants. Token-level advantage with per-token standard deviation drops TIIF-L to 82.1, the lowest across all configurations.

The experiments systematically evaluate reward function design, reward model backbones, RL algorithms, and advantage granularity for text-to-image reinforcement learning. Image-conditioned prompt likelihood consistently outperforms scalar ratings and VQA-based scores, and self-rewarding using the policy's own understanding branch matches or exceeds larger external models, demonstrating that reward-policy alignment matters more than scale. Sequence-level advantage provides stable and effective overall results, while the AWM algorithm with Self-SpectraReward achieves the best performance across GenEval, TIIF-Bench, and WISE benchmarks, including robust transfer from 512 to 1024 resolution.