One-sentence Summary

VBVR-Pro, from Nanyang Technological University, UC Berkeley, UC San Diego, et al., is a closed-loop testbed featuring 300 procedurally generated tasks, verifiable rule-based reward scorers that outperform VLM-as-a-judge for multi-task reinforcement learning, and controlled studies across 30+ generators, demonstrating that video generation excels at spatiotemporal tracking and that vision-native trajectories surpass explicit linguistic chains of thought for visual reasoning.

Key Contributions

• Introduces VBVR-Pro, a closed-loop testbed with 300 procedurally generated visual reasoning tasks, and demonstrates that models trained on it transfer to seven external benchmarks (e.g., RISE-Video, MME-CoF-Pro, BabyVision), with ablation evidence confirming the gains stem from visual reasoning rather than instruction-pattern fitting.
• Provides verifiable, task-specific reward scorers that avoid recurring VLM-as-a-judge failure modes and achieve fine-grained human alignment, and uses these scorers as reliable reward signals for multi-task reinforcement learning across 50 tasks, yielding stronger post-RL performance on visual reasoning tasks.
• Enables controlled modality studies across more than 30 image, video, and interleaved generators, finding that video generation is strongest for persistent spatiotemporal state tracking, interleaved generation offers a compute-efficient alternative, and ablations and probing indicate vision-native trajectories are a more crucial reasoning substrate than explicit linguistic chains of thought.

Introduction

Large language models have made language the dominant medium for studying machine reasoning, but many forms of intelligence in the physical world are inherently non-linguistic, involving spatial transformations, temporal continuity, object persistence, and dynamic interactions. Prior work has struggled to study native visual reasoning systematically: existing benchmarks are mostly evaluation-only with little training data, large-scale synthetic sources focus on simplified symbolic settings, and VLM-based judges are unreliable for tasks requiring exact counts, fine-grained spatial relations, or temporal consistency. The authors introduce VBVR-Pro, a closed-loop infrastructure that makes visual reasoning through generation trainable, verifiable, optimizable, and experimentally controllable. It provides a procedurally generated task suite with 300 tasks and aligned modalities (video, keyframe images, and interleaved text-image data), verifiable reward scorers that outperform VLM judges in per-instance human agreement and support reinforcement learning, and a controlled comparison across over 30 generators. Their results show that video generation excels at persistent spatiotemporal tracking, interleaved generation offers a compute-efficient alternative, and image-only generation often lacks expressive capacity for procedural reasoning, with diagnostics confirming that transfer gains stem from learned visual reasoning operations rather than instruction-pattern fitting.

Dataset

The authors construct VBVR-Pro-Dataset, a large-scale synthetic dataset designed to improve transferability in visual reasoning by expanding task coverage rather than only increasing instance count.

• Dataset composition and sources

  • The dataset is built through a collaborative effort involving over 50 researchers and engineers.
  • It contains 300 tasks, of which 150 are reimplemented and revised from the existing VBVR benchmark, and 150 are newly designed to broaden coverage.
  • Tasks are organized under a cognitive taxonomy with five faculties: perception, spatiality, transformation, abstraction, and knowledge. A single task can carry multiple faculty labels.
  • Each task is generated by a parameterized program, or generator, that samples configurations such as grid size, object count, layout, appearance, and difficulty, then instantiates a problem instance with a programmatically derived solution.

• Key details for each subset

  • The full dataset spans 300 tasks.
  • From the 300 tasks, 50 are held out for out-of-domain evaluation.
  • From the remaining 250 tasks, the authors sample 5,000 instances per task, yielding 1.25 million training instances.
  • VBVR-Pro-Bench contains 100 tasks total: the 50 out-of-domain tasks plus 50 in-domain tasks selected from the 250 tasks that have training data.
  • A dedicated reinforcement learning training set covers the 50 in-domain tasks and is used in the experiments.

• How the paper uses the data

  • The training set is used for supervised fine-tuning and reinforcement learning experiments.
  • In the RL experiments, the authors use a 50K instance subset spanning the 50 in-domain tasks, starting from the same base model checkpoint.
  • The dataset supports video, image, and interleaved modalities under a unified task distribution.

• Cropping strategy, metadata construction, and other processing details

  • Each solved instance includes a metadata file with the random seed, problem specification, complete solution, and key element attributes. This metadata provides structured ground truth for verifiable reward scorers and is used for de-duplication and split construction.
  • Each instance is rendered into aligned video and image modalities. The image modality uses three output regimes: Last-Frame for tasks solvable with a single final state, Key-Frame for tasks requiring selected states with solution trajectories drawn for path-based tasks, and Multi-Frame for tasks that evaluate process integrity or temporal continuity.
  • Videos and images are different renderings of the same underlying problem with equivalent solutions, enabling controlled comparison between modalities.
  • All 300 generators are reviewed by inspecting rendered samples and verifying solver implementations. Resolution, frame rate, and frame count are checked for every task.
  • For the interleaved setting, the authors use Gemini-3.1-Pro-Preview to generate textual descriptions of intermediate solution steps, with a dedicated prompt applied uniformly to all instances from each task.
  • Compared with the 150 reworked tasks, the 150 new tasks show higher visual complexity, with a median of 80 distinct connected color regions per frame versus 12, and deeper reasoning requirements, with 47% requiring multi-step reasoning versus 7% for the reworked tasks.

Method

The authors leverage 300 task-specific generators under a unified framework to construct VBVR-Pro, sampling configurations from a structured parameter space to instantiate problem instances. Each instance is accompanied by metadata containing the random seed, problem specification, and complete solution, providing programmatically derived supervision. A key feature of this framework is the rendering of instances into aligned visual modalities. As shown in the figure below, the image modality adopts three distinct output regimes: Last-Frame displays only the final state for tasks solvable by a single image; Key-Frame presents selected states capturing essential transitions, often drawing the complete solution trajectory for path-based tasks; and Multi-Frame uniformly samples intermediate states to evaluate process integrity or temporal continuity.

To evaluate visual reasoning reliably, the authors identify critical limitations in the VLM-as-a-judge paradigm. As illustrated in the figure below, VLM judges frequently exhibit three representative failure modes: imprecise perception on fine-grained visual details, oversight of decisive errors, and fundamental misunderstanding of the task itself.

Beyond inaccuracy, VLM judges are computationally expensive and lack reproducibility, producing inconsistent scores across repeated evaluations. The figure below demonstrates that the proposed deterministic scorers achieve higher agreement with human preferences at a substantially lower cost per evaluation compared to both proprietary and open-source VLM judges.

To overcome these limitations, the authors design a suite of verifiable reward scorers that operate on task-relevant semantic entities rather than raw pixels. Each scorer locates objects of interest and extracts attributes such as color, shape, position, and count using classical computer-vision methods like HSV color segmentation and contour detection. The figure below illustrates this end-to-end evaluation process. For tasks with soft criteria, the final score is computed as a weighted sum of task-specific checks. For tasks with hard constraints, checks are combined multiplicatively so that a single violation sharply reduces the final score, providing interpretable error decomposition.

For large-scale multi-task reinforcement learning, the authors develop an end-to-end RL baseline tailored for native visual reasoning. Standard stochastic perturbations in group-relative optimization often fail to induce meaningful semantic exploration, as they typically alter low-level appearance while preserving the underlying decision. The figure below compares Coefficients-Preserving Sampling (CPS) with conventional Flow-SDE sampling. Increasing SDE stochasticity introduces severe visual corruption and background artifacts. In contrast, CPS enables diverse semantic decisions and a wider range of rewards while largely preserving clean shapes and backgrounds.

The authors adopt CPS for RL exploration, formulating the reverse denoising process as:

$$x_{t - \Delta t} = (1 - (t - \Delta t)) \widehat{x}_{0} + (t - \Delta t) \cos \left(\frac{\eta \pi}{2}\right) \widehat{x}_{1} + (t - \Delta t) \sin \left(\frac{\eta \pi}{2}\right) \epsilon$$xt−Δt​=(1−(t−Δt))x0​+(t−Δt)cos(2ηπ​)x1​+(t−Δt)sin(2ηπ​)ϵ

where $\widehat{x}_{0}$x0​ and $\widehat{x}_{1}$x1​ denote the predicted clean sample and predicted noise, $\epsilon \sim \mathcal{N}(0, I)$ϵ∼N(0,I), and $\eta$η controls the stochasticity. Crucially, the coefficients satisfy $\cos^2(\eta \pi / 2) + \sin^2(\eta \pi / 2) = 1$cos2(ηπ/2)+sin2(ηπ/2)=1, ensuring that increasing $\eta$η replaces part of the predicted noise with newly sampled Gaussian noise without increasing the total noise coefficient. The training pipeline evaluates three variants on a 50K RL subset: Supervised Fine-Tuning (SFT) as a non-RL baseline, Reinforcement Learning with VLM Rewards (RLVLM) using a VLM judge, and Reinforcement Learning with Verifiable Rewards (RLVR) using the proposed deterministic scorers, isolating the impact of the verifiable reward signal on policy optimization.

Experiment

The experiments validate a new benchmark and training paradigm for native visual reasoning in generative models. A rule-based verifiable reward scorer achieves the highest agreement with human preferences among all evaluators while being the least expensive. Training on the VBVR-Pro dataset improves all nine open-source models across image, interleaved, and video generation, with video models excelling at state transitions and interleaved models offering a better accuracy-efficiency trade-off. Ablations and counterfactual diagnostics show that visual trajectory supervision contributes more than textual semantics, and that models learn reusable visual relations rather than memorizing templates. Reinforcement learning with verifiable rewards (RLVR) outperforms SFT and RLVLM, improving both in-domain and out-of-domain generalization, and case studies show RLVR enables later-step self-correction and more coherent multi-step spatial planning.

VBVR-Pro is a large-scale visual reasoning benchmark that surpasses existing resources in task count and data volume, uniquely covering video, image, and interleaved modalities under a unified distribution. It includes dedicated training and evaluation splits for reinforcement learning and employs fully verifiable task-specific scorers, contrasting with the VLM-based evaluation commonly used in other benchmarks. The dataset also expands visual complexity and reasoning depth, with newly developed tasks featuring more intricate scenes and multi-step reasoning compared to reworked ones. VBVR-Pro provides the largest training set among compared benchmarks and uniquely supports video, image, and interleaved generation tasks. It includes a dedicated split for reinforcement learning with 50 in-domain tasks and a separate bench of 100 tasks for evaluation. Newly developed tasks have higher visual complexity, such as a median of 80 distinct color regions per frame versus 12 for reworked tasks. Existing benchmarks often rely on VLM judges, which are shown to be inaccurate, expensive, and non-reproducible, whereas VBVR-Pro uses deterministic verifiable scorers.

Repeated evaluations of the same videos show that VLM judges change their scores for a large fraction of samples, while a deterministic verifiable scorer produces identical results every time. This inconsistency is especially problematic when scores are used as per-sample rewards, as it can introduce noisy optimization signals. The verifiable scorer is fully reproducible, with zero score changes across runs. VLM judges alter scores for over half of the samples, with some changing scores for more than 90% of cases. Score changes from VLM judges can be substantial, reaching up to 0.221 on a 0-1 scale.

The table lists configurations of open-source generative foundation models, distinguishing between image generation and interleaved text-image generation models. It reports total and activated parameters, with mixture-of-tokens (MoT) models showing sparse activation and dense models using all parameters. The cited experiment sections discuss reinforcement learning with verifiable rewards (RLVR) improving visual reasoning, including sustained performance gains and qualitative changes in reasoning trajectories. MoT models like BAGEL-7B-MoT and ThinkMorph-7B have 14B total parameters but activate only 7B, while SenseNova-U1-8B-MoT activates 8B of 16B. Dense models such as FLUX.2-dev and Qwen-Image-Edit use all their parameters, with 32B and 20B respectively. RLVR shows a clear upward trend across overall, in-domain, and out-of-domain metrics, unlike RLVLM which yields smaller and less stable gains. RLVR enables later-step self-correction in denoising trajectories, while the pre-RL model fixes its answer early and does not revise. RLVR improves temporal consistency in generated video frames, maintaining coherent multi-step spatial plans such as valid maze routes.

Top-tier proprietary image generation models outperform most open-source models on VBVR-Pro-Bench, with Seedream-5.0-Pro achieving the highest overall score. Task-specific training on VBVR-Pro-Dataset consistently improves all nine models across image, interleaved text-image, and video generation, with larger gains in in-domain settings than out-of-domain. Despite these improvements, even the strongest trained model remains below human-level performance. Proprietary models like Seedream-5.0-Pro and Seedance 2.0 lead the benchmark, while most academic open-source models lag substantially. Task-specific training yields an average overall gain of 0.290, with in-domain gains (+0.401) more than double out-of-domain gains (+0.179). The strongest trained model still falls short of human-level performance, indicating significant room for improvement.

An ablation study on interleaved modalities shows that removing intermediate visual states (using a single image instead of multiple) causes a clear drop in performance, while replacing meaningful reasoning text with placeholders has a much smaller effect. This indicates that explicit visual state transitions contribute more than intermediate textual reasoning for visual reasoning tasks. Compressing visual trajectories to a single image reduces overall scores by 0.024 and 0.111 for the two models, respectively. Replacing reasoning text with placeholders while keeping multi-image trajectories results in minimal performance changes (overall scores drop by only 0.001 and 0.009). The findings suggest that tracking visual state transitions is more critical than textual semantics for spatial and transformation tasks.

VBVR-Pro is introduced as a large-scale visual reasoning benchmark that uniquely covers video, image, and interleaved modalities, with fully verifiable deterministic scorers instead of unreliable VLM judges, which are shown to alter scores for over half of samples. Experiments demonstrate that reinforcement learning with verifiable rewards (RLVR) yields sustained gains across in-domain and out-of-domain tasks, enabling later-step self-correction and better temporal consistency in generation. Task-specific training on VBVR-Pro consistently improves all nine tested models, with larger in-domain gains, though even the strongest model remains below human performance. An ablation on interleaved modalities reveals that explicit visual state transitions matter far more than intermediate textual reasoning, as removing multi-image trajectories causes clear drops while replacing reasoning text with placeholders has minimal effect.