One-sentence Summary

Researchers from the University of Illinois Urbana-Champaign, Tsinghua University, the University of Waterloo, MIT, and other institutions introduce VGI-Bench, a benchmark of $27$27 tasks and $810$810 instances organized by a two-level taxonomy, to probe zero-shot visual reasoning in video generation models, and their evaluations show that even the strongest model, Seedance $2.0$2.0, achieves only $51.0\%$51.0%, revealing limited self-correction where later denoising steps refine early hypotheses rather than fix reasoning errors.

Key Contributions

• Introduces VGI-BENCH, a benchmark with 27 tasks and 810 instances, organized by a two-level taxonomy of task domains and skill tags, to evaluate visual reasoning in video generation models under realistic inputs, process-sensitive task design, and calibrated difficulty.
• Presents a broad evaluation of current image and video generation models, showing emerging reasoning abilities but far from reliable performance, with the strongest model (Seedance 2.0) achieving only 51.0% under the benchmark's criteria.
• Provides multi-faceted diagnostic analyses of video model reasoning, covering failure modes (e.g., physical collapse, rule violation, object/state inconsistency), input condition sensitivity, bounded transfer from synthetic fine-tuning, and denoising dynamics, revealing limited self-correction where later steps refine early hypotheses rather than fix reasoning errors.

Introduction

Video generation models are increasingly viewed not only as visual world simulators but also as potential visual reasoners, capable of expressing solutions to reasoning tasks through generated frame sequences. However, existing benchmarks for evaluating such zero-shot reasoning have notable gaps: they often rely on abstract or line-art inputs that mismatch the natural-image priors of these models, include tasks that can be solved without simulating scene evolution, and fail to control for task difficulty, leading to evaluations that may reflect domain mismatch or infeasibility rather than true reasoning limitations. To address these issues, the authors introduce VGI-BENCH, a benchmark designed to evaluate visual intelligence in video generation models through photorealistic inputs, process-sensitive tasks that require valid intermediate trajectories, and calibrated difficulty levels near the current capability boundary. The authors systematically evaluate a range of contemporary video and image generation models, finding that even the strongest model, Seedance 2.0, achieves only 51.0 under their criteria, with common failure modes including physical collapse, rule violation, and object or state inconsistency. Their diagnostic analyses further reveal that performance is sensitive to input conditions and visual style, that synthetic fine-tuning transfers only partially to realistic tasks, and that current models exhibit limited self-correction during denoising, providing a detailed view of the factors that shape and limit reasoning in video generation models.

Dataset

The authors construct VGI-BENCH, a benchmark for evaluating visual intelligence in generative models. The dataset is organized around a two-level taxonomy: four mutually exclusive domains and seven non-exclusive skill tags. The domains are Visual Organization, Physical Manipulation, Structured Puzzles, and Spatiotemporal Dynamics. The skill tags, inspired by visual cognition theories, include Spatial, Temporal, Planning, Attribute Grounding, Physics, Topology, and Affordance. This design allows for both coarse- and fine-grained diagnosis of model capabilities.

Task Collection and Composition

• Each task follows a unified input/output format: the model receives a text prompt and an input image as the first frame, then generates a video completing the specified visual procedure.
• Tasks are designed to be process-sensitive, meaning success depends on intermediate state evolution and rule-preserving trajectory, not just final-state correctness.
• Each task is instantiated at three difficulty levels, with roughly ten instances per level. Each instance consists of an input image, a text prompt, and task-specific evaluation criteria.
• Input images are collected from web images, existing datasets, or generated with image generation models such as GPT-Image-2 and Nano Banana Pro. Generated inputs undergo a human-in-the-loop review process to remove unintended artifacts. All images are standardized to a 16:9 aspect ratio.
• The text prompt specifies the task goal, relevant objects, attributes, allowed actions, prohibited shortcuts, and task-specific rules. It also includes task-agnostic controls on background, layout, camera motion, and video speed to keep the generated video focused on procedural reasoning.
• Each task is paired with a reference solution, either as an image (e.g., highlighting a valid path in a maze) or a textual description of the desired final state (e.g., for untying a knot). These references guide task proposal and criteria construction.

Quality Control and Filtering

• A pre-generation stage calibrates task difficulty. For each proposed task, the authors sample two easiest-level instances and test them on several state-of-the-art video generation models, including Sora2, Veo3.1, and Kling3.0. A task is accepted only if at least one model solves it and at least one model fails it; otherwise, the task design and input materials are revised. This filters out tasks that are trivially solvable or entirely infeasible for current models.
• Each task instance is manually reviewed to ensure it remains faithful to the intended goal, fits within typical video duration limits (5 to 10 seconds), and is described by a clear prompt. Unqualified instances are revised or discarded.

Evaluation Criteria and Data Usage

• The evaluation uses two complementary metrics: Completeness (Comp.) and Rubric Score (Rub.). Completeness measures global progress toward the task goal using a tiered standard (complete, partial, or failed) judged by a VLM on uniformly sampled frames at 2fps. Rubric Score measures local process validity via a fine-grained checklist, using a coarse-to-fine adaptive frame sampling strategy (starting at 4fps, then resampling flagged intervals at 8fps) with a sliding focus window of 10 frames.
• Each rubric item is scored with an inverse decay penalty, 1/(x + 1), where x is the number of violations. The Rubric Score is the average over all item-level scores.
• The Final Score multiplies Comp. by Rub., penalizing either type of failure. This ensures that both global goal completion and local rule adherence are jointly necessary for a high score.

Data Augmentation

• To extend the benchmark to image generative models, the authors repurpose some tasks into a single-image output format while preserving the task goal. For example, MAZE is adapted into drawing a valid path from start to goal, and RECOVER 2D NET is adapted into rendering the completed 3D structure. This branch enables comparison of static and procedural reasoning under related task goals.

Method

To ensure that each task in the benchmark is both meaningful and tractable for current video generation models, the authors introduce a multi-stage quality control pipeline. This pipeline operates before the final task instances are included in the benchmark and is designed to filter out tasks that are either trivially easy or entirely infeasible.

The first stage is a pre-generation calibration step. For each proposed task, the authors sample two instances at the easiest difficulty level and test them on several state-of-the-art video generation models, including Sora2, Veo3.1, and Kling3.0. A task is accepted only if at least one of these models successfully solves the sampled instance while at least one other model fails it. This acceptance criterion ensures that the task is neither trivially solvable by all models nor completely beyond the current capabilities of any model. If a task does not meet this condition, the authors revise the task design and its input materials. Importantly, this stage acts as a sanity filter: it verifies that a task can plausibly be expressed as a video process, rather than certifying that any model can fully solve it. The second stage involves manual review. Each task instance is examined by human annotators to confirm three properties: the instance remains faithful to the intended task goal, the instance fits within the typical video duration limit of 5 to 10 seconds, and the instance is described by a clear and unambiguous prompt. Any instance that fails these checks is either revised or discarded.

Once the task instances pass quality control, the authors define a dual-metric evaluation protocol that accounts for the process-sensitive nature of the tasks. Because a correct-looking final state is insufficient if the trajectory violates task rules, and a locally plausible video may still fail by making little progress toward the goal, the evaluation assesses both global completion and local process validity.

The first metric, Completeness, captures the global progress toward the task goal. Since different tasks have different goal states, the authors define a task-specific tiered standard that maps the generated video to one of three levels: complete, partial, or failed. A VLM-based judge receives a set of uniformly sampled frames at 2 frames per second (fps) along with the tiered standard, and returns the corresponding tier.

The second metric, Rubric Score, measures local process validity throughout the video. For each task, the authors design a fine-grained checklist that covers explicit rules and other constraints. Because some critical violations may occur only briefly, the evaluation uses a coarse-to-fine adaptive frame sampling strategy. The VLM-judge first inspects the video at a coarse sampling rate of 4 fps, checking it against the checklist and flagging intervals that require closer inspection. These flagged intervals are then resampled at a finer rate of 8 fps and re-evaluated, allowing the judge to capture transient violations without densely sampling the entire video. To keep each judgment focused, the authors adopt a sliding focus window of 10 frames with edge frame overlapping, so that only a small local segment is inspected at a time. Each rubric item is scored using an inverse decay penalty, $1/(x+1)$1/(x+1), where $x$x is the number of violations for that item. This decay reflects the intuition that once a violation occurs repeatedly, additional occurrences have a diminishing marginal effect. The Rubric Score is the average over all item-level scores. The authors note that other monotonic decay functions, such as exponential decay, are also applicable and preserve the same qualitative trend.

The final score for a video combines the two metrics multiplicatively: the Final Score is the product of Completeness and Rubric Score. This multiplicative design penalizes either type of failure. For example, an almost static video preserves most local constraints, yielding a high Rubric Score, but makes little progress toward the task goal, resulting in a low Completeness score. Conversely, a video may appear to reach the target state, giving a high Completeness score, while violating the rules, leading to a low Rubric Score. The aggregation treats both aspects as jointly necessary conditions, preventing either aspect alone from dominating the evaluation.

To extend the benchmark as a testbed for reasoning in image generative models, the authors repurpose a subset of the video tasks into a single-image output format while preserving the original task goal. This adaptation does not contradict the process-sensitive task design: the original video tasks evaluate whether a video model can express the procedure through temporal state evolution, while the image version asks whether an image model can infer and render the target state or visual solution. For example, the MAZE task is adapted into drawing a valid path from the start to the goal, and the RECOVER 2D NET task is adapted into rendering the completed 3D structure. This branch enables a direct comparison of static and procedural reasoning under related task goals.

Experiment

The evaluation uses two complementary metrics: completeness for global goal progress and a rubric score for local process validity, combined multiplicatively to require both. Commercial video models outperform open-source ones but remain far from solved, with structured puzzles and topology/temporal skills being the weakest areas. The VLM-based evaluator shows strong agreement with human judgments, and ablations confirm that adaptive frame sampling and sliding focus windows are both necessary. Fine-tuning on synthetic data transfers unevenly, improving skills well represented in the training distribution while leaving physical interaction and strong temporal dependencies largely unchanged, and analysis of denoising dynamics reveals that self-correction is rare, with revisions mostly moving between incorrect states rather than toward the correct one.

The benchmark comparison shows that most existing video benchmarks lack either photorealistic inputs, process-sensitive tasks, or difficulty control, while the proposed benchmark uniquely combines high reasoning demand, realistic appearance, process sensitivity, and difficulty calibration. This design aims to reduce visual domain mismatch and ensure that tasks require explicit rollout reasoning rather than being answerable from static inputs. Only the proposed benchmark combines high reasoning demand with photorealistic inputs, process-sensitive tasks, and difficulty control. Several existing benchmarks use abstract or synthetic inputs, which can cause domain mismatch and weaken evaluation validity. Most benchmarks do not require process-sensitive reasoning, meaning they can be answered without simulating scene evolution. Difficulty control is absent in many benchmarks, leading to tasks that are too hard to be diagnostic.

Commercial video generation models consistently outperform open-source models, with Seedance-2.0 achieving the highest overall score, yet all models still struggle significantly, particularly on structured puzzles and tasks requiring topology and temporal reasoning. The gap between commercial and open-source performance is evident across both easy and hard levels, and even the best models fall far short of perfect success rates. Seedance-2.0 leads overall with a score of 51.0, but all models remain far from solving the tasks. Commercial models outperform open-source models across all categories, with the largest gaps in Visual Organization. Structured Puzzles is the most challenging domain, exposing failures in multi-step rules and state tracking. Topology and Temporal are the weakest skill dimensions, indicating issues with connectivity preservation and multi-step state tracking. Strict success rates, requiring perfect completeness and rubric scores, are notably lower, highlighting the difficulty of the benchmark.

The VLM-based evaluator's reliability is measured against human annotations, with the full method achieving the highest agreement. Ablations removing either adaptive frame sampling or the sliding focus window reduce performance, and swapping the base model for cheaper alternatives also degrades reliability. The full evaluator with Gemini-3-Flash achieves the highest AUC and pairwise accuracy, outperforming all ablations. Removing adaptive frame sampling or the sliding focus window lowers both AUC and pairwise accuracy, confirming both components are necessary. Replacing the base model with cheaper alternatives (GPT-5-mini, Claude-Haiku-4.5, Qwen3.6-Plus) leads to substantial drops in agreement, with the weakest model showing near-chance performance.

Commercial image models outperform open-source ones on the adapted static goal-state subset, with Nano-Banana-Pro leading the average success rate. Performance drops from easy to hard tasks across most models, indicating a clear difficulty gradient. This setting only tests target-state inference and rendering, not the temporal process validity evaluated in the video benchmark. Nano-Banana-Pro achieves the highest average success rate among commercial models, while open-source models trail significantly. Most models show lower success rates on hard tasks compared to easy ones, confirming a meaningful difficulty gradient. The commercial-open-source performance gap mirrors the video branch, but all models remain far from perfect on the adapted subset.

Fine-tuning video generation models on synthetic data improves reasoning performance across all structural overlap groups, with the largest gains on tasks most aligned with the training distribution. However, transfer is limited by structural coverage, and task-level improvements are uneven, with some non-overlap tasks still benefiting while certain capabilities remain hard to improve or degrade. Performance gains decrease as structural overlap with the training distribution decreases for all three base models. The VBVR-Wan2.2 model shows the largest absolute improvement on overlap tasks, while non-overlap gains are smaller but still positive. The VBVR-Wan2.1 model actually regresses on non-overlap tasks, indicating negative transfer in that setting. Task-level outcomes vary: for instance, UN-TIE_KNOT shows little change in success rate but a large rubric score increase, suggesting more controlled behavior rather than new reasoning ability. Improvements concentrate on skills well represented in synthetic data, such as planning and spatial reasoning, while physical interaction and strong temporal dependency remain hard to improve.

The proposed benchmark uniquely combines photorealistic inputs, process-sensitive tasks, difficulty control, and high reasoning demand, addressing gaps in existing video benchmarks that often rely on abstract inputs or lack process sensitivity. Evaluations show commercial video generation models consistently outperform open-source ones, yet all struggle significantly, especially on structured puzzles and topology or temporal reasoning, with strict success rates notably low. A VLM-based evaluator achieves the highest reliability when using adaptive frame sampling, a sliding focus window, and a strong base model, as ablations of these components degrade agreement with human annotations. On an adapted static image subset, commercial models again lead but remain far from perfect, and fine-tuning video models on synthetic data yields uneven gains, improving reasoning on overlap tasks while risking negative transfer on non-overlap ones.