One-sentence Summary

University of Southern California researchers introduce ActiveVision, a benchmark of 17 tasks across 3 categories that forces repeated visual perception, revealing that frontier multimodal LLMs like GPT-5.5 ($10.6\%$10.6%) and Claude Fable 5 ($3.5\%$3.5%) far underperform humans ($96.1\%$96.1%), even when models attempt to write and run their own vision code, thereby highlighting the lack of active observation and motivating architectures and training objectives that close the perception–reasoning loop.

Key Contributions

• ActiveVision is presented, a benchmark that isolates active visual observation through 17 tasks across 3 categories, each requiring iterative scanning, tracing, and comparison with photorealistic renderings that preserve exact geometry and resist single-language-description shortcuts.
• Frontier MLLMs collapse on the benchmark: GPT-5.5 solves only 10.6% and Claude Fable 5 solves 3.5% of items, compared to 96.1% by three human participants, with even increased reasoning effort barely improving performance.
• Equipping models with the ability to write and execute vision code yields unequal gains: the strongest agent solves only half of the items while taking far more time and compute than unaided humans, and code often fails on realistic imagery, exposing a fundamental lack of robust active perception.

Introduction

Multimodal large language models (MLLMs) have saturated static-image benchmarks, yet real-world tasks in robotics, manufacturing, and scientific discovery demand that a system iteratively revisit visual evidence to form, test, and refine hypotheses—a process known as active observation. Existing evaluations do not capture this requirement because they rely on single-image question answering or captioning that can often be solved from a fixed text description of the scene, leaving the need for perceptual hypothesis-testing loops unmeasured. The authors introduce ActiveVision, a benchmark that isolates active observation across 17 tasks spanning distributed scanning, sequential traversal, and fine-grained visual comparison. Tasks are procedurally generated with exact underlying geometry and photorealistically rendered so that no single language summary carries the answer, forcing models to re-examine pixels as they reason. Frontier MLLMs achieve at most 10.6% accuracy while humans average 96.1%, and even coding agents that write vision tools reach only 50.6%, revealing a large and persistent gap that positions active vision as a distinct capability current models lack.

Dataset

The authors introduce ActiveVision, a diagnostic dataset designed to probe whether vision-language models can perform active, serial visual operations beyond a single glance. The dataset is not a fixed collection of pre-existing images; it is procedurally generated on the fly from a suite of task-specific generators.

Dataset composition and sources

• 17 tasks organized into three families, each targeting a core perceptual routine:
  • Distributed Scanning: spatially distributed signals (dots, strokes, regions, graph faces) that must be exhaustively located and counted.
  • Sequential Traversal: connected structures such as arrow chains, tangled curves, or winding tubes that require step-by-step following.
  • Visual Attribute Transfer: fine-grained comparison of properties (length, curvature, thickness, color pattern, etc.) between a reference region and multiple candidates.
• Every task isolates a discriminative visual state whose information content exceeds what a single language description can losslessly carry, forcing models to repeatedly inspect pixel-level evidence.

Task instantiation and processing

• Each task generator produces a synthetic scaffold (using Matplotlib), a natural-language question, and a ground-truth answer from a deterministic seed, making instances fully reproducible.
• To avoid cartoon-clean inputs, a two-stage rendering pipeline is applied:
  1. A procedural Matplotlib scaffold is created with attached ground truth.
  2. The scaffold is re-rendered with GPT-image-2 using a task-specific prompt that maps the abstract primitives to a photorealistic, real-world setting.
• Only the final photorealistic rendered image is shown to the model; the scaffold and rendering prompts remain hidden.
• Answer distributions are deliberately broad and flat per task, so neither the most frequent answer nor the task identity can serve as a shortcut.

How the data is used

• ActiveVision is used purely as an evaluation benchmark for multimodal models; no training split or mixture ratios are specified (the dataset is designed for zero-shot or few-shot probing, not fine-tuning).
• Models receive only the rendered image and the question, and must perform the targeted active-vision operation.
• The dataset is procedurally unlimited — generators can produce arbitrarily many examples — but the paper does not fix a static dataset size, treating it as a controlled diagnostic environment rather than a finite test set.

Method

The authors design ActiveVision around three elemental operations of human vision: exhaustive enumeration, curve tracing, and fine-grained comparison. These operations are instantiated as three distinct task families, each targeting a specific dimension of active observation.

The first family, Distributed Scanning, requires the model to find and accumulate spatially distributed local signals, such as dots or regions, where difficulty scales with the number of signals. The second family, Sequential Traversal, involves following a connected structure step by step while maintaining position and direction, challenging the model to avoid gestalt interpolation. The third family, Visual Attribute Transfer, demands fine-grained comparison of visual properties across different regions, testing the model's ability to extract and match attributes without relying on linguistic priors.

To ensure these tasks genuinely require iterative perception, the authors enforce a core design principle: every task instance carries discriminative visual state whose information content exceeds what a single language description can losslessly carry. This is achieved through three properties. First, items are placed at arbitrary, continuous positions rather than on a grid. Second, region boundaries and shapes are synthesized freshly for every instance using random Fourier harmonics or splines, creating a continuous and high-variance silhouette space. Third, the routes to be followed are smooth random splines with numerous inflection points. Together, these properties make iterative perception the only natural solution path, as a one-pass observer cannot reliably preserve the necessary visual state.

The authors instantiate this design with 17 task generators that produce synthetic scaffolds, questions, and ground-truth answers from deterministic seeds. To place these operations in realistic settings and avoid the cartoon-input confound, they employ a two-stage generation pipeline.

In the first phase, a deterministic Python generator emits a geometric scaffold with full ground truth attached. In the second phase, a task-specific prompt is used to re-render the scaffold into a photorealistic image using GPT-image-2. This re-rendering maps the procedural primitives to a real-world setting without altering positions, counts, or topology. This pipeline ensures that the perceptual difficulty is dominated by the task's discriminative structure rather than an unfamiliar rendering style, and it bounds what tool-using scripts can recover, making the diagnosis externally meaningful for downstream applications.

Experiment

The benchmark probes active visual reasoning via 85 photorealistic renderings, with a human exact-match baseline of 96.1%. Frontier multimodal LMs solve at most 10.6%, and increasing chain-of-thought reasoning effort yields negligible improvement; models characteristically undercount crowded scenes, lose spatial tracing in the first steps, and default to answering “same” in comparisons. Coding agents that invoke classical vision tools reach 50.6% but only on tasks amenable to algorithmic reduction, while traversal and counting remain largely unsolved because the agents lack the perceptual ability to catch their own mistakes.

ActiveVision defines 17 task generators across cognitive axes that isolate perceptual operations such as counting graph components, following arrow chains, and identifying tangled loops. Each generator yields reproducible instances with broad answer distributions to prevent shortcut learning, and the images are re-rendered photorealistic to match real-world settings like medical scans and satellite imagery. The task set spans distributed scanning (e.g., counting bounded faces, connected components, regions, and singleton shapes) and sequential traversal (e.g., arrow chain following). Agentic tool-use evaluation shows attribute transfer tasks are largely solved while traversal tasks remain near zero, and all agents fail Tangled Loop Counting, underscoring the perceptual difficulty.

Frontier multimodal models achieve at most 10.6% exact-match accuracy on ActiveVision, with the best model failing entirely on most tasks. Human participants average 96.1% accuracy, exposing a large gap between machine and human visual reasoning in complex scenes. All six frontier models scored zero on Tangled Loop Counting, while the human average was perfect. GPT-5.5, the top model, solved only 9 of 85 items and scored zero on 11 of 17 tasks, whereas human participants averaged 96.1% accuracy.

Models evaluated on visual difference tasks display a strong conservative bias, frequently defaulting to a 'none' answer. This leads to high miss rates for real differences while false alarms remain low, even in runs where almost every item is answered with 'none', suggesting a safe response strategy rather than perceptual accuracy. Miss rates are high across all runs, with several configurations missing every real difference, while false alarm rates stay low, often at zero in the most conservative runs. Runs that answer 'none' on nearly every item achieve zero false alarms but miss all true differences, exposing a response bias away from claiming a difference when visual comparison becomes too difficult.

Agentic systems that replace visual perception with code execution incur severe resource costs: each item consumes 12–15 minutes of wall-clock time and $2.74–$2.74–7.63 of compute, while using dozens of tool calls and tens of thousands of output tokens. The most capable agent answers just over half the benchmark, far below the 96.1% accuracy of unaided humans who need only about half a minute per item and zero tool calls. This efficiency and accuracy gap highlights the fundamental limitation of substituting active vision with tool-orchestrated reasoning on noisy real-world imagery. Human participants achieved 96.1% accuracy in 0.56 minutes per item with no tool calls, whereas the strongest agent reached only 50.6% accuracy while taking 13.9 minutes and spending $7.63 per item. Even the least expensive agent cost$7.63peritem.Eventheleastexpensiveagentcost2.74 per item and delivered only 37.6% accuracy, far below human performance, showing that tool use does not close the gap. Agents spent 12.5–14.7 minutes on each item, about 25 times longer than the half-minute human annotation time, with each item generating tens of thousands of output tokens.

The ActiveVision benchmark uses 17 perceptual task generators with photorealistic imagery to isolate active scanning and traversal operations. Frontier multimodal models and agentic systems show near-zero performance on core tasks like Tangled Loop Counting, while humans achieve near-perfect accuracy; even code-equipped agents answer only about half the benchmark, incurring extreme time and cost overhead. Additionally, models exhibit a strong conservative bias in visual difference tasks, frequently defaulting to 'none' and missing almost all real differences.