One-sentence Summary

Researchers from Xiamen University, Nanyang Technological University, and the Key Laboratory of Multimedia Trusted Perception and Efficient Computing introduce RoTA-Bench, an interactive 3D-to-2D projection alignment benchmark that builds silhouette-preserving 3D objects from 2D icons with reversible initialization to guarantee solvability, and find that GPT-5.6-Sol succeeds on only 17% of a stratified 100-task test set versus 100% for 20 undergraduate participants, exposing a substantial human-model gap in history-sensitive fine-grained 3D control.

Key Contributions

• RoTA-Bench is an interactive benchmark for evaluating whether vision-language models can turn 2D observations into 3D understanding that supports precise action through closed-loop 3D-to-2D projection alignment. It builds silhouette-preserving 3D objects from 2D icons and guarantees solvability through reversible initialization, yielding 674 tasks and a stratified 100-task test set.

• GPT-5.6-Sol succeeds on only 17% of tasks, while 20 undergraduate participants collectively complete the test set four times with 100% success, revealing a substantial human-model gap.

• Image-space and pose-space progress can diverge: at matched states, the model more often improves silhouette overlap while worsening pose alignment than humans (19.4% vs. 10.9%). Removing interaction history leaves 96.8% of the model's actions at a coarse 30-degree step size and sharply reduces observed task success.

Introduction

Vision-language models are increasingly used in visual question answering, interactive agents, and embodied systems where they must convert spatial understanding into precise physical action, yet they still struggle with 3D positions, orientations, and spatial relationships. Existing static benchmarks evaluate spatial reasoning through passive, single-turn questions without action execution or closed-loop visual feedback, while interactive benchmarks tend to emphasize high-level planning, tool use, or pathfinding over discrete axis-aligned objects rather than perceiving rotational pose variation. To address this gap, the authors introduce RoTA-Bench, an automatically constructed benchmark for interactive 3D-to-2D projection alignment. Its pipeline extrudes 3D objects from 2D icons, initializes orientations through reversible rotations to guarantee solvability, and requires models to iteratively select among 24 discrete rotation actions to align an object silhouette with a target view, yielding 674 tasks with a stratified 100-task test set.

Dataset

The authors construct RoTA-Bench as an automated benchmark for evaluating whether vision-language models can turn 2D observations into 3D understanding for precise action. The dataset is built in three phases: task generation, reference search, and dataset construction.

• Sources and composition

  • 2D vector icons are curated from the open-source Game-icons library.
  • Each icon is rasterized into a high-resolution binary mask after filtering on connectivity, occupancy, aspect ratio, and foreground point count.
  • The planar icon point set is scaled within a disk of radius $r$r and symmetrically extruded along the depth axis using a spherical envelope $h(x,y)=\sqrt{\max(r^2-x^2-y^2,0)}$h(x,y)=max(r2−x2−y2,0)​.
  • The resulting solid is discretized as a dense 3D point cloud.
  • Visual observations are rendered via orthographic projection with depth-dependent shading and Gaussian smoothing to provide monocular depth cues and reduce discretization gaps.
  • The task generation pipeline yields 674 canonical tasks, each from a unique source icon.

• Key subset details

  • The canonical target pose is fixed to the identity rotation.
  • Initial poses are generated through 6 to 16 discrete 30-degree rotations.
  • Rotation directions are sampled uniformly from six candidate camera directions, with immediate inverse rotations forbidden to avoid cancellations.
  • Initial poses already projection-aligned are discarded.
  • The final frozen test set contains 100 tasks.
  • Test tasks are selected using proportional stratified sampling across reference path lengths of at least 4 steps, via Hamilton’s method and deterministic hash sorting.
  • The remaining 574 tasks are reserved for future training and research.

• How the data is used

  • The paper uses the 100-task stratified test split for closed-loop evaluation of vision-language models.
  • Reference paths are computed with pose-guided beam search and 30-degree-restricted breadth-first search, producing shorter trajectories than the raw reversed generation paths.
  • Because all rotations are invertible, executing the reversed generation sequence restores the canonical pose, ensuring 100% reachability and solvability by construction.
  • No training mixture ratio is used in the current benchmark; only the frozen test split is evaluated, while the remaining tasks are held for future work.

• Additional processing details

  • The spherical extrusion bounds every rotated pose within $x^2+y^2+z^2 \le r^2$x2+y2+z2≤r2, which prevents viewport clipping and fixes the camera scale across all tasks.

Method

The authors formulate the interactive projection-matching task to evaluate whether vision-language models can translate 2D observations into 3D understanding for precise action. Each task instance involves a rigid 3D object $G \subset \mathbb{R}^3$G⊂R3 centered at the origin, an initial pose $R_0 \in \mathrm{SO}(3)$R0​∈SO(3), and a canonical target pose $R^\star = I_3$R⋆=I3​. At each step $t$t, the environment provides an observation triplet $O_t = (I^\star, V_t, I_t)$Ot​=(I⋆,Vt​,It​), where $I^\star$I⋆ is the invariant target silhouette, $I_t$It​ is the current projection silhouette, and $V_t$Vt​ is a depth-shaded auxiliary view providing missing depth and occlusion cues. The agent selects an action $a_t$at​ from a discrete vocabulary consisting of 24 camera-fixed rotation actions and a terminal STOP command. These actions span pitch, yaw, and roll across four step sizes to enable coarse and fine adjustments. The object pose updates via $R_{t+1} = Q(a_t)R_t$Rt+1​=Q(at​)Rt​. Task success is determined by the projection Intersection over Union (IoU) between the current and target silhouettes, requiring $s_t = \mathrm{IoU}(I_t, I^\star) \ge 0.90$st​=IoU(It​,I⋆)≥0.90.

The automated pipeline constructs the benchmark through three sequential phases: task generation, reference search, and dataset construction. In the task generation phase, 2D vector icons are rasterized into binary masks and converted into 3D objects. To strictly bound the spatial profile under arbitrary rotations, the planar point set is scaled within a disk of radius $r$r and symmetrically extruded along the depth axis under a spherical envelope defined by $h(x, y) = \sqrt{\max(r^2 - x^2 - y^2, 0)}$h(x,y)=max(r2−x2−y2,0)​. This spherical invariance prevents viewport clipping and fixes the camera scale globally. Initial poses are generated via a sequence of discrete rotations, and because every rotation in $\mathrm{SO}(3)$SO(3) is invertible, executing the reversed sequence guarantees 100% reachability and task solvability.

In the reference search phase, the authors compute reference paths using pose-guided beam search and $30^\circ$30∘-restricted breadth-first search to discover significantly shorter trajectories than the raw reversed generation paths. Finally, a stratified test set is frozen across reference path lengths using proportional stratified sampling. During closed-loop evaluation, the solver receives visual observations and interaction history to select rotations. The environment applies the chosen rotation, updates the 3D state, and renders new observations. The episode terminates upon reaching the IoU threshold, voluntary halting, reaching the maximum action budget, or stagnation.

Experiment

The experiments evaluate interactive 3D projection alignment on RoTA-Bench across multiple vision-language models, agent frameworks, human participants, and stochastic or greedy baselines, using task success, IoU, pose-error, and rotation direction and angle diagnostics. Frontier models perform poorly while humans solve all tasks, and a pose-greedy baseline succeeds fully whereas an IoU-greedy baseline often fails, indicating that reliable 3D pose estimation is essential. Failure analysis shows models view enough surface but choose rotation directions and angles less effectively than humans, and ablations reveal that removing interaction history severely degrades fine-grained angle control. Action-level comparisons further indicate that models rely more on image-space overlap than on underlying pose discrepancy, unlike humans who balance both signals.

Human participants complete all tasks with high final overlap and accurate action choices, while the best frontier systems succeed in under one fifth of tasks and reach much lower final IoU. Several models view a similar fraction of the object surface as humans, but their action selection is worse: effective directions are near random and angle precision is lower. Pose-greedy actions achieve perfect success in relatively few steps, highlighting the gap between current model action selection and effective 3D alignment. Human performance is nearly perfect, but frontier models reach only single-digit to low-teens success rates. Several frontier models achieve visual surface coverage close to humans, so failure is not primarily due to insufficient exploration. Effective-direction rates for frontier models are far below human rates and near random selection. Even when models choose effective directions, their selected angles are less accurate than human angles, with larger angle gaps and lower best-angle rates. A pose-greedy baseline attains perfect success with fewer steps, while an IoU-greedy baseline is much weaker, indicating that pose-aware action selection is critical. SpatioLM-Understanding-InternVL3.5 suffers a policy collapse, with most actions degenerating into one repeated yaw action and final IoU below random exploration.

Ablating auxiliary views or interaction history degrades GPT-5.6-Sol's task success and final IoU, with history removal causing the larger drop and bringing final IoU close to the uniform random baseline. Effective direction rate remains nearly unchanged across conditions, while rotation-angle selection shifts strongly toward 30-degree rotations when inputs are removed. The correlation between pre-action IoU and rotation angle magnitude weakens most without interaction history, indicating that history mainly supports fine-grained rotation magnitude decisions. Removing interaction history reduces task success to zero and lowers final IoU to near the uniform random baseline, a larger drop than removing the auxiliary view. Effective direction rate stays nearly unchanged across ablation conditions even as success and final IoU decline. Without interaction history, 30-degree rotations become the dominant action and fine 1-degree adjustments nearly disappear, weakening the IoU-angle correlation.

Across matched states, humans entered actions with fewer prior rotations and later achieved larger average gains in both silhouette overlap and pose error reduction than the model. Human actions more often reduced underlying pose error even when 2D overlap temporarily dropped, while the model more often improved overlap at the expense of pose accuracy. Humans started matched actions with fewer prior rotations and similar visible coverage compared with the model. Human outcomes favored pose-error reduction more than the model, with more overlap-decreasing but pose-error-decreasing actions and far fewer overlap-improving but pose-error-increasing actions.

The experiments evaluate human and frontier vision-language models on a 3D object alignment task, with ablations and matched-state analyses isolating the role of action selection and input history. Humans achieve near-perfect success and accurate pose-aware actions, while frontier models fail on most tasks despite viewing similar surface coverage, showing near-random effective direction selection and imprecise rotation angles; a pose-greedy baseline confirms that better action selection alone would solve the task. Ablations indicate interaction history mainly supports fine-grained rotation magnitude decisions rather than direction choice, and matched-state comparisons show humans prioritize pose error reduction over 2D overlap, whereas models often improve overlap at the cost of pose accuracy.