One-sentence Summary

Wuhan University, DreamX Team, and Alibaba Group propose Urbex, an agentic vision-language framework that casts city-scale 3D grounding as active spatial search in an interactive multi-view environment, allowing a VLM agent to query landmarks, zoom into local regions, render oblique views, and commit grounded bounding boxes; trained with reinforcement learning on CityRefer using smooth localization and lightweight shaping rewards, Urbex achieves the best Hit@1 and IoU on CityRefer, supports candidate-free grounding without ground-truth boxes, and improves zero-shot robustness on STPLS3D-Refer.

Key Contributions

• The paper introduces Urbex, an agentic vision-language framework that recasts city-scale 3D grounding as active spatial search in an interactive multi-view environment, where a VLM agent can query landmarks, zoom into local regions, render oblique views, and commit grounded bounding boxes instead of ranking precomputed candidate instances.
• It develops a reinforcement learning formulation for the tool-use policy on CityRefer, using smooth localization rewards and lightweight shaping rewards to encourage accurate and efficient exploration under a limited interaction budget.
• Experiments show that Urbex attains the best Hit@1 and IoU on CityRefer, performs candidate-free grounding without ground-truth boxes, runs substantially faster than the strongest city-scale ranking baseline, and yields improved zero-shot cross-scene robustness on STPLS3D-Refer.

Introduction

Grounding 3D objects from natural language is a core capability for embodied intelligence, with applications in autonomous driving, robotic navigation, and aerial inspection. City-scale grounding is especially challenging because outdoor scenes contain millions of points and queries often require reasoning over fine-grained appearance and long-range spatial cues. Existing indoor methods downsample point clouds and lose critical geometry, while city-scale candidate-ranking approaches depend on precomputed proposals, incur inference cost that grows with the number of candidates, and reduce localization to static retrieval. The authors propose Urbex, an agentic framework that recasts city-scale grounding as active spatial search. A vision-language model iteratively queries landmarks, zooms into promising regions, inspects oblique views for 3D confirmation, and commits a bounding box, with reinforcement learning optimizing both final localization accuracy and intermediate search behavior.

Method

Method

Urbex formulates city-scale 3D grounding as active spatial search rather than static candidate retrieval. The authors expose a large 3D scene through a small number of coordinate-aligned visual observations that a vision-language model can inspect and act upon. The method has three main components: an interactive environment that converts city-scale geometry into a VLM-operable multi-view space, an agentic reasoning loop that uses structured tool calls, and a reinforcement learning objective that learns when and how to use those tools.

Interactive environment for city-scale 3D grounding

Given a city-scale 3D scene $S$S and a natural-language query $q$q, the goal is to localize the referred object $o^{\star}$o⋆ and output its spatial extent. Instead of assuming a candidate set and learning a scoring function over candidates, Urbex receives no candidate set at inference time. The environment exposes the scene through a set of views $\mathcal{V}$V that the agent can extend through tool calls. The policy predicts a free-form box $\hat{b}_v$b^v​ in a known bird's-eye-view-aligned view $v$v, and the environment maps this prediction to a world-space box. Ground-truth boxes are used only for training rewards and evaluation, not as inference inputs.

Each view $v \in \mathcal{V}$v∈V contains an image observation $I_v$Iv​ and a mapping $T_v$Tv​ from the view coordinate frame to world coordinates. The initial observation is a global bird's-eye-view image $I_{\mathrm{global}}$Iglobal​. The agent expresses boxes in a normalized 2D coordinate frame, and the environment maintains the corresponding view-to-world transform. This gives the VLM a consistent spatial interface while preserving metric grounding in the original 3D scene.

The environment provides three complementary tools. Landmark Query resolves a named landmark, such as a building or road name, and returns its world position and its projection in a selected view. This supports global spatial anchoring when the expression contains proper nouns. Zoom-in maps a selected box to the original high-resolution bird's-eye-view image and returns a local crop with a new view identifier and transform. Because the original bird's-eye-view image can exceed 10,000 pixels on the long side, zooming recovers detail that is absent from the downsampled global observation. Oblique Rendering renders a selected candidate region from a specified cardinal direction, producing a 3D-aware view that helps verify object shape, height, and spatial placement. The oblique view is used as supporting evidence rather than as the coordinate frame for the final box submission.

Agentic reasoning with multi-view tool use

The policy $\pi_{\theta}$πθ​ is a VLM that receives the current observation $o_t$ot​ and the interaction history $h_t$ht​. The history contains the original query, previous tool calls, returned view identifiers, coordinate conventions, and visual observations. At each step, the policy produces a reasoning trace and an action:

The action $a_t$at​ is either a structured tool call or a final answer. A tool call can specify a landmark name, a source view, a 2D crop box, a candidate box, or an oblique rendering direction. The environment executes the tool and appends a textual response and, when applicable, a new image observation to the history.

This inference procedure decomposes into global search, local observation, and candidate confirmation. The agent first parses the expression to identify target attributes and spatial anchors in the bird's-eye view, then zooms into promising regions to resolve small objects or dense local context, and finally renders oblique views when top-down evidence is insufficient for 3D verification. Because tool responses are appended to the history, the model can refer back to earlier views and maintain coordinate consistency while switching between global and local evidence. This protocol can be used directly with a prompted VLM to form a zero-shot tool-use baseline, but the search space remains large. The model must learn not only how to interpret each view, but also when to query landmarks, zoom, render, or stop.

Reinforcement learning for active spatial search

An episode is modeled as a finite-horizon Markov decision process. The state consists of the interaction history, the current views, and tool feedback. The action space includes valid tool invocations and final answer submission. Given an action and a scene, the environment transition is deterministic, while trajectory-level stochasticity comes from policy sampling. The episode ends when the agent submits an answer or reaches the maximum number of assistant turns.

The terminal reward uses the submitted world-space box $\hat{b}$b^ and the ground-truth box $b^{\star}$b⋆:

Here $d(\hat{b}, b^{\star})$d(b^,b⋆) is the center distance in the ground plane, $T$T is the number of assistant turns, and $T_{\max}$Tmax​ is the interaction budget. The IoU term rewards precise localization. The distance term provides a continuous learning signal even when overlap is zero, which is common early in training on large city-scale scenes. The step cost discourages unnecessarily long trajectories.

In addition to the terminal reward, lightweight process rewards shape intermediate tool-calling behavior:

For Zoom-in, the reward is positive when the zoom center moves closer to the target compared with the previous zoom and negative when it moves away, with a small cap to prevent it from dominating the terminal objective. For Oblique Rendering, the reward is proportional to the overlap between the rendered candidate region and the target region, again capped as a shaping signal. Invalid tool names, malformed arguments, impossible boxes, and tool execution errors receive negative rewards. These tool-level rewards are privileged training signals derived from ground-truth annotations and are never included in the agent's text observations, so they do not leak target information at inference time.

Urbex is trained with Group Relative Policy Optimization. To avoid the memory overhead of a separate value network, GRPO estimates the baseline directly from multiple concurrent rollouts. For each input query and initial environment state, the behavior policy $\pi_{\theta_{\mathrm{old}}}$πθold​​ samples a group of $G$G trajectories $\{\tau_1, \tau_2, \dots, \tau_G\}${τ1​,τ2​,…,τG​}. After evaluating the cumulative reward $R(\tau_i)$R(τi​) for each trajectory, the group-relative advantage is computed as:

where $\mu_R$μR​ and $\sigma_R$σR​ are the mean and standard deviation of the group rewards. The policy is optimized by maximizing the clipped GRPO objective:

where $r_{i,t}(\theta) = \pi_{\theta}(a_{i,t} \mid s_{i,t}) / \pi_{\theta_{\mathrm{old}}}(a_{i,t} \mid s_{i,t})$ri,t​(θ)=πθ​(ai,t​∣si,t​)/πθold​​(ai,t​∣si,t​), $\epsilon_c$ϵc​ is the clipping threshold, and $\beta$β controls the token-level KL penalty to the reference policy $\pi_{\mathrm{ref}}$πref​.

By relying solely on this RL objective without supervised trajectory demonstrations, the model explores the multi-view environment and learns a task-specific tool-use policy. In particular, it learns when to query landmarks, how large a zoom region should be, whether an oblique view justifies its step cost, and when the accumulated evidence is sufficient to commit to a final bounding box.

Experiment

The experiments evaluate Urbex on CityRefer for in-domain grounding and STPLS3D-Refer for zero-shot cross-scene generalization, comparing it with general multimodal models, indoor grounding methods, and city-scale candidate-ranking baselines under With-GT and Without-GT protocols. Urbex achieves the best overall accuracy on most metrics while being about ten times faster than the strongest ranking baseline, and it transfers best to unseen scenes, indicating a learned reusable search strategy. Ablations show that each tool, especially Landmark Query, and each reward component contribute to active search, with an eight-turn budget providing the best accuracy-efficiency tradeoff. Further analysis suggests the model benefits from geographic cues and bounded verification, but its predicted boxes can remain larger than the target, motivating post-selection box refinement.

On CityRefer, Urbex achieves the strongest candidate-free localization under the with-GT setting, leading on both test splits in Hit@1 and Hit@3 and on one split in Hit@5. Prompted closed-source VLMs using the same tool environment reach non-trivial accuracy but remain far below Urbex, while indoor grounding methods transfer poorly to city-scale scenes. Urbex is also substantially more efficient than the strongest ranking baseline because it inspects a small number of informative regions instead of exhaustively scoring many instances. Urbex leads most with-GT metrics across both test splits, while CityAnchor only retains the highest Hit@5 on one split. Prompted closed-source VLMs with tool access improve over naive use but still fall far short of the trained Urbex policy. Indoor 3D grounding methods perform poorly on city-scale inputs, with one baseline almost failing on both splits after point cloud downsampling. Urbex is an order of magnitude faster than the strongest ranking baseline under online timing, since it scores a small number of regions rather than all candidate instances. In the without-GT setting, candidate-ranking methods are further disadvantaged because they require a separate proposal-generation stage.

Zero-shot transfer to STPLS3D-Refer is difficult for indoor 3D grounding methods, which degrade sharply under domain shift. Closed-source VLMs with access to the interactive tool environment perform better as baselines, but the proposed Urbex agent outperforms all reported methods on every accuracy metric. The re-annotation results provide complementary evidence of Urbex's transfer, with stronger target identification than the compared baselines. Indoor 3D grounding baselines struggle on STPLS3D-Refer zero-shot transfer, with EDA showing near-zero top-1 accuracy and very low spatial overlap. Closed-source VLMs using the interactive tool environment are the strongest reported baselines, with GPT-5.4 higher than Qwen3.6-plus on Hit@1, Hit@3, and Hit@5. Urbex leads across all reported accuracy metrics and also shows higher Hit@1 on re-annotated expressions without further training.

Ablating Urbex's tool interface and interaction budget shows that each component contributes to active search. Without tools, the agent performs substantially worse on target identification and localization, while adding zoom, landmark query, and oblique rendering progressively improves both Hit@3 and 2D IoU. Increasing the turn budget from seven to eight improves results, but nine turns yields no further identification gain and slightly lower localization. The largest individual tool gain comes from adding landmark query, where Hit@3 rises substantially and 2D IoU more than doubles over zoom-in alone. Oblique rendering further improves results by adding 3D-aware visual verification after landmark query. An eight-turn budget gives the best accuracy-efficiency trade-off; seven turns underperform and nine turns slightly reduce localization without improving identification.

The reward ablation shows that an IoU-only reward produces the weakest target identification and localization. Adding distance shaping or tool-level shaping each improves both metrics, with distance shaping providing the larger individual gain. Combining all reward terms yields the best results, indicating that localization quality, distance progress, and stable tool use are complementary. Full reward achieves the highest Hit@3 and 2D IoU among the reward settings. Distance shaping improves both metrics more than tool-level shaping alone. Tool-level shaping still helps over IoU-only but is less impactful than distance shaping.

The experiments evaluate Urbex on city-scale 3D language grounding using CityRefer and zero-shot transfer to STPLS3D-Refer, comparing it with indoor grounding methods, candidate-ranking baselines, and closed-source VLMs that have access to the same interactive tools. Urbex consistently achieves the strongest target identification and localization while remaining more efficient by inspecting a small set of informative regions rather than exhaustively scoring candidates. Ablations further show that interactive tools, especially landmark query and oblique rendering, an eight-turn interaction budget, and combined reward shaping with distance and tool-level terms all contribute to the agent’s active search performance.