One-sentence Summary

Researchers from Shanghai Jiao Tong University, National University of Singapore, Meituan, and other institutions propose URBANGROUND, the first sandbox for testing multimodal large language model agents in a physically constrained real-scale Hong Kong replica built from territory-wide 3D geospatial data, with closed-loop first-person exploration and interactive mapping, and show that local perception supports short-range spatial reasoning while orientation and extended navigation remain unreliable.

Key Contributions

• The paper introduces URBANGROUND, the first sandbox for evaluating urban spatial agency in a physically constrained, georegistered replica of Hong Kong built from territory-wide 3D geospatial data, supporting closed-loop first-person interaction and an interactive navigation map.
• The work contributes a staged evaluation protocol covering active scene-grounded spatial question answering, navigation to increasingly distant or less explicit destinations, and adaptation under changed route availability and pedestrian motion.
• Evaluation of contemporary MLLM agents in this environment shows useful atomic abilities in visual recognition and short-range spatial reasoning, while orientation and pedestrian-aware movement remain unreliable and errors accumulate over extended exploration.

Introduction

Recent multimodal large language model (MLLM) agents can recognize urban objects from individual observations, but real city-scale agency requires keeping a usable spatial estimate as each movement changes the first-person view. Prior evaluations do not fully capture this because game sandboxes depend on game-specific mechanics, indoor physical environments are bounded, and urban imagery or simulators either lack continuous physical movement or do not preserve a real metropolis’s full georegistered structure. The authors introduce URBANGROUND, a real-scale interactive replica of Hong Kong built from official 3D visualization and pedestrian network data, streamed into Unity with physical collision and geographic coordinates. It supports first-person perception, physical control, and an interactive map under controlled illumination, weather, road closures, and moving pedestrians, enabling evaluation of how local perception scales into sustained navigation and adaptation in a real city.

Dataset

The authors build the simulation environment on two geospatial datasets provided by the Hong Kong Lands Department as part of its territory‑wide 3D Digital Map release.

• 3D Visualisation Map A tile‑based textured mesh reconstructed from oblique aerial imagery. It covers the entire Hong Kong territory, is distributed as Cesium 3D Tiles in the WGS84 reference system, and serves as the visible urban surface.

• 3D Pedestrian Network Georeferenced 3D line features derived from pedestrian‑related road records. Each vertex keeps its geographic position and attributes, and each edge represents a pedestrian connection between two locations. The network does not restrict movement but provides a registered path graph for later comparison.

The framework loads the 3D Tiles hierarchy into Unity at run‑time, transforms tile coordinates into a shared simulation frame, and streams tiles according to the current viewing scale. Active tile geometry supplies both the rendered city and the collision geometry for the character controller. The pedestrian network is imported into the same coordinate system as a connected graph. Agents move freely in continuous space; their trajectory can be evaluated against the network after execution. The authors use these two datasets together as the full‑scale, realistic geospatial layer to test spatial agency of embodied agents.

Method

URBANGROUND is a Unity-based framework designed to transform territory-scale geospatial data into a physically constrained city environment for evaluating Multimodal Large Language Model (MLLM) agents. The system is structured to assess whether evidence grounded at a specific viewpoint remains useful as the agent navigates through the environment. The framework operates through a closed-loop interaction defined by three cumulative capabilities: grounding, persistence, and adaptation.

Let $s_t$st​ denote the hidden interaction state at turn $t$t, which includes the georegistered agent pose, current interface state, task progress, and city conditions. The model-visible observation bundle is $o_t$ot​, comprising the current first-person or map view along with task-specific notices. The model-facing interface deliberately excludes privileged information such as remaining distance or shortest path APIs. Using $g$g for the task objective, $m_t$mt​ for the working spatial state, and $a_t$at​ for the structured action executed by the simulator, the closed-loop interaction is formulated as:

$$\begin{array}{c} o _ {t} = \mathcal {O} (s _ {t}), \\ m _ {0} = \mathcal {U} _ {0} (o _ {0}, g), \\ m _ {t} = \mathcal {U} (m _ {t - 1}, o _ {t}, a _ {t - 1}, g), \quad t \geq 1, \\ a _ {t} = \pi (m _ {t}, g), \\ s _ {t + 1} = \mathcal {T} (s _ {t}, a _ {t}, \xi_ {t}). \end{array}$$ot​=O(st​),m0​=U0​(o0​,g),mt​=U(mt−1​,ot​,at−1​,g),t≥1,at​=π(mt​,g),st+1​=T(st​,at​,ξt​).​

Here, $m_t$mt​ represents the task-relevant spatial information induced in the model's interaction context, while $\xi_t$ξt​ accounts for dynamic environmental changes such as road closures or pedestrian motion.

The framework realizes this loop through three distinct layers: the geospatial layer, the simulation layer, and the agent layer.

The geospatial layer anchors the visible city and pedestrian connectivity within a unified geographic frame. It utilizes the 3D Digital Map released by the Hong Kong Lands Department, specifically the 3D Visualisation Map and the 3D Pedestrian Network. The 3D Visualisation Map provides a tile-based textured mesh reconstructed from oblique aerial imagery, distributed as Cesium 3D Tiles under the WGS84 reference system. URBANGROUND loads this hierarchy into Unity at runtime, transforming geographic coordinates into a shared simulation frame and streaming tiles based on the current viewing scale. The 3D Pedestrian Network is imported as a connected graph where vertices retain geographic positions and edges represent pedestrian segments. This network does not restrict the agent to predefined edges, allowing continuous movement while enabling post-execution trajectory analysis.

The simulation layer transforms the registered city into a continuously evolving environment, controlling embodied movement, collision, time, weather, and pedestrian activity while preserving geographic coordinates, as shown in the figure below:

Embodiment and physical constraints are handled by instantiating the controlled agent as a first-person character in Unity. Its continuous motion is resolved against collision geometry derived from the city mesh, ensuring that buildings, walls, and elevation changes constrain the trajectory. The agent moves on the surface visible in its observation and cannot cross visible structures through coordinate updates alone. The simulation layer also incorporates a continuous time-of-day system that controls the sky, sun position, ambient illumination, and shadows, allowing the environment to transition from day to dusk and night. Additionally, a configurable weather system supports conditions such as rain and fog, which alter visibility and surface rendering. Rain produces particles that interact with the scene, generating an exposure signal that distinguishes covered paths from open streets. Finally, the pedestrian population is populated with animated avatars spawned on the registered network and assigned routes over its edges, with positions and collisions recorded at every step to support safety analyses alongside navigation success.

The agent layer connects an external MLLM to the running Unity environment via a client-server interface, allowing different models to control the same embodiment without modifying the Unity implementation. The primary observation is an RGB image rendered from the agent's first-person camera, presenting the real city from the pose produced by the previous physical action. For tasks requiring global reasoning, an interactive map tool provides a georeferenced overhead view of the city, marking the agent's current location and allowing panning and zooming without exposing computed routes. At each interaction turn, the agent selects exactly one action object from the action space $\mathcal{A} = \mathcal{A}_{\mathrm{fp}} \cup \mathcal{A}_{\mathrm{map}} \cup \{\text{terminate}\}$A=Afp​∪Amap​∪{terminate}. The first-person actions $\mathcal{A}_{\mathrm{fp}}$Afp​ include move, sprint, look, jump, and open_map, while the map actions $\mathcal{A}_{\text{map}}$Amap​ include map_select, map_pan, map_zoom, map_orbit, and close_map. The move and sprint actions specify movement directions and durations, potentially including yaw and pitch rates.

To systematically evaluate spatial agency, the authors organize the tasks as a five-level ladder that increases the spatial state required for success while keeping the interaction interface fixed. Refer to the framework diagram:

Level 1 focuses on local environment understanding through visual recognition, orientation, and active exploration, probing the grounding capability. Level 2 introduces navigation under explicit instructions, varying the range and constraints to stress persistence. Level 3 moves to exploration under implicit instructions, requiring the agent to infer destinations before navigating. Level 4 extends persistence to multi-task planning, where the agent maintains and executes a plan across several goals. Level 5 introduces dynamic environment interaction, such as road closures and moving pedestrians, examining adaptation. The ladder is instantiated across various urban regions of Hong Kong, with tasks distributed spatially to capture differences in street patterns, terrain, and vertical pedestrian connections. As shown in the figure below:

The spatial distribution of these tasks across the selected districts ensures a comprehensive evaluation of the agent's capabilities under diverse urban conditions.

Experiment

UrbanGround evaluates contemporary MLLMs on a five-level spatial agency ladder across Hong Kong, using first-person RGB observations plus map and physical actions within a fixed interaction budget. The first set of experiments validates local grounding: models recognize visible evidence reliably but struggle with directional orientation and pedestrian-network adherence. The second set validates how local grounding scales into navigation, showing strong short-range performance but systematic failure on longer routes and instruction-following because errors compound and route state is not maintained. Dynamic-city experiments validate adaptation, finding that weather and illumination changes affect question answering unevenly, while road closures and moving pedestrians expose persistent difficulties in revising routes and avoiding collisions even when local movement remains compliant.

Across evaluated MLLM agents, visual recognition is the most accurate task and orientation is the weakest, often falling toward random guessing. Active exploration occupies a middle ground, and pedestrian-network adherence is highest during active exploration while visual recognition adherence is comparatively lower. The results show that identifying visible landmarks does not guarantee reliable directional grounding. Visual recognition accuracy is generally strong across models, while orientation accuracy drops sharply and several systems approach random guessing for a four-option question. Active exploration answer accuracy is moderate and declines only moderately from recognition, even though agents must take short action sequences to gather missing evidence. Pedestrian-network adherence is highest in active exploration and lowest in visual recognition, while older and newer model generations differ more on orientation and active exploration than on recognition.

Agents show meaningful success only on short navigation when the destination is visible, but performance collapses on longer routes. Requiring the destination to be inferred causes a steep decline even at comparable distances, and tasks with multiple constraints remain almost completely unsolved across all models. The best short provided-destination navigation scores reach high values, yet long-range versions of the same task drop to near zero for every model. Inferred-destination tasks (place search and intent inference) cut success substantially relative to short provided-destination tasks, despite similar route lengths. Multi-stop and time-window navigation show negligible success, indicating that maintaining several active goals over a route is beyond current agents. Instructional and constrained navigation outcomes are low, with most models failing to reliably follow verbal directives or restricted paths.

Local question-answering accuracy falls under dusk and night relative to clear daytime, with dusk consistently worse than night across models. Short-navigation success is less systematically affected, with condition orderings varying by model and not closely matching QA robustness. Strong QA performance does not guarantee reliable navigation, as Gemini-3.6-Flash shows high QA but low navigation success. Across all models, local QA accuracy is lower at dusk and night than in clear daytime, and dusk is consistently the weakest condition. Short-navigation success varies by model and condition, with no consistent ordering across weather and time. Gemini-3.6-Flash has high and relatively stable QA accuracy but among the lowest short-navigation success rates. GPT-5.5 remains comparatively stable across conditions on both tasks, while GPT-5.4 and GPT-5.2 show sharper QA drops under low visibility.

Across dynamic navigation changes, agents maintain high pedestrian-network adherence but fail to convert that local compliance into successful route completion. Road-closure success is near zero for most models and safe progress remains mostly low, while pedestrian-collision rates stay high despite strong network adherence. The results reveal a consistent gap between locally compliant movement and adaptive, goal-directed action. In road-closure episodes, pedestrian-network adherence stays high while goal-reaching remains near zero for most models and safe progress stays mostly low. Under moving pedestrians, network adherence remains strong, yet collision rates are high for every model, showing that staying on the pedestrian network does not imply collision-aware control. Only modest safe progress and low success rates emerge under road closures and pedestrian obstacles, separating local motion competence from route revision and planning.

The experiments evaluate multimodal LLM agents on embodied urban tasks including visual recognition, orientation, active exploration, navigation with varying route demands, lighting conditions, and dynamic obstacles. Visual recognition is generally strongest and orientation is weakest, often near random guessing, while active exploration shows moderate accuracy and the highest pedestrian-network adherence. Navigation succeeds mainly on short routes with visible destinations, but long routes, inferred destinations, and multi-constraint tasks drop to near zero, and low-light conditions hurt local question answering most at dusk without consistently predicting navigation reliability. Under dynamic changes, agents maintain local pedestrian-network compliance but fail route completion, show high collision rates, and display weak adaptive planning.