One-sentence Summary

Researchers from MirroS, Tsinghua University, Peking University, and Nanyang Technological University introduce Code-as-World, an executable world representation that encodes physical composition, dynamic evolution, and visual appearance as code and uses an abductive agentic discovery loop to propose, execute, render, verify, and refine world hypotheses from multimodal observations; using verified worlds as scalable physical supervision, Code-as-World-VL achieves state-of-the-art performance on QuantiPhy and surpasses leading proprietary models.

Key Contributions

• The paper introduces Code-as-World, a paradigm that represents physical worlds as executable code encoding physical composition, dynamic evolution, and visual appearance to provide a compact, quantitatively grounded, and controllable abstraction.
• It develops an agentic discovery loop inspired by abductive reasoning that constructs executable world hypotheses from multimodal observations such as natural-language descriptions or real-world videos by proposing, executing, rendering, verifying, and refining them.
• It demonstrates that verified executable worlds can generate scalable physical supervision for training vision-language models, with Code-as-World-VL achieving state-of-the-art performance on QuantiPhy and surpassing leading proprietary models.

Introduction

Understanding physical scenes requires more than recognizing or describing events; it requires recovering the object states, physical parameters, and mechanisms that explain and predict behavior. Prior representations have complementary limits: pixel-level video models predict visually plausible futures without disentangling underlying causes, 3D reconstruction preserves geometry but not causal physical behavior, and natural language abstracts semantics but poorly expresses continuous quantitative states. The authors introduce Code-as-World, which represents task-relevant physical worlds as executable code covering physical composition, dynamic evolution, and visual appearance. They also formulate world acquisition as an agentic discovery loop that proposes, instantiates, executes, renders, and verifies executable hypotheses against multimodal evidence. These verified executable worlds then provide scalable physical supervision for vision-language models, improving quantitative physical reasoning from monocular videos.

Dataset

The authors construct supervision from text and video inputs, and they build a pixel-level measurement dataset from existing visual annotations.

Sources and composition

• Text inputs are converted into structured semantic evidence: explicit entities, spatial relations, physical events, and intended outcomes.
• Video inputs are converted into visual evidence: depth maps, instance masks, and object tracks. Segmented objects are also lifted into meshes using a 3D object generation model.
• The pixel-level dataset $D_{\mathrm{pix}}$Dpix​ is derived from existing visual datasets by converting bounding boxes, masks, and object tracks into quantitative question-answer pairs.
• The QA pairs cover object extent, position, displacement, velocity, and acceleration. They require no world-space calibration.
• The excerpt does not report the exact dataset scale or the names of the underlying visual datasets.

Processing and evidence construction

• For text-driven world construction, because textual descriptions rarely specify geometry, physical parameters, or camera configurations completely, the authors combine physical priors with default conditions to initialize an executable world hypothesis. For downstream sim-to-real transfer, a video generation model enriches objects, materials, backgrounds, and lighting while preserving physical trajectories.
• For video-driven abstraction, depth maps constrain spatial structure and relative distances, masks identify object boundaries and geometric extent, and tracks capture temporal correspondences and image-plane motion. Meshes are combined with depth and tracking evidence to recover object positions, scales, and dynamic states.
• For image-space measurements, given a trajectory $\{c_t\}_{t=1}^{T}${ct​}t=1T​ sampled at interval $\Delta t$Δt, displacement, velocity, and acceleration are computed as:
  • $\mathbf{d} = \mathbf{c}_{t_2} - \mathbf{c}_{t_1}$d=ct2​​−ct1​​
  • $\mathbf{v}_t = \frac{\mathbf{c}_{t+1} - \mathbf{c}_{t-1}}{2\Delta t}$vt​=2Δtct+1​−ct−1​​
  • $\mathbf{a}_t = \frac{\mathbf{c}_{t+1} - 2\mathbf{c}_t + \mathbf{c}_{t-1}}{\Delta t^2}$at​=Δt2ct+1​−2ct​+ct−1​​
• Object extent and position are read directly from bounding boxes or masks, while motion quantities are derived from object tracks.

How the data is used

• The pixel-level dataset $D_{\mathrm{pix}}$Dpix​ is used for supervised fine-tuning with the loss:

• This stage teaches the model to localize, measure, and track objects, providing the visual foundation for later world-space physical reasoning.
• Text and video evidence constrain the executable world representation space. Candidate world rollouts are projected back into the input view and refined by checking consistency with observed geometry, depth, masks, and trajectories.

Method

Method

The authors introduce Code-as-World, a paradigm that represents the physical world through executable code rather than through static pixel-level descriptions. In this formulation, code acts as a structured and compositional abstraction: entities, relations, physical parameters, and events are represented as operations over states, while incidental visual details are abstracted away. The representation separates the underlying structured state of a physical process from the continuous appearance through which it is observed. The structured state supports composition, editing, and explicit constraints, while the appearance component retains visual detail and uncertainty. Executing and rendering the code can produce observations that are consistent with the represented world, where physical equivalence is prioritized over pixel-level duplication.

A practical instantiation of this idea is an executable world representation (EWR), coupled with a programmatic interface to a physical simulation engine. An EWR is defined as

where the three components jointly specify what a world contains, how it evolves, and how it appears. The physical composition $\mathcal{C}$C describes the objects, their geometry, metric dimensions, and physical properties such as mass, friction, and gravity. Static environmental structures such as floors, tables, and walls are also treated as physical entities that can participate in support, contact, and collision. The dynamic evolution $\mathcal{E}$E encodes initial states, temporal changes, key events, and simulation duration, enabling the composition to be expanded into a complete state trajectory. The visual appearance $\mathcal{A}$A captures camera parameters, backgrounds, materials, lighting, frame rates, resolution, and rendering or video-generation settings. This component controls presentation without altering the physical process. Because the EWR is executable, individual components can be inspected, modified, and rerun independently, allowing changes to an object, a physical parameter, a dynamic condition, or a camera setting while preserving the rest of the world.

The problem of recovering such an executable world from partial and heterogeneous observations is formulated as an agentic discovery process rather than a direct prediction task. Given multimodal input evidence such as text descriptions or real videos, a discovery agent constructs modality-specific constraints and iteratively searches for an EWR that is consistent with the evidence. The shared procedure is a propose-instantiate-execute-render-verify loop. The resulting EWR externalizes the underlying physical mechanism as a queryable and modifiable interface for downstream simulation, verification, and data generation.

Evidence construction differs by modality. For text inputs, an evidence adapter extracts explicit entities, spatial relations, physical events, and intended outcomes, and organizes them as structured semantic evidence. Because text rarely fully determines geometry, physics, or camera settings, the agent combines physical priors with reasonable defaults to initialize an executable hypothesis, then refines it through simulation and semantic verification. For downstream applications, the resulting executable world can be passed to a video generation model for sim-to-real transfer, which enriches objects, materials, backgrounds, and lighting while retaining the underlying physical trajectories. For video inputs, the adapter extracts depth maps, instance masks, and object tracks. Depth constrains spatial structure and relative distances, masks identify object boundaries, and tracks capture temporal correspondence and image-plane motion. Segmented objects are converted into meshes with a 3D object generation model, and these meshes are combined with depth and tracking evidence to recover spatial positions, scales, and dynamic states. Candidate rollouts are projected back into the input view and refined by comparing their geometry, depth, masks, and trajectories with the observations.

In the discovery loop, the agent proposes or updates an EWR from the current evidence, the existing hypothesis, and structured feedback from the previous iteration. The EWR is instantiated as simulator-ready parameters $\theta$θ, and the simulator executes them to produce a state trajectory $\tau$τ that records object states, contacts, collisions, and event outcomes. The trajectory is then rendered into predicted observations; for video inputs, the simulated states are additionally projected into depth maps, instance masks, and image-plane trajectories. A verifier compares predictions with evidence at selected key frames. For text input, verification mainly evaluates semantic and physical constraints; for video input, it jointly compares RGB appearance, depth, masks, and trajectories. Discrepancies are aggregated into structured feedback $\Delta$Δ, which guides local revision of the relevant EWR component. The loop terminates when the current EWR explains the input sufficiently and parsimoniously, or when the iteration budget is exhausted.

For quantitative physical reasoning, the method uses verified executable worlds to supply physical supervision that real videos rarely provide. The task is defined as predicting a numerical answer $\hat{y}=f_{\theta}(V,q)\in\mathbb{R}$y^​=fθ​(V,q)∈R from a monocular video $V$V and a quantitative physical question $q$q. World-space queries provide a reference quantity with a known world-space value $\rho$ρ. Given pixel-space target measurement $y^{\mathrm{pix}}$ypix and reference measurement $\rho^{\mathrm{pix}}$ρpix, the relative scale is

The pipeline uses a two-stage training procedure. First, in image-space measurement grounding, existing bounding boxes, masks, and object tracks are converted into pixel-level quantitative question-answer pairs involving object extent, position, displacement, velocity, and acceleration. For a trajectory $\{\mathbf{c}_t\}_{t=1}^{T}${ct​}t=1T​ sampled at interval $\Delta t$Δt, displacement, velocity, and acceleration are computed as

The model is supervised fine-tuned on the resulting pixel-level dataset $\mathcal{D}_{\mathrm{pix}}$Dpix​ with

This stage trains the model to localize, measure, and track objects.

Second, in world-space physical calibration, the method generates exact world-space labels from verified executable worlds. Each EWR provides a synchronized video and simulated state trajectory containing object geometry, camera parameters, timestamps, and time-varying states. The agent samples a target object, timestamps, a physical quantity, and an output unit, optionally providing a reference quantity as a scale prior. Object size is read from scene geometry, and displacement, velocity, and acceleration are computed from the state trajectory. Retained instances form world-level training data. Text-driven and video-driven executable worlds use the same format, yielding $\mathcal{D}_{\mathrm{text}}$Dtext​ and $\mathcal{D}_{\mathrm{video}}$Dvideo​, which are combined into unified world-level training data. The model is optimized on these examples with Group Relative Policy Optimization (GRPO). The reward combines scale-normalized numerical accuracy with auxiliary rewards for unit correctness and response format:

Text-driven worlds provide fully observable simulator states and numerically exact supervision, while video-driven worlds better match real appearance and motion distributions. Joint training therefore combines accurate physical supervision with visual generalization to real videos.

Experiment

The experiments evaluate Code-as-World for constructing executable world representations from text and video inputs, using filtered video observations and an agentic discovery loop that iteratively refines worlds against independent visual and motion metrics. Qualitative results show that recovered worlds act as editable programs supporting counterfactual physical edits and camera changes, and the agentic loop improves reconstruction over one-shot generation and Best-of-5 sampling. The study further trains Code-as-World-VL models for quantitative physical reasoning, finding that world-space supervision preserves image-space measurement and yields strong QuantiPhy performance that scales from 4B to 9B and further improves with a 27B reasoning model, though the evaluation remains limited to monocular scale calibration and the discovery process itself is not internalized.

On QuantiPhy-validation, Code-as-World-VL models improve quantitative physical reasoning for estimating object size, displacement, velocity, and acceleration from monocular video. The 4B direct-answer variant outperforms larger open-weight baselines and is competitive with leading proprietary systems, while the 9B direct-answer variant achieves the best average among direct-answer models. A reasoning-enabled 27B variant further exceeds the direct-answer 9B and the strongest proprietary baseline, with average MRA scaling upward across the three Code-as-World-VL sizes. Code-as-World-VL-9B achieves the best average performance among direct-answer variants across the 2S, 2D, 3S, and 3D subsets. The reasoning-enabled Code-as-World-VL-27B exceeds the direct-answer 9B variant and the strongest proprietary baseline.

Experiments on QuantiPhy-validation evaluate quantitative physical reasoning from monocular video for estimating object size, displacement, velocity, and acceleration. Direct-answer Code-as-World-VL models improve over larger open-weight baselines and are competitive with leading proprietary systems, with the 9B variant achieving the best average direct-answer performance across the 2S, 2D, 3S, and 3D subsets. The reasoning-enabled 27B variant further surpasses both the direct-answer 9B model and the strongest proprietary baseline, indicating that scaling model size and adding reasoning both strengthen quantitative physical understanding.