One-sentence Summary

Ropedia et al. propose HOLI-WORLD, an integrated ecosystem spanning experience capture, world representation, automated data construction, model learning, and independent evaluation, in which HOLIWORLD-Foundry combines static and dynamic perception to automatically construct world records from roughly 100 self-collected scenes into HOLIWORLD-Data with about 80K QA pairs, and training on these resources improves HOLIWORLD-VL-4B's spatial and temporal reasoning on HOLIWORLD-Bench and EgoDynamic4D.

Key Contributions

• The paper introduces HOLIWORLD, an integrated ecosystem connecting experience capture, world representation, automated data construction, model learning, and independent evaluation.
• HOLIWORLD-Foundry combines complementary static and dynamic perception to automatically construct world records that link persistent scene structure, evolving object states, and interaction histories with supporting observations.
• HOLIWORLD-Data organizes records from approximately 100 self-collected scenes into about 80K QA pairs used to train HOLIWORLD-VL-4B, while HOLIWORLD-Bench provides independent human annotations for evaluation. Experiments on ScanNet, ScanNet++, Aria Digital Twin, HOLIWORLD-Bench, and EgoDynamic4D show improved static perception, dynamic tracking, and spatial and temporal reasoning.

Introduction

Building a coherent understanding of the physical world across viewpoints and time remains an open AI challenge, even with vision-language and predictive video modeling. Egocentric recordings offer a direct route to environments, objects, and interactions as people experience them, but turning these recordings into detailed supervision typically requires substantial manual annotation and curation. Static scene reconstruction and egocentric object tracking have advanced separately, yet linking scene geometry, object semantics, and interaction histories into traceable learning resources is still difficult. The authors introduce HOLIWORLD, an integrated ecosystem whose HOLIWORLD-Foundry pipeline automatically reconstructs static scenes and tracks dynamic object states from in-the-wild egocentric video, producing HOLIWORLD-Data and evidence-driven QA supervision for training HOLIWORLD-VL. They also establish HOLIWORLD-Bench through independent human annotation to evaluate both the recovered world records and downstream spatial reasoning.

Dataset

The authors construct HoliWorld-Data from Foundry’s world records. The dataset organizes supervision into grounding, captioning, and spatial, temporal, and interaction QA. Each example keeps links to its source observations and annotations, making outputs traceable to specific objects, times, and evidence.

• Sources and scope: Built from Foundry world records. Static objects and their geometry support spatial understanding. Dynamic tracks with consistent identities support motion and temporal understanding. Tracks paired with interaction semantics support descriptions of object manipulation and state changes.

• Annotation construction: QA pairs are generated from spatial, temporal, and interaction annotations, following the QA patterns of EgoDynamic4D. Grounding links referring expressions to 2D observations, static 3D boxes, or dynamic tracks. Spatial QA uses object geometry and camera poses to derive direction, distance, containment, support, and viewpoint-dependent relations. Temporal QA compares object states over time to describe motion, event ordering, and duration. Interaction QA uses interaction semantics to describe manipulated objects and state changes. Scene captions and inventory questions aggregate object records across frames. Dynamic supervision assumes at most one moving object at each time.

• Composition: HoliWorld-Data contains roughly 100 scenes, 444 egocentric videos, about 100 3D Gaussian splatting reconstructions, 1,322K RGB frames with depth maps and camera poses, 90K 2D boxes, 2,466 static 3D boxes, and 2,095 dynamic tracks. It supports about 4K 3D grounding annotations, 80K captions, 13,647 static QA pairs, and 64,409 dynamic QA pairs, totaling about 80K QA pairs.

• Usage: The authors use the dataset to provide grounding, captioning, spatial QA, temporal QA, and interaction QA supervision. Source observations and annotations remain attached so target outputs can be traced to specific objects and times.

Method

The authors present HoliWorld, a comprehensive ecosystem that converts egocentric recordings into spatially and temporally aligned supervision for training vision-language models. The framework integrates hardware capture, scene reconstruction, annotation, and dataset generation to produce high-quality spatial understanding data.

The pipeline begins with the Eagle Capture Device, which records synchronized stereo RGB and IMU measurements. HoliWorld-Foundry then reconstructs the scenes and annotates both static objects and dynamic tracks. HoliWorld-Data organizes these outputs into grounding, captioning, and spatial QA supervision for HoliWorld-VL. Independent human annotations of held-out recordings form HoliWorld-Bench to evaluate annotation quality and the model's spatial understanding.

Within HoliWorld-Foundry, the process starts with geometric preparation and shared semantic cues. The authors first recover camera poses and sparse points from stereo RGB and IMU using visual-inertial SLAM. They then estimate depth and dense points from RGB keyframes and these poses using Depth Anything 3 (DA3). Using RGB observations and this geometry, they optimize a 3D Gaussian scene representation $\mathcal{G}$G, render refined depth at the observed poses, and complete dynamic regions with per-frame DA3 depth estimation. The resulting observations are organized as:

where $I_t$It​ is an RGB frame, $D_t$Dt​ is a depth map, $\mathbf{K}_t$Kt​ is the camera intrinsic matrix, and $\mathbf{T}_t$Tt​ maps camera coordinates to world coordinates. To route static and dynamic objects to their respective perception pipelines, a vision-language model (VLM) extracts categories, attributes, motion cues, and interaction descriptions from RGB keyframes in $\mathcal{O}$O. Categories guide static image-based proposals, while motion and interaction cues identify dynamic tracking targets.

For static object perception, the authors employ a dual-path 3D detection strategy to capture both object details and scene-level geometry. In the image-based path, they run Boxer separately on the left and right RGB streams to capture complementary object details. Open-vocabulary 2D detections are lifted into metric 3D boxes using the corresponding depth and camera poses, then associated across frames. In parallel, TR3D detects objects in the reconstructed scene point cloud $\mathcal{P}$P, capturing large-object extents beyond individual views. To handle multiple proposals referring to the same object and discard those with little geometric support, low-cost geometric checks merge spatially compatible boxes and filter out those with insufficient point support in $\mathcal{P}$P before VLM validation. A VLM then checks each selected box against source observations and supplementary views rendered from $\mathcal{G}$G. Based on this evidence, it can keep the box, refine its geometry, relabel its category, or reject it. If rejected, a preselected fallback is evaluated. Accepted boxes form $\mathcal{B}^{\text{sta}}$Bsta with links to their supporting observations.

Dynamic object perception handles manipulated objects that may disappear and reappear. To recognize the same object across interruptions, the authors use hand regions and VLM interaction descriptions to identify the target in each segment, comparing descriptions and keyframe evidence to establish consistent object identities. Guided by these identities, the Segment Anything Model (SAM) selects and initializes the target mask, then propagates it within each interaction segment to obtain $\{m_t\}${mt​}, where $m_t$mt​ denotes the foreground pixels of the moving object at time $t$t. To recover object states in world coordinates, the pixels of each valid tracked mask are back-projected using the DA3-completed dynamic depths in $D_t$Dt​ and the camera pose:

where $\bar{\mathbf{u}}$uˉ is the homogeneous image coordinate and $\mathcal{X}_t$Xt​ contains the object-supported world points. After filtering unsupported depth samples, an oriented bounding box (OBB) is fitted to each point set, and boxes with the same identity are linked across time to form a dynamic 4D track.

The Foundry output is unified into a world representation $\mathcal{S}(t) = \big(\mathcal{O}_t, \mathcal{U}(t), \mathcal{G}, \mathcal{P}, \mathcal{B}^{\text{sta}}, \mathcal{B}^{\text{dyn}}(t), \mathcal{Z}(t)\big)$S(t)=(Ot​,U(t),G,P,Bsta,Bdyn(t),Z(t)), where $\mathcal{U}(t)$U(t) contains detected 2D boxes and tracked masks, and $\mathcal{Z}(t)$Z(t) records associated labels, identities, confidence scores, interaction semantics, and evidence links. This representation allows supervision to refer to specific objects, times, and supporting observations.

Finally, HoliWorld-Data organizes these world records into grounding, captioning, and spatial, temporal, and interaction QA supervision. Static objects and their geometry support spatial understanding, while dynamic tracks with consistent identities support motion and temporal understanding. QA pairs are constructed following established patterns, where grounding tasks associate referring expressions with 2D observations, static 3D boxes, or dynamic tracks. Spatial QA derives direction, distance, and containment relations using object geometry and camera poses. Temporal QA compares object states over time, and interaction QA describes manipulated objects and state changes using interaction semantics. The resulting dataset comprises approximately 100 scenes, 444 egocentric videos, 1,322K RGB frames with depth maps and camera poses, 2,466 static 3D boxes, and 2,095 dynamic tracks, supporting roughly 80K QA pairs and 80K captions.

Experiment

The experiments evaluate the pipeline for generating object-centric 3D annotations in static scenes and for tracking moving objects, with separate assessments of static perception, dynamic tracking, downstream understanding, and ablations. The approach consistently outperforms baselines in static object detection and dynamic tracking, showing more complete object coverage, better-aligned spatial extents, and more consistent trajectories. Downstream fine-tuning on the generated data substantially improves spatial and temporal reasoning, and ablations confirm that static components such as image lifting, point-cloud detection, stereo fusion, and caption guidance provide complementary benefits, while dynamic components improve identity continuity and trajectory stability.

HOLIWORLD-FOUNDRY attains the strongest static 3D object perception results across ScanNet, ScanNet++, and HoliWorld-Bench, leading in both AP25 and AP50 on all three datasets. Its AP50 margin over the best baselines is especially large on ScanNet and ScanNet++, pointing to more precise detections under stricter geometric matching. Among the compared model-based baselines, Map-Det3D generally produces the best detection AP, while several other approaches remain near zero. HOLIWORLD-FOUNDRY achieves the highest AP25 and AP50 on all three static perception benchmarks. The AP50 lead over the strongest baselines is substantially larger on ScanNet and ScanNet++, indicating better performance under stricter matching. Map-Det3D is the strongest model-based baseline for detection AP but still falls well behind HOLIWORLD-FOUNDRY. Several model-based baselines, including SceneScript and LLaVA-3D, have near-zero AP scores across the evaluated datasets.

HOLIWORLD-FOUNDRY leads dynamic 3D object tracking on both ADT20 and HoliWorld-Bench, with substantially higher HOTA than the strongest compared pipelines. The gains are accompanied by higher association and trajectory coverage scores, pointing to more consistent object identities and broader tracking coverage. Complementary box metrics also reflect improved spatial extent alignment on matched objects. HOLIWORLD-FOUNDRY achieves the best HOTA on both datasets, with a particularly large lead on HoliWorld-Bench. Higher AssA and PCL indicate more consistent identities and wider trajectory coverage than baselines. Box metrics provide complementary evidence of better spatial alignment on matched objects.

Fine-tuning Qwen3.5-4B on HoliWorld-Data leads to large spatial reasoning gains on both HoliWorld-Bench and EgoDynamic4D. The resulting HoliWorld-VL-4B surpasses zero-shot Qwen3.8-27B on all four reported metrics across both benchmarks. The improvements span yes/no, multiple-choice, and numerical accuracy, indicating broad benefits across question formats. Token F1 improves from 16.35% to 78.02% on HoliWorld-Bench and from 18.29% to 66.39% on EgoDynamic4D after fine-tuning. HoliWorld-VL-4B achieves the highest accuracy on every metric, including yes/no accuracy of 99.67% on EgoDynamic4D and multiple-choice accuracy above 90% on both benchmarks.

The experiments evaluate HOLIWORLD-FOUNDRY on static 3D object perception, dynamic 3D object tracking, and fine-tuned spatial reasoning. In static perception, it outperforms all baselines on ScanNet, ScanNet++, and HoliWorld-Bench, with especially clear gains under stricter geometric matching, while several model-based baselines remain near zero. For dynamic tracking, it achieves the strongest object identity consistency and trajectory coverage on ADT20 and HoliWorld-Bench. Fine-tuning Qwen3.5-4B on HoliWorld-Data also yields broad spatial reasoning improvements on HoliWorld-Bench and EgoDynamic4D, surpassing a much larger zero-shot model across question formats.