One-sentence Summary

Ropedia, Nanyang Technological University, Tsinghua University, and Peking University present EGOXPERIENCE, a full-stack approach for learning world-grounded egocentric representations from human experience that jointly reconstructs observer motion, surrounding geometry, and hand-object interactions with aligned supervision, addressing the lack of coherent joint reconstruction and aligned supervision in existing egocentric video methods and datasets.

Key Contributions

• Introduces EgoXperience, a unified approach to learning world-grounded egocentric representations through task-driven capture, geometrically aligned annotation, and joint reconstruction.
• Provides EgoX-Engine, which delivers aligned multimodal annotations for the 50-hour EgoX-Data-6M and curates the held-out one-hour EgoX-Bench, covering cameras, depth, hands, and rigid-object annotations.
• Presents EgoX, which jointly reconstructs camera poses, dense depth, hand geometry, and 6-DoF rigid-object poses in a common metric world frame from RGB observations and optional first-frame object masks. EgoX achieves state-of-the-art results on four benchmarks, ranking first in depth, hand, and object metrics and best camera accuracy on HOI4D.

Introduction

Egocentric video is a practical capture modality for human activity and embodied AI because it records camera motion, scene context, hands, and object interactions from a wearable sensor. However, prior reconstruction methods typically recover only a subset of these components, and stitching separate outputs creates scale and coordinate-frame inconsistencies. Existing datasets also struggle to provide joint annotations at scale because calibrated multi-camera or motion-capture setups are expensive and hard to deploy across diverse environments. The authors introduce EgoXperience, a full-stack paradigm that combines an automatic annotation engine, EgoX-Engine, a 50-hour aligned dataset, a benchmark, and EgoX, a feed-forward model that jointly reconstructs camera pose, scene depth, hands, and rigid-object pose in a shared metric world frame from monocular egocentric video.

Dataset

The authors construct EGOX-Data-6M from task-driven egocentric recordings and use it alongside public datasets for joint reconstruction training.

• Composition and sources:

  • EGOX-Data-6M contains 50 hours of video, 101 rigid objects, and 270 task scripts.
  • Captures come from the HOMIE Gen2 head-mounted rig with synchronized calibrated stereo and four fisheye cameras.
  • Activities cover workspace manipulation, cross-room interactions, and outdoor loco-manipulation.

• Processing and annotation:

  • EGOX-Engine provides a session-fixed metric frame using SLAM camera trajectories.
  • Stereo depth supplies metric scene geometry; hand keypoints are triangulated and used to constrain MANO fitting.
  • Object meshes are built from 31 synchronized dome cameras using SAM 3 masks and MV-SAM3D, with an AprilTag board for scale.
  • Object motion is initialized from a first-frame mask, tracked with FoundationPose across the interaction interval, and verified against observed depth.
  • All camera, scene, hand, and object annotations are aligned in the same metric frame. Second-level task captions are included.

• Subsets and filtering:

  • EGOX-Bench is a quality-controlled one-hour subset spanning all three activity settings, reserved for evaluation.
  • Additional public datasets are converted to a common coordinate and scale convention, cleaned of erroneous annotations, and depth-aligned with hand and object geometry. They can contribute partial labels.

• Use in the model:

  • EGOX-Data-6M and the public datasets are used as complementary training data.
  • EGOX-Bench is used exclusively for joint evaluation of camera, scene, hand, and object reconstruction.
  • The final training mixture ratios are detailed in Appendix F.
  • The provided text does not specify a cropping strategy.

Method

The authors introduce the EGOXPERIENCE paradigm, which unifies data capture, annotation, and model learning for the joint metric reconstruction of the camera, scene depth, hands, and objects from egocentric videos. This comprehensive framework integrates the EGOX-Engine for data processing and annotation, the large-scale EGOX-Data-6M dataset, and the EGOX model for joint reconstruction from monocular video.

To generate the necessary aligned annotations, the authors develop the EGOX-Engine. Learning to jointly reconstruct egocentric activity is challenging because the observer and interacting entities move simultaneously, making it difficult to distinguish between camera motion and entity motion. The EGOX-Engine resolves this ambiguity by leveraging stereo depth to recover geometry at a metric scale and utilizing SLAM-estimated camera poses to align different annotations throughout each recording. The pipeline begins with object model preparation via multi-view cage scans to obtain complete object meshes with frozen geometry and metric scale. During the ego capture phase, the system records RGB, depth, camera poses, and MANO hand annotations. For object tracking, the engine employs an identity-safe mask tracking mechanism that anchors the first mask, propagates it using SAM3, and recovers identity by rejecting competing objects. Finally, a verified 6DoF trajectory is generated using FoundationPose proposals and evidence verification based on depth, silhouette, coverage, and motion, resulting in structured outputs including object 6DoF trajectories, ego-view mask sequences, and captions.

To translate the aligned supervision into joint reconstruction, the authors propose the EGOX model. Given monocular observations and optional first-frame object masks, EGOX jointly predicts camera intrinsics and poses, dense depth, two-hand MANO states, object origins and orientations, and a positive metric scale in a single forward pass. All reconstructed outputs are expressed in the coordinate system of the first-frame camera.

The architecture relies on a joint scene and entity representation encoded by a shared Transformer. Starting from image patch features extracted by DINOv3, the model introduces camera, hand, and object tokens in each view. Distinct left and right embeddings identify the hands, while first-frame mask descriptors associate target objects across views. These tokens gather image evidence through within-view attention and exchange information across observations through cross-view attention, forming a joint representation for reconstruction.

From this joint representation, the model reconstructs scene geometry and hand-object motion in a common reference frame. To combine predictions across views, it estimates camera intrinsics and extrinsics. These parameters allow the recovery of scene points, wrist positions, and object origins from their image coordinates and predicted depths. For a homogeneous image position $\tilde{u}$u~ and normalized depth $z$z, its position in the world frame $\mathcal{W}$W is calculated as:

This shared mapping ties scene and entity positions to the same camera estimates. For hand and object orientations, the inverse camera rotation is used to transform camera-frame orientations into the world frame. To recover metric scale, a shared ScaleHead predicts a positive factor for the sequence, scaling scene points and camera, wrist, and object positions consistently.

The authors employ geometrically coupled supervision to train the model. They supervise not only individual predictions but also the resulting hand-object geometry in the world frame. By transforming hand and object predictions through the estimated cameras and comparing the reconstructed geometry with ground truth, the resulting loss backpropagates through the camera transformations, updating both entity and camera predictions. The training objective combines these supervision terms:

where $\mathcal{L}_{\text{local}}$Llocal​ supervises camera and view-local entity predictions, $\mathcal{L}_{\text{dense}}$Ldense​ supervises depth and scene points, $\mathcal{L}_{\text{coupled}}$Lcoupled​ supervises the composed hand-object geometry, and $\mathcal{L}_{\text{scale}}$Lscale​ supervises metric scale. To accommodate varying annotation coverage across datasets, each loss is applied only where its required labels are available.

Experiment

The experiments evaluate camera, depth, hand, and object reconstruction on HOT3D, EgoX-Bench, H2O, and HOI4D using identical 150-observation clips, excluding evaluation sequences from training and comparing each method only on the outputs it provides. EgoX, the only end-to-end approach that jointly reconstructs cameras, depth, hands, and object 6-DoF poses in a shared world frame from RGB and first-frame masks, achieves the best hand and object results on all benchmarks and the best depth results, with competitive camera motion relative to strong geometry baselines. Ablations show that removing metric scale prediction or dense depth prediction degrades joint hand and object reconstruction, supporting the value of these components beyond simple rescaling.

EGOX-Data-6M is a mid-scale egocentric dataset in recorded hours and modeled rigid object count, and it uniquely combines single-area, cross-area, and outdoor capture. Existing resources are mostly single-area and show uneven annotation coverage, with no compared dataset providing all annotation types together. Its aligned camera, scene, hand, and object annotations in a shared metric frame distinguish it for joint reconstruction across workspace, cross-room, and outdoor activities. Large-scale existing datasets report no modeled objects, while the object-rich dataset is single-area, indicating a gap that EGOX-Data-6M fills with moderate scale and 3D object coverage. Only one compared dataset includes outdoor capture, and none include cross-area capture, whereas EGOX-Data-6M spans all three activity settings.

The compared reconstruction methods support different subsets of outputs and use either end-to-end or multi-stage inference. Several end-to-end approaches recover camera and depth but omit hands and object pose, while some hand-focused methods add hands but still omit object 6-DoF. EgoX is the only end-to-end approach in this comparison that jointly reconstructs cameras, depth, hands, and object 6-DoF poses in a shared world frame. Camera and depth reconstruction is common among several end-to-end methods, but hands and object 6-DoF are often unsupported. Hand3R and MINT add hand reconstruction and world-space output within end-to-end models, yet they do not include object 6-DoF. HaWoR is the only listed hand-based method with multi-stage inference rather than end-to-end inference. EgoX uniquely supports cameras, depth, hands, and object 6-DoF together in an end-to-end framework.

EgoX produces the most accurate hand reconstructions across all four benchmarks, achieving the lowest PA-, WA-, and W-MPJPE among the compared methods. The advantage is especially clear in world-space alignment, where strong baselines still show substantial drift from ground-truth positions. EgoX achieves the best hand reconstruction errors on HOT3D, EgoX-Bench, HOI4D, and H2O across all three metrics. On HOT3D, EgoX lowers W-MPJPE substantially relative to HaWoR, the strongest baseline on that metric, despite HaWoR using adaptive SLAM and metric depth scale. On EgoX-Bench, EgoX reduces W-MPJPE from 328.81 mm for WiLoR + SLAM to 113.24 mm. The results indicate better world-space hand placement rather than only better local pose estimation.

EgoX achieves the best camera-frame object pose reconstruction results across all four benchmarks, with the lowest ADD and ADD-S errors and the highest AUC-S50. These gains hold even against baselines that use additional inputs such as estimated depth, intrinsics, metric CAD assets, or per-frame masks. The results indicate more accurate object poses can be obtained with fewer auxiliary inputs. EgoX reports the lowest ADD and ADD-S and the highest AUC-S50 on HOT3D, EgoX-Bench, HOI4D, and H2O. On EgoX-Bench, EgoX outperforms the strongest competing result from GigaPose + MegaPose despite that pipeline using metric CAD assets and per-frame masks. On HOI4D, EgoX surpasses Any6D, which receives estimated depth, intrinsics, and per-frame masks.

Across camera and depth reconstruction, EgoX achieves the lowest depth error and highest depth inlier ratio on all four benchmarks while also improving camera motion over ReViV. Camera metric results are more dataset dependent, with VGGT-Ω leading on HOT3D and EgoX-Bench, EgoX leading on HOI4D, and DA3 Nested leading on H2O. EgoX obtains its depth accuracy using its own predicted metric scale rather than scale fitted from DA3 Nested depth. EgoX reports the strongest depth reconstruction on every benchmark, with especially large gains over ReViV and VGGT-Ω on EgoX-Bench. EgoX achieves lower camera translation and rotation error than ReViV on all four benchmarks. In camera metrics, the best result varies by dataset: VGGT-Ω leads on HOT3D and EgoX-Bench, EgoX leads on HOI4D, and DA3 Nested leads on H2O.

The experiments introduce EGOX-Data-6M as a mid-scale egocentric dataset that uniquely spans single-area, cross-area, and outdoor capture with aligned camera, scene, hand, and object annotations in a shared metric frame, addressing gaps in existing benchmarks. They compare EgoX with end-to-end and multi-stage reconstruction methods on HOT3D, EgoX-Bench, HOI4D, and H2O, showing that EgoX is the only end-to-end approach that jointly reconstructs cameras, depth, hands, and object 6-DoF poses. Results validate that EgoX achieves the most accurate hand reconstructions and camera-frame object pose estimates across all benchmarks, with better world-space alignment and fewer auxiliary inputs than competing pipelines, while also providing the strongest depth reconstruction and improved camera motion over ReViV; the best camera metric remains dataset dependent.