One-sentence Summary

Researchers from Nanyang Technological University, Ropedia, Duke University, and other institutions introduce Ropedia W0, the first egocentric loco-manipulation world foundation model spanning humans and robots, whose unified kinematic representation and curated Xp-Data achieve state-of-the-art performance across five cross-embodiment benchmarks and enable real-time VR interaction, planning, and decision-making applications.

Key Contributions

• The paper presents Ropedia W0, the first egocentric loco-manipulation world foundation model spanning human, gripper-robot, and dexterous-humanoid embodiments. It uses a unified kinematic representation that encodes locomotion through camera poses and manipulation through projected skeletons.
• The work introduces Xp-Data, a curated dataset of 10k hours of egocentric human data and 3k hours of robot data with unified multimodal annotations and systematic quality control, along with Xp-Teleop, a long-horizon dexterous humanoid loco-manipulation benchmark built from high-quality teleoperation data.
• Evaluations show state-of-the-art performance on five cross-embodiment benchmarks (DreamDojo, Ψ, ω-HOME, Xp-Teleop, and AgiBot) and held-out human egocentric video, with human pre-training scaling improving both human and robot results. Distillation enables real-time minute-long rollouts and supports policy evaluation, model-based planning, and world-action modeling.

Introduction

World models that predict how environments evolve under actions are valuable for embodied intelligence, but most existing models concentrate on fixed-base, gripper-based manipulation. Locomotion-induced viewpoint changes, dexterous hand-object interaction, and cross-embodiment generalization remain underexplored. Prior methods such as DreamDojo and Cosmos still require embodiment-specific control interfaces or additional adaptation. Large-scale egocentric human video could provide useful learning signal, yet public datasets often lack metric camera trajectories, reliable hand poses, and consistent annotation formats. The authors introduce Ropedia W0, a unified egocentric world model for cross-embodiment loco-manipulation. It uses a shared explicit kinematic representation combining camera poses and projected skeletons, and it is trained on Xp-Data, a curated collection of over 10k hours of human video and 3k hours of robot video.

Dataset

The authors use a human-in-the-loop data pipeline to create Xp-Data for training and Xp-Teleop for held-out evaluation.

• Dataset composition and sources

  • Xp-Data contains over 10k human hours and 3k robot hours of synchronized RGB, camera trajectories, manipulator skeletons, and captions; wrist poses and wide-FOV skeletons are included where available.
  • Human data combines 14 public egocentric datasets with over 6,000 hours of in-house recordings captured with a four-fisheye-camera rig, covering daily activities and object interactions.
  • Robot data draws from 13 public datasets spanning parallel grippers and dexterous hands.
  • AgiBot World Beta contributes two sources: a parallel-gripper stream and a dexterous-hand head-camera stream.
  • Xp-Teleop is a separate held-out benchmark of about one hour, recorded on a Unitree G1 humanoid with two six-DoF dexterous hands.

• Subset details

  • Xp-Data human subset: 14 public egocentric datasets plus over 6,000 in-house hours.
  • Xp-Data robot subset: 13 public robot datasets with parallel grippers and dexterous hands.
  • Xp-Teleop: 27 long-horizon takes at 30 fps, about one hour of egocentric video, divided into 543 human-annotated atomic segments spanning 30 skill classes.

• Filtering rules

  • A vision-language model screens raw segments for severe blur, poor or uneven illumination, over/under-exposure, and excessive camera jitter.
  • Segments with neither camera motion nor visible hands are discarded.

• Annotation and metadata construction

  • Camera intrinsics and metric trajectories are obtained from available calibration or estimated with an in-house SLAM pipeline or DA3.
  • Grounding DINO identifies dynamic regions such as hands and people for exclusion from camera-trajectory consistency checks.
  • Human kinematics use WiLoR-reconstructed MANO keypoints projected into images.
  • Captions describe hands, objects, and interactions at task, subtask, and atomic-action levels.
  • Robot keypoints are manually annotated and corrected because direct forward-kinematics projections often misalign with hands and grippers in public robot recordings.
  • Wrist poses are derived through forward kinematics where available.

• Quality checks

  • Camera trajectories are checked for depth-reprojection consistency and temporal smoothness.
  • Human hand annotations are cross-checked against independent 2D detections from MediaPipe and a fine-tuned MMPose model, with a vision-language model flagging false detections and hands belonging to other people.
  • Robot keypoints are cross-checked by two detectors fine-tuned on manually annotated robot-hand data.
  • Captions are verified with Qwen, with human review resolving discrepancies.

• Training clip construction and model use

  • A clip or episode is a contiguous source recording; a window is an 81-frame training sample; a source is one dataset and end-effector stream.
  • From quality-checked subtasks, the authors build 81-frame clips at 640 × 480 with adjusted intrinsics, uniformly resampled from a contiguous source interval.
  • A language model merges keyframe captions within a window into one window-level caption; a default caption is used for the small fraction of robot windows lacking a generated caption.
  • Each training clip provides synchronized RGB, projected skeletons, camera parameters, and a caption, plus wide-FOV skeletons and wrist poses where available.
  • The shared format across human and robot data supports training a cross-embodiment world model.
  • Xp-Teleop uses the same schema and is held out for evaluation; it includes failed grasps and recoveries to test off-nominal outcomes.

The provided section reports hour counts and source counts but does not specify explicit training mixture ratios beyond the separation of Xp-Data for training and Xp-Teleop for held-out evaluation.

Method

To train a general-purpose world model for cross-embodiment loco-manipulation, the authors develop a human-in-the-loop data pipeline that unifies collection, multimodal annotation, and quality checks.

The pipeline processes heterogeneous human and robot recordings through five stages. First, data sources include public human egocentric datasets, in-house multi-view recordings, and public robot datasets. Second, a vision-language model filters raw recordings for visual quality and task relevance, removing clips with severe blur, poor illumination, or no camera motion and visible hands. Third, multimodal annotation extracts geometry, kinematics, and semantics. Camera intrinsics and metric trajectories are obtained via SLAM or DA3, while human hand keypoints are projected from WiLoR-reconstructed MANO poses. For robots, forward kinematics provides initial estimates, which are refined through manual image-space keypoint annotation and correction. Hierarchical captions describe tasks, subtasks, and atomic actions. Fourth, quality checks verify annotation completeness, correctness, and cross-modal consistency using automated geometric checks, detector cross-checks, and a caption cascade involving Gemini annotation, Qwen verification, and human review. Finally, training-clip construction cuts quality-verified subtasks into standardized 81-frame multimodal clips, yielding a unified cross-embodiment corpus. The resulting dataset spans diverse environments and tasks while sharing a single annotation format across human hands, dexterous robot hands, and parallel grippers.

Egocentric loco-manipulation requires jointly modeling locomotion-induced viewpoint changes and manipulator motion. The authors encode these factors through camera poses and projected kinematic skeletons, providing a unified interface for humans, grippers, and dexterous humanoids. Given an initial image $I_0$I0​, a text condition $c_{\mathrm{txt}}$ctxt​, and a per-frame kinematic condition sequence $c_{1:T}$c1:T​, the model predicts future egocentric frames:

where $c_t^{\mathrm{cam}}$ctcam​ and $c_t^{\mathrm{skel}}$ctskel​ denote the camera-ray map and projected skeleton rendering, respectively.

Camera motion is expressed relative to the initial frame and encoded as dense Plücker-ray maps. Let $\mathbf{R}_t$Rt​ map directions from the current camera frame to the initial camera frame, and $\mathbf{o}_t$ot​ denote the camera center. Given camera intrinsics $\mathbf{K}_t$Kt​ and a homogeneous pixel coordinate $\tilde{\mathbf{u}}$u~, the direction is computed as:

This six-channel encoding associates camera geometry with each image location.

Kinematic skeletons for manipulation are rendered into $c_t^{\mathrm{skel}}$ctskel​. For robots, the initial forward kinematics projection of keypoint $j$j on embodiment $e$e is:

where $\mathcal{F}_j^{(e)}$Fj(e)​ maps joint states $\mathbf{q}_t$qt​ to a keypoint position in the robot base frame. Wrist-pose renderings $c_t^{\mathrm{wrist}}$ctwrist​ and wide-FOV skeletons $c_t^{\mathrm{wide}}$ctwide​ supplement the core representation when available.

The authors propose Ropedia W0, a latent video diffusion transformer conditioned on the initial observation, text, and kinematic controls.

Spatially aligned conditions are added to the video tokens. Camera-ray maps and skeleton renderings are encoded into condition latents $z^{\mathrm{cam}}$zcam and $z^{\mathrm{skel}}$zskel, aligned with the video latents in space and time. Separate projections map these latents to the transformer token dimension:

where $z_{\tau}$zτ​ denotes the video latents at noise level $\tau$τ, and $P_{\mathrm{vid}}$Pvid​ is the pretrained video projection. The camera and skeleton projections are initialized to zero. Text embeddings enter through cross-attention layers. Wrist-pose renderings follow the same aligned injection scheme, while wide-FOV skeletons enter through an additional cross-attention branch due to their lack of direct pixelwise correspondence.

Training proceeds in two stages. First, human pre-training on large-scale egocentric human videos conditioned on text, camera trajectories, and projected hand skeletons establishes a foundation for cross-embodiment transfer. Second, human-robot mid-training continues on a balanced mixture of human and robot data, introducing robot-specific appearance and manipulator geometry while maintaining exposure to diverse human experience. Availability gates accommodate samples with different annotation coverage for wrist-pose and wide-FOV conditions.

To enable real-time interactive rollouts, the authors first obtain a block-causal, 4-step autoregressive generator using Causal Forcing. They then extend it to long-horizon streaming generation using LongLive. Asymmetric distribution-matching distillation contrasts the clean-latent predictions of a frozen bidirectional real denoiser $\mu_{\psi}$μψ​ and an online bidirectional fake denoiser $\mu_{\phi}$μϕ​:

where $z_0$z0​ is the student output and $z_{\tau}$zτ​ its noised version. LongLive subsequently adapts the few-step generator for long-video streaming with a bounded KV cache, allowing the student to stream fixed-size latent blocks at 21.3 fps for minute-long interactive rollouts.

Experiment

The evaluation spans five cross-embodiment benchmarks and zero-shot human egocentric video, testing direct transfer, few-shot post-training, and full-data post-training against DreamDojo, Cosmos3, and other recent world models. Ropedia W0 consistently leads across benchmarks, with especially strong data-efficient and zero-shot transfer, and a single mid-training checkpoint generalizes across multiple embodiments. Ablations show that skeleton and camera conditioning, human-video pretraining, and model scaling all contribute to prediction quality, while distillation enables real-time generation. The model also proves useful for policy evaluation, model-based planning, world-action learning, and live teleoperation.

Among the compared humanoid teleoperation datasets, tabletop collections emphasize dexterity but offer little to no locomotion and partial recording modalities. Xp-Teleop differs by capturing locomotion and dexterous manipulation together in long synchronized takes. It is the only dataset in the comparison that combines long-horizon loco-manipulation with the full set of RGB-D, IMU, whole-body joint, tactile, calibration, and keypoint signals. Tabletop teleoperation datasets generally have minimal or zero locomotion range and omit tactile, calibration, and keypoint recordings. Xp-Teleop uniquely provides synchronized long-horizon locomotion plus dexterity with full multimodal coverage.

On a 200-clip egocentric human-video evaluation set combining EgoSuite and OpenAoE, Ropedia W0 achieves the lowest camera-trajectory errors, the highest hand-following accuracy, and the best video prediction quality among compared methods. The evaluation uses datasets unseen during training by Ropedia W0 and all compared methods, indicating strong generalization in both motion following and visual quality. Ropedia W0 records the lowest camera-trajectory errors and the highest hand-following accuracy on the unseen egocentric evaluation set. Ropedia W0 also leads in visual quality, with the best PSNR, SSIM, and LPIPS among compared methods. Among the listed baseline rows, Cosmos3-Nano has the lowest trajectory errors and highest detection rate, while DreamDojo variants lag in hand keypoint accuracy and detection.

On the DreamDojo benchmark, Ropedia W0 variants consistently outperform the DreamDojo baselines across all reported metrics and test splits. The 5B model already surpasses the larger 14B baseline, and the A14B variant extends that lead. Gains are strongest in the low-data setting, supporting data-efficient transfer. Ropedia W0 (5B) outperforms the larger DreamDojo (14B) baseline on every reported PSNR, SSIM, and LPIPS value in the 1% post-training setting. Ropedia W0 (A14B) achieves the highest visual fidelity across In-lab, EgoDex, and DreamDojo-HV in the reported results. The largest gains are reported in the low-data regime, highlighting data-efficient adaptation to the target embodiment.

Ropedia W0 leads across zero-shot, few-shot, and full-data settings on the evaluated dexterous-humanoid and loco-manipulation benchmarks. The A14B midtrain variant matches or exceeds baseline full-data results with little or no post-training, and its zero-shot performance on long-horizon loco-manipulation surpasses every baseline full-data result. Post-training improves unseen scenes in step with seen scenes, indicating adaptation rather than memorization. On Ψ0, the largest Ropedia W0 variant reports the highest PSNR and SSIM and the lowest LPIPS in all three post-training settings among the compared models. The A14B midtrain variant matches or exceeds every baseline's full-data result with little or no post-training and shows the smallest seen-unseen gap on Ψ0 and ω-HOME without post-training. On Xp-Teleop, the A14B midtrain zero-shot result exceeds every baseline's full-data result, and post-training lifts unseen scenes similarly to seen scenes.

On held-out AgiBot gripper windows, Ropedia W0 5B midtrain achieves the highest PSNR and the best skeleton and background consistency among compared models, while its SSIM and LPIPS are close to the leading DreamDojo 14B. Scaling the model to A14B yields an additional 2.3 dB PSNR gain on this in-distribution set. Other models such as Cosmos3 lead in dynamic degree but show lower fidelity and consistency. Ropedia W0 5B midtrain records the highest PSNR and best consistency scores on the AgiBot set. A14B scaling further improves AgiBot PSNR by 2.3 dB over the 5B midtrain result, while Cosmos3 models show more dynamic motion but weaker fidelity.

The paper evaluates Xp-Teleop as a humanoid teleoperation dataset that uniquely captures synchronized long-horizon locomotion and dexterous manipulation with full multimodal coverage, unlike tabletop datasets. Ropedia W0 is tested on unseen egocentric video, DreamDojo, loco-manipulation, and held-out AgiBot benchmarks, where it consistently leads in motion following and visual fidelity over baselines, with stronger gains in low-data settings. The A14B variant improves further, and its midtrain checkpoint matches or exceeds baseline full-data results with little or no post-training, while post-training improves unseen and seen scenes similarly, indicating generalization and data-efficient adaptation rather than memorization.