One-sentence Summary

Researchers from Rice University and NVIDIA introduce RoboTok, an internet-scale data engine that retrieves manipulation-relevant human demonstrations from web videos by learning a latent motion space from 3D hand trajectories in actor-centered reference frames, enabling efficient cross-video comparison and improving dexterous robot policy learning over existing retrieval approaches.

Key Contributions

• RoboTok retrieves manipulation-relevant human demonstrations from web videos by representing behaviors as canonicalized 3D hand trajectories in an estimated actor-centered reference frame, focusing retrieval on underlying motion rather than scene appearance or labels.
• A learned motion embedding space, supervised by a spatiotemporal trajectory alignment metric, enables efficient comparison of manipulation behaviors across variations in viewpoint, scene, and occlusions while supporting continual indexing over internet-scale video collections.
• Evaluation on retrieval benchmarks and downstream robot policy learning shows that RoboTok retrieves more relevant demonstrations than existing approaches and improves task success. This establishes hand-pose trajectory-aware retrieval as a method for turning web video into a scalable, continuously growing source of supervision for dexterous manipulation.

Introduction

The authors tackle the challenge of scaling robot manipulation data by turning to internet video, an abundant source of human demonstrations that humanoid robots can physically reproduce. Prior retrieval methods are limited to fixed robot datasets or rely on visual appearance, semantics, or camera-relative motion, which fail to robustly compare manipulation behaviors across diverse viewpoints and scenes. The main contribution is RoboTok, a data engine that retrieves manipulation-relevant clips from web video by representing 3D hand trajectories in an estimated torso-centered reference frame and learning an embedding space supervised by a spatiotemporal alignment metric, enabling efficient, behavior-aware search without task labels or scene matching.

Dataset

The authors construct the RoboTok training dataset from the large-scale Action100M internet video corpus, which is pre-filtered for human actions. The final dataset consists of egocentric 3D hand-pose trajectories paired with motion-similarity targets derived via Dynamic Time Warping (DTW). Key details are:

• Source and filtering

  • Clips are drawn from Action100M and further filtered to retain segments that are 4–8 seconds long, have a near-static camera, and contain at most one left and one right hand.
  • Overlapping segments are removed greedily, favoring longer clips to maximize usable content.

• Hand pose extraction and metric grounding

  • 3D hand keypoints are estimated at 5 fps with WiLoR, and handedness is stabilized across frames.
  • Metric depth is recovered using MoGe‑2, converting weak‑perspective hand poses into metric camera‑frame coordinates.
  • Missing poses are infilled with HaWoR, yielding variable‑length sequences of metric 3D hand poses in the camera frame.

• Egocentric canonicalization

  • A lightweight torso‑frame estimator, trained on wrist trajectories alone (no body visibility required), transforms the camera‑frame poses into a static, actor‑centered egocentric coordinate frame.
  • This canonicalization makes trajectories comparable across different viewpoints and occlusions.

• Supervision signal

  • Pairwise DTW distances are computed on the egocentric trajectories to serve as pseudo‑ground‑truth motion similarity.
  • These DTW‑derived similarities are used as the offline supervision oracle for training the RoboTok retrieval encoder, avoiding manual action labels or semantic annotations.

The processed dataset is used directly to train the retrieval model; the paper does not report explicit train/val split sizes or mixture ratios, focusing instead on the pipeline that turns raw internet clips into motion‑comparable trajectories and similarity labels.

Method

The authors formulate the retrieval of human demonstration clips as finding a subset of clips that exhibit manipulation behavior most similar to a query clip. Rather than relying on visual appearance, they define similarity based on how hands move over time. To compare manipulation behavior, each clip is represented as a canonicalized 3D hand trajectory, and similarity is measured using Dynamic Time Warping (DTW). DTW aligns two hand-pose sequences while accommodating local differences in execution speed. Formally, for 21-joint hand-pose trajectories $x_i$xi​ and $x_j$xj​, DTW finds the valid alignment path $\pi$π that minimizes the cumulative pose distance:

$$\mathrm{DTW}(x_i, x_j) = \min_{\pi \in \Pi} \sum_{(t, u) \in \pi} d(x_i^t, x_j^u)$$DTW(xi​,xj​)=π∈Πmin​(t,u)∈π∑​d(xit​,xju​)

where $d(x_i^t, x_j^u) = \|x_i^t - x_j^u\|_2$d(xit​,xju​)=∥xit​−xju​∥2​ is the Euclidean distance between two hand poses. The trajectory similarity is defined as the length-normalized negative alignment cost:

$$s(i, j) = -\frac{\mathrm{DTW}(x_i, x_j)}{\frac{1}{2}(L_i + L_j)}$$s(i,j)=−21​(Li​+Lj​)DTW(xi​,xj​)​

Because computing DTW between a query and every trajectory in an internet-scale dataset is impractical, the authors use DTW as a supervision signal to learn an embedding $\Gamma: \mathcal{X} \to \mathbb{S}^{d-1}$Γ:X→Sd−1. The embedding is trained such that inner-product similarity in the embedding space preserves the ranking induced by DTW.

To prepare the training data, the authors process segments from a large-scale internet video corpus. As shown in the figure below:

The pipeline begins by filtering candidate clips to retain those with visible hands and near-static cameras. For each retained clip, 3D hand keypoints are estimated and stabilized across frames. Metric depth is used to transform the hand poses into metric camera coordinates. To make trajectories comparable across different camera configurations and arbitrary occlusions, the authors transform hand poses into an egocentric coordinate frame. They train a lightweight human torso-frame estimator that takes only hand trajectory wrist frames as input to predict the demonstrator's static torso frame. After this canonicalization, DTW is used to compute pseudo-ground-truth motion similarity between pairs of trajectories, serving as an offline supervision oracle.

The RoboTok retrieval model is designed to learn an embedding space that approximates this DTW-defined manipulation similarity while supporting efficient nearest-neighbor retrieval at scale. Refer to the framework diagram:

The encoder is a lightweight trajectory encoder that operates directly on the egocentric 3D hand representation. Each frame is positionally encoded and pooled by a lightweight cross-attention network into an $l_2$l2​-normalized $d$d-dimensional embedding for efficient cosine-similarity retrieval.

To train the model effectively at scale, the authors construct batches from anchor-centered groups rather than random sampling. For an anchor $a$a, its relevant set is defined as the top $K=20$K=20 neighbors under the DTW oracle. For each anchor, two trajectories are sampled from this relevant set as positives, and one trajectory immediately outside the relevant set is sampled as a boundary negative. Each batch contains 49 such groups. This sampling strategy provides informative supervision near the relevance threshold while avoiding trivially dissimilar random negatives.

The training objective combines a set loss and a rank loss: $\mathcal{L} = \mathcal{L}_{\text{set}} + \lambda \mathcal{L}_{\text{rank}}$L=Lset​+λLrank​. The set loss encourages oracle top-K neighbors to score above boundary and negative trajectories, determining which trajectories enter the retrieved neighborhood. The rank loss encourages the relative ordering of sampled positives to match the DTW-induced ranking, optimizing the portion of the ranking most relevant at retrieval time.

At inference time, retrieval operates in two stages. Every egocentric trajectory is encoded once and stored in an inner-product index offline. Given a query clip, nearest neighbors are retrieved directly from this index via online cosine similarity search without computing DTW against the full corpus. This shifts expensive trajectory comparison to training while allowing retrieval to scale efficiently with corpus size. Newly collected clips can be embedded and added directly to the index without retraining the model.

Experiment

The retrieval quality of RoboTok is evaluated against baseline methods on both in-domain and cross-dataset corpora using DTW-based pseudo-ground truth, where RoboTok consistently retrieves motion-aligned clips with near-oracle alignment cost and strong ranking while baselines perform near chance. Downstream policy experiments on a dexterous manipulation benchmark show that demonstrations retrieved by RoboTok substantially improve policy success rates over baselines, both under standard and more challenging task formulations with full 3D control and sparse rewards. These findings demonstrate that high-quality motion-based retrieval directly translates to better policy learning, even without action labels or robot demonstrations.

RoboTok retrieves demonstrations from internet human videos using canonicalized 3D hand trajectories in an actor-relative frame, while other methods rely on robot data or simpler representations and target parallel grippers. In dexterous manipulation tasks, policies guided by RoboTok retrievals consistently outperform those using other retrieval methods, with the advantage becoming much larger when hand motion is unrestricted and dense rewards are removed. The same ordering of methods holds for learning speed, showing that RoboTok's motion-centric retrieval provides more effective guidance for downstream policy learning. RoboTok is the only method that queries internet human videos, uses explicit 3D hand pose in an actor-relative reference frame, and targets dexterous hands. On harder dexterous manipulation tasks, RoboTok-guided policies improve success rates by 17.8 to 59.8 percentage points over the next best retrieval method. The performance ranking of retrieval methods remains consistent across tasks and also reflects faster policy learning with RoboTok demonstrations.

RoboTok dramatically outperforms all baseline retrieval methods on the RoboTok evaluation corpus, achieving near-perfect recall of true DTW neighbors and a mean DTW cost only 16% above the oracle. The strongest baseline, STRAP, barely exceeds random performance, while other methods remain indistinguishable from chance. The high ranking correlation confirms that RoboTok not only finds the correct neighbors but also orders them correctly. RoboTok achieves near-perfect recall, recovering almost every true neighbor within the top 20, while the best baseline STRAP recovers only a small fraction. The mean DTW cost of RoboTok retrievals is close to the oracle cost, whereas random clips incur a cost more than four times higher. Baseline methods Flow and HAND perform similarly to random retrieval across all metrics, with STRAP only marginally better. RoboTok's Kendall tau indicates strong neighbor ordering, in stark contrast to the near-zero ordering of baselines.

On the AssemblyHands corpus, RoboTok surpasses all baselines on every retrieval metric, achieving a mean average precision at five of 0.261 compared to 0.133 for the next best method, STRAP. Its retrieved clips incur a DTW cost only 13% above the ground-truth optimum, while random retrieval yields a cost nearly double the optimal. The relative method ordering matches that observed on the in-domain RoboTok corpus, confirming cross-dataset transfer. RoboTok nearly doubles the mAP@5 of the strongest baseline, STRAP (0.261 vs. 0.133). RoboTok retrieves clips with a DTW cost only 13% above the ground-truth optimum, whereas random retrieval is nearly twice as expensive.

On the original VTDexManip benchmark, policies guided by RoboTok retrievals consistently outperform all other retrieval-based methods and the best pretrained baseline across most tasks. RoboTok achieves the highest success rates on BottleCap Turning, Faucet Screwing, and Lever Sliding for both seen and unseen objects, with particularly dramatic improvements on the difficult Lever Sliding task where vanilla PPO nearly fails. Only on Table Reorientation does it slightly trail the pretrained baseline, while still surpassing all other retrieval-guided approaches. RoboTok-guided policies exceed the best pretrained baseline (VT-JointPretrain) by 7.45% on seen objects and 5.83% on unseen objects on average across all six tasks. On Lever Sliding, RoboTok boosts success from 5.8% (Base) to 95.5% on seen objects and from 2.2% to 92.0% on unseen objects, far outperforming every other method. Random retrieval guidance already improves over vanilla PPO, but RoboTok provides substantial additional gains, demonstrating the value of high-quality retrievals. The original benchmark formulation is saturated: the best pretrained baseline reaches above 80% on several tasks, and RoboTok pushes success rates near ceiling for most tasks.

On harder dexterous manipulation tasks with full 3D hand control and removed dense rewards, policies guided by RoboTok-retrieved demonstrations substantially outperform those using other retrieval methods. RoboTok achieves the highest success rates across all three tasks, with improvements of 17.8 to 59.8 percentage points over the next best method on seen objects. Base PPO and random retrieval fail to learn, while HAND is the strongest baseline but still far behind RoboTok. RoboTok reaches 77.3% success on BottleCap Turning (seen), 44.8% on Faucet Screwing, and 79.3% on Lever Sliding, outperforming all baselines. The improvement over the next best method (HAND) ranges from 17.8 to 59.8 percentage points on seen objects. Base PPO and random retrieval achieve near-zero success, showing that demonstration guidance is essential for these harder tasks. HAND is the most competitive baseline, but RoboTok more than doubles its success on Faucet Screwing and Lever Sliding. On unseen objects, RoboTok maintains a clear lead, though absolute success rates drop compared to seen objects.

RoboTok retrieves demonstrations from internet human videos by representing hand motion with canonicalized 3D trajectories in an actor-relative frame, while other methods rely on robot data or simpler representations and target parallel grippers. Across retrieval benchmarks and cross-dataset evaluations, RoboTok consistently recovers near-perfect true neighbors with low trajectory cost, whereas baselines perform near chance. When used to guide dexterous policy learning, RoboTok-retrieved demonstrations yield substantially higher success rates and faster learning than all other retrieval methods, with the advantage becoming dramatic on harder tasks where dense rewards are removed and full 3D hand control is required. These results demonstrate that high-quality motion-centric retrieval from internet human video provides effective guidance for complex dexterous manipulation.