One-sentence Summary

NVIDIA, the University of Michigan, and New York University introduce \text{DynPose-100\text{K}}, a large-scale dataset of dynamic Internet videos with camera pose annotations, built via a filtering pipeline that combines task-specific and generalist models and a pose-estimation approach integrating point tracking, dynamic masking, and structure-from-motion to outperform state-of-the-art methods and support downstream applications such as realistic video generation and simulation.

Key Contributions

• Introduces DynPose-100K, a large-scale dataset of 100,000 dynamic Internet videos annotated with camera poses, sourced from Panda-70M and publicly released, surpassing concurrent datasets such as CamCo (12K) and B-Timer (40K).
• Presents a video filtering pipeline that combines task-specific specialist models with a vision-language model, achieving far higher selection precision than alternative approaches.
• Proposes a dynamic camera pose estimation pipeline integrating BootsTAP long-term tracking, dynamic masking, and Theia-SfM, reducing pose error by up to 90% and enabling DUSt3R fine-tuning that outperforms synthetic data from MonST3R.

Introduction

The authors introduce DynPose-100K, a large-scale dataset of dynamic Internet videos annotated with camera poses, designed to advance computer vision and robotics applications such as camera-controlled generative models, view synthesis for extended reality, and imitation learning in realistic simulation environments. Prior datasets fall into two constrained categories: small synthetic collections (typically under 500 videos) that suffer from a sim2real gap and high asset costs, or real-world captures that rely on restricted domains like egocentric, self-driving, or turntable-style footage with dense viewpoints, LiDAR, or multi-camera setups that simplify structure-from-motion (SfM). These approaches limit diversity and applicability to unconstrained Internet video, where moving objects, post-processing, and varied appearance make standard SfM unreliable.

The authors' main contribution is a full pipeline that overcomes these challenges. A filtering stage combines specialized experts to reject unsuitable videos (cartoons, heavy post-processing, lack of reference frames) alongside a generalist vision-language model that catches broader issues, applied to 3.2 million videos from Panda-70M to yield 100K high-quality candidates. For pose estimation, they merge state-of-the-art motion masking, long-term correspondence tracking, and SfM with robust global bundle adjustment, replacing the weaker optical-flow-based correspondences and simpler masking used in prior work. Their experiments show the filtering pipeline selects videos with far higher precision than alternatives, and the pose estimation reduces error by up to 90% across metrics and settings.

Dataset

The authors introduce DynPose-100K, a large-scale video dataset of dynamic Internet content with camera pose annotations. The dataset is constructed by filtering and processing videos from the Panda-70M dataset.

Composition and Sources

• The dataset is curated from the 3.2 million videos in the Panda-70M dataset.
• The filtering process begins with 3.2 million videos and systematically reduces this pool to a final set of 100,000 dynamic videos.
• Videos are selected based on three criteria:
  • Real-world quality videos without post-processing or non-perspective distortion.
  • Feasibility for pose prediction, excluding severe zoom, shot changes, or overly blurred backgrounds.
  • Dynamic camera and scene content, with moving cameras and non-static subjects.

Filtering Pipeline and Scale

• The authors filter the initial 3.2 million videos through several sequential stages for efficiency:
  • Lightweight filters (Hands23, optical flow, focal length) reduce the set to 1.63 million videos.
  • A distortion filter brings the count to 1.53 million.
  • Point tracking reduces the set to 679,000 videos.
  • Dynamic object masking leaves 462,000 videos.
  • A vision-language model (GPT-4o mini) applies the final strict filtering to leave 137,000 videos.
• Pose estimation is run on these 137,000 videos. The authors keep only videos where at least 80% of frames are registered, resulting in the final 100,000 videos.
• The filtering pipeline achieves high precision and recall. On a held-out test set of 1,000 videos (Panda-Test), the pipeline reaches a precision of 0.78 at the threshold used for dataset collection.

Pose Estimation and Processing Details

• Dynamic masking is performed using a combination of models:
  • OneFormer for semantic segmentation of common dynamic classes.
  • Hands23 for segmenting hand-object interactions.
  • Motion segmentation from RoDynRF to handle dynamic objects outside common classes.
  • SAM2 for mask propagation to ensure smooth masks across frames.
• Correspondences are estimated using the BootsTAP point tracking method, applied in a sliding window fashion over a grid of points.
• Global bundle adjustment (Theia-SfM) is used to estimate the final camera poses, using correspondences extracted from tracklets while excluding pairs where points fall within dynamic masks.

Dataset Statistics

• DynPose-100K contains 100,000 videos, far more than existing dynamic camera pose datasets, which are often limited to specific settings such as kitchens or driving.
• Caption analysis shows the videos span a wide range of nouns and verbs, indicating diverse content.
• The majority of videos are between 4 and 10 seconds long, providing enough duration for substantial camera motion.
• The apparent size of dynamic objects varies from small to large. Videos where dynamic objects occupy nearly the entire frame are excluded as pose estimation becomes infeasible.

Method

The authors leverage state-of-the-art methods to address the challenges of pose estimation on dynamic internet videos, such as occluding dynamic objects and appearance changes in static scenes. Refer to the framework diagram for the overall architecture.

The system processes the input video through two parallel branches: point tracking and dynamic masking.

In the point tracking branch, the system estimates dense correspondences using a sliding window approach. Specifically, the method utilizes BootsTAP to track a grid of points forward across several frames. This sliding window strategy ensures that each frame maintains sufficient correspondences even if previous windows suffer from occlusion or drift. By tracking points over extended durations, the method generates long-term correspondences that aid in loop closure and reduce drift.

Simultaneously, the dynamic masking branch identifies and segments regions containing moving objects to prevent them from corrupting the pose estimation. This masking process is composed of four complementary components. First, semantic segmentation using OneFormer identifies common dynamic classes like humans and vehicles. Second, object interaction segmentation using Hands23 masks held objects. Third, motion segmentation using RoDynRF handles dynamic regions not covered by semantic classes, such as rustling leaves, by removing regions with high Sampson error based on optical flow. Finally, mask propagation using SAM2 ensures smooth masks across frames and precise boundaries.

These two streams converge in the final stage: global bundle adjustment. The system extracts correspondences from the point tracklets, explicitly excluding any pairs containing frames where the tracklet falls within a dynamic mask. These static correspondences are then fed into Theia-SfM for global bundle adjustment. This step effectively handles the challenges posed by internet videos, resulting in accurate camera poses.

Experiment

The experiments validate both the video filtering pipeline and the resulting DynPose-100K dataset for camera pose estimation. On Panda-Test, the filtering method achieves high precision and recall, surpassing all baselines with each component contributing, particularly the VLM which provides the largest boost. On the photorealistic Lightspeed benchmark with ground truth poses, the method reduces trajectory error by 50% over the best alternative, and on real dynamic Internet videos with annotated correspondences it cuts mean error by over 35% on sequences where all methods succeed. Static and learning-based baselines struggle to register dynamic scenes, while the proposed pipeline consistently handles large dynamics and varied appearance.

DynPose-100K contains far more videos than existing diverse datasets, though each video is short, resulting in fewer frames. The dataset emphasizes high dynamics and diverse content, whereas many alternatives are limited to specific settings or are private. DynPose-100K has the highest number of videos among diverse datasets, while existing alternatives are often limited to narrow domains like kitchens, driving, or walking. Short video lengths (4 to 10 seconds) in DynPose-100K provide rich dynamics with substantial camera motion despite yielding fewer total frames per video. The largest comparable datasets are either private or restricted to uniform capture settings, unlike DynPose-100K's diverse Internet video sources.

The proposed pose estimation method consistently registers all test sequences and substantially reduces trajectory error compared to existing static and dynamic approaches, with the largest improvements on sequences where all methods succeed. Static methods often fail to register or produce inaccurate trajectories, while other dynamic methods show higher errors or instability on certain clips. Ours cuts trajectory error by 50% across all videos and by 90% on sequences where every method converges. Static methods like COLMAP and DUSt3R fail to register many challenging sequences or yield high trajectory and rotation errors. Dynamic alternatives such as MonST3R and LEAP-VO improve over static baselines but still produce less accurate trajectories than the proposed method.

Fine-tuning DUSt3R with DynPose-100K achieves a lower mean reprojection error on dynamic Internet videos than training with synthetic data from MonST3R, while maintaining comparable or better accuracy across most error thresholds. This improvement is obtained despite using far fewer training frames, demonstrating the efficiency of real-world dynamic video supervision. DynPose-100K fine-tuning reduces mean error to 8.78 versus 9.86 for synthetic MonST3R training, and raises the proportion of videos within 10 and 30 pixel error thresholds. Training uses only 140K frames compared to 1.3M for MonST3R, yet achieves similar or better registration accuracy, indicating efficient supervision.

The experiments validate DynPose-100K as a diverse, high-dynamics video dataset with more videos than existing alternatives, despite shorter clip lengths, and show its effectiveness for training. The proposed pose estimation method consistently registers all test sequences, reducing trajectory error by about half overall and by 90% on sequences where all methods converge, outperforming static and other dynamic baselines. Fine-tuning DUSt3R on DynPose-100K yields lower reprojection error and better registration accuracy than training on synthetic data from MonST3R, even with far fewer training frames, highlighting the efficiency of real-world dynamic supervision.