One-sentence Summary

Google DeepMind, University of Toronto, et al. propose GenCeption, a perception model built on a pre-trained video generative diffusion backbone that leverages large-scale text-to-video generation as a pre-training paradigm, achieving state-of-the-art performance across diverse vision tasks including depth, surface normal, and camera pose estimation, expression-referring segmentation, and 3D keypoint prediction, often matching or surpassing specialized models while exhibiting superior data efficiency ($7\times$7× to $500\times$500× less training data than leading models) and emergent generalization to real-world footage and out-of-distribution categories, suggesting video generation as a foundational path toward generalist vision intelligence.

Key Contributions

• GenCeption repurposes a pretrained text-to-video diffusion model into a feed-forward perception architecture, directly translating spatiotemporal priors and vision-language alignment from video generation into text-steered visual tasks without iterative sampling.
• GenCeption achieves state-of-the-art or matching performance against specialized models on depth, surface normal, camera pose, expression-referring segmentation, and 3D keypoint estimation, and the video generative backbone outperforms alternative pretraining paradigms such as V-JEPA and Video MAE under comparable settings.
• The model exhibits exceptional data efficiency, matching leading specialists like D4RT and VGGT-Ω with 7× to 500× less training data, and demonstrates emergent generalization, transferring from synthetic human videos to real-world footage and out-of-distribution categories including animals and robots.

Introduction

The field of computer vision remains fragmented into specialized models, each designed for a particular task with a custom architecture, unlike NLP where a single foundation model unifies diverse challenges. Prior perception models, such as Segment Anything or Depth Anything, excel at individual tasks but lack an inherent understanding of spatiotemporal dynamics and cannot flexibly handle multiple vision tasks under a common framework. While self-supervised video representation learning and diffusion-model repurposing have advanced, they struggle to scale due to high computational cost, weak vision-language alignment, or a focus on single-image or single-task outputs without temporal consistency.

The authors introduce GenCeption, a generalist video perception model that treats large-scale text-to-video generation as a universal pre-training paradigm. By repurposing a pre-trained video diffusion backbone into a single feed-forward architecture, the model internalizes 4D world priors, aligns vision with language, and scales with data and compute. Post-training on diverse synthetic data enables GenCeption to perform a wide range of pixel-level, 3D, and sparse perception tasks using a unified architecture, matching or surpassing specialized state-of-the-art models while exhibiting data efficiency, sim-to-real transfer, and out-of-distribution generalization.

Dataset

The authors construct a synthetic dataset to supply the multi-modal ground truth needed for training, sidestepping limitations of real-world datasets.

• Source assets and generation The dataset is built from 800 RenderPeople human assets animated with 200 motions from the CMU motion capture dataset. Diverse 3D full scenes and HDRI backdrops are used as backgrounds, while focal lengths, camera positions, and trajectories are varied to enrich viewing conditions. This yields 7,500 synthetic videos covering a wide range of identities, motions, and environments.

• Annotations and modalities Each video provides RGB frames along with pixel-level ground truth for depth, surface normals, segmentation masks, dense pose, 2D keypoints, 3D keypoints, and camera poses. Depth, normals, and masks are rendered via separate passes in Blender. Human joint positions are recovered from the rigged RenderPeople assets to serve as ground truth for 2D and 3D keypoint regression.

• Processing and preparation All generated videos are trimmed to the target number of frames required by the video model. Before training, an offline preprocessing stage precomputes and caches the RGB video latents, the target-modality video latents, and the corresponding text conditioning embeddings.

• Usage in the model The full synthetic dataset is used as the primary training source. The paper does not describe dividing it into separate training, validation, or test splits, nor does it specify mixture ratios with other datasets.

Method

The authors introduce GenCeption, a framework that reformulates video perception as a post-training endeavor built atop a pretrained video generative model. The methodology is driven by three foundational principles: leveraging multimodal generative pre-training for robust representation learning, adopting a task-agnostic post-training approach within a unified architecture, and transforming the multi-step generative backbone into a single-step, feed-forward perception model.

Refer to the framework diagram for the overall architecture.

The model repurposes a text-to-video diffusion architecture, which comprises a VAE encoder-decoder pair, a text encoder, and a transformer-based latent diffusion model (DiT). To transition from a generative model to a perception model, the authors replace the iterative sampling process with a single forward pass. Specifically, the clean latent of the input video is fed directly into the DiT, and the conditioning timestep is fixed to $t=0$t=0 to signify a noise-free input. Since the DiT is trained under the Rectified Flow objective to predict the velocity $v = \epsilon - x_0$v=ϵ−x0​, the raw output is negated prior to decoding. The negated output $-v = x_0 - \epsilon$−v=x0​−ϵ aligns more closely with the target video latent, which empirically accelerates convergence and improves performance.

To handle diverse vision tasks without task-specific architectures, the authors unify dense tasks within the standard 3-channel RGB space. Single-dimensional outputs like depth and segmentation replicate the three RGB channels, while three-dimensional tasks like surface normals use distinct dimensions. For higher-dimensional modalities such as camera poses, the authors employ a pixel-space raymap that fits within the 3-channel constraint.

As shown in the figure below, the 6-channel ray data is spatially partitioned to create a "Rothko" Raymap, allocating ray origins to the central region and ray directions to the periphery.

This layout preserves the single-decoder framework while fully leveraging pre-trained visual priors. For sparse tasks requiring structured outputs like 2D or 3D coordinates, the authors append learnable tokens to the video latents. An MLP decodes these tokens to predict the target dimensions per frame, utilizing the native 3D RoPE of the base DiT model with learnable spatial positions and interpolated temporal positioning.

To support this unified architecture, the authors design a scalable synthetic data generation workflow. They utilize RenderPeople assets animated with CMU motion capture data, generating diverse human-centric videos with ground-truth annotations for depth, normals, segmentation, dense pose, and keypoints. The training recipe employs a fully unified $L_2$L2​ loss applied in the latent space for dense tasks and the output space for sparse tasks. Task-specific customization is shifted entirely to data representation. For instance, to resolve scale ambiguity in monocular depth estimation, depth maps are normalized using the median depth and mapped to the standard RGB range via a nonlinear function $d' = \text{clip}(\alpha \log(d + 1), 0, 1)$d′=clip(αlog(d+1),0,1). This data-centric harmonization eliminates the need for specialized loss functions and allows task balancing to be managed solely through data mixture ratios.

Experiment

The evaluation covers a wide range of vision tasks including geometric understanding, segmentation, and keypoint estimation, using a unified feedforward model built on a video diffusion backbone and trained primarily on synthetic data. The experiments demonstrate that the model not only matches or surpasses specialized state-of-the-art methods across these benchmarks but also exhibits emergent generalization from synthetic to real-world videos, zero-shot handling of multiple instances, and out-of-distribution class robustness, all while being remarkably data-efficient. Ablations reveal the critical importance of generative video pretraining and the challenges of joint training for tasks requiring architectural modifications, highlighting the potential of video diffusion models as general-purpose spatiotemporal feature extractors.

A single unified model, trained primarily on synthetic data, matches or surpasses heavily specialized state-of-the-art models across diverse dense prediction tasks. It outperforms dedicated surface normal estimators and is competitive with leading depth and camera pose foundation models, while also benefiting from joint training on foreground segmentation. On surface normal estimation, the model exceeds specialized approaches like NormalCrafter and Lotus-2. For depth and camera pose, the method is on par with or outperforms dedicated foundation models, despite using only synthetic training data.

A generative video pretraining approach (WAN 2.1) achieves superior depth estimation compared to representation learning methods like V-JEPA and VideoMAE v2 when fine-tuned on the same small synthetic dataset. The model exhibits clear scaling with size, and the 14B variant attains better average accuracy than a state-of-the-art depth model trained on orders of magnitude more real data, underscoring both the strength of the diffusion prior and remarkable data efficiency. Diffusion-based pretraining (WAN 2.1) substantially outperforms V-JEPA and VideoMAE v2 on depth estimation across Sintel, KITTI, and ETH3D, despite all methods using the same limited synthetic training set. Scaling the generative backbone from 1.3B to 14B parameters consistently reduces average AbsRel from 0.122 to 0.093 and lifts δ1 from 85.8 to 90.7, showing promising model scaling behavior. Trained on only 7.5K synthetic videos, the 14B WAN 2.1 model achieves better average depth metrics (AbsRel 0.093, δ1 90.7) than DepthAnything V3 trained on over 1.2M videos from 22+ datasets, highlighting exceptional data efficiency.

A single model trained primarily on synthetic data matches or surpasses heavily specialized state-of-the-art models across diverse dense prediction tasks, outperforming dedicated surface normal estimators and remaining competitive with leading depth and camera pose foundation models. Generative video pretraining with WAN 2.1 substantially exceeds representation learning methods like V-JEPA and VideoMAE v2 on depth estimation when fine-tuned on the same small synthetic dataset, exhibits clear scaling with model size, and achieves better average accuracy than a depth foundation model trained on orders of magnitude more real data, highlighting exceptional data efficiency.