One-sentence Summary

Robbyant researchers present LingBot-Video, a Mixture-of-Experts video pretraining paradigm for embodied intelligence that scales a DiT-based architecture from scratch, integrates internet and robot footage through a data profiling engine, and enforces physical realism via a multi-dimensional reward system, releasing the first open-source MoE video foundation model to bridge digital creation and physical actuation.

Key Contributions

• A sparse Mixture-of-Experts (MoE) video diffusion framework with a scalable training paradigm is introduced, enabling an efficient trade-off between modeling capacity and inference cost for complex spatiotemporal dynamics.
• A dedicated data profiling engine systematically analyzes, filters, and rebalances heterogeneous video sources, integrating internet-scale videos with robot manipulation, navigation, and egocentric datasets to strengthen grounding in physical interactions and action semantics.
• A multi-dimensional reward system incorporates physical plausibility and task-completion signals, moving beyond aesthetic and text-alignment objectives to encourage learning of dynamics consistent with embodied environments.

Introduction

The authors address the use of diffusion-based video models as implicit world simulators for embodied intelligence, where such models can support planning and policy learning in robotics and autonomous driving. Prior video foundation models are tuned for perceptual quality rather than physical correctness, rely on dense computations that limit scalability, and are trained on internet videos that lack robot-specific interaction dynamics and physical grounding. The work introduces LingBot-Video, a sparse Mixture-of-Experts diffusion transformer that integrates a robot-augmented pretraining corpus with a multi-dimensional reward system to achieve efficient, physically consistent, and embodiment-aware video generation for active embodied reasoning.

Dataset

The authors build a large-scale multi-modal training corpus by combining a variety of image and video sources, with a dedicated focus on embodied and action-rich content. The dataset is processed through a unified infrastructure that profiles, annotates, and organizes every sample, then feeds it into a progressive five-stage training curriculum.

• Data Profiling Engine: Every raw sample is converted into a structured, queryable record covering structural metadata (resolution, frame rate, shot boundaries via TransNetV2), semantic labels (objects, scenes, styles, entities), motion and camera dynamics (camera vs. subject motion, tracked motion from LocoTrack), cinematic attributes (shot size, angle, lighting, etc.), and quality/aesthetics (HPSv3, AI-generated content detection via OmniAID, clarity, exposure, artifacts like watermarks and subtitles). These records drive all downstream filtering and sampling.

• World-Knowledge Topological Graph: Semantic tags are expanded, embedded, and clustered into a hierarchical tree of 50,000 fine-grained leaf concepts and 25 top-level visual groups. For videos, an action tree with several hundred canonical action nodes (manipulation, sports, daily activities) is built from normalized and description-augmented action tags. The graph enables distribution-aware sampling: rare or difficult nodes are up-weighted, and loss feedback from early training stages refines sampling weights to focus on under-represented or challenging content.

• Dense Structured Captioning: All training data is re-annotated with structured JSON captions. For images: global description, camera tag set, world-knowledge entities, and a list of prominent elements with location, size, texture, and person attributes (pose, clothing, etc.). For videos: the schema adds timestamped actions per element and camera movement descriptions. Robot manipulation (VLA) videos and egocentric videos use the video schema with minor adaptations (e.g., dropping gender fields, annotating hands and grippers with phased actions).

• Stage-wise Data Curriculum: The corpus is consumed in five progressive pre-training stages, with resolution and mixture evolving over time.

• Stage 1: 192p images only, filtered for aesthetic quality and minimum resolution, discarding lowest-tier samples.

• Stage 2: Introduces 192p video alongside a tighter image pool; video admission uses resolution, aesthetic, and motion filters (combining VLM-based and geometry-grounded motion signals). This stage injects over 70,000 hours of embodiment-oriented footage—robot manipulation (real-robot, simulated, open-source, humanoid/quadruped platforms), navigation, egocentric video, and text-rich video.

• Stage 3: Scales to 480p, with stricter aesthetic and motion criteria, keeping high-motion video while maintaining rigorous image quality gates.

• Stage 4: 480p rebalancing for embodied intelligence; general videos face heavy quality filtering, while scarce high-value sources (manipulation, navigation, egocentric, benchmark data) undergo minimal filtering to maximize coverage of long-tail action-centric data.

• Stage 5: A small, high-quality 1080p video subset (well under 1% of the initial pool) serves as the refinement set for the cascaded refiner.

• Training Data Packing: During pre-training, heterogeneous samples are packed into 1D sequences based on a target token budget. Visual and conditioning tokens are concatenated with attention masks, allowing dynamic scheduling and efficient joint processing of images and videos without modality-specific loaders.

Method

The authors leverage a cascaded design consisting of a task-unified base generator and a refiner. The base generator employs a Single-Stream Diffusion Transformer to process compact visual latents and multimodal conditions.

Unified Input and Single-Stream Backbone Each training sample is represented as a single token sequence consisting of visual latent patches and condition tokens. After projecting visual patches and condition features into the same hidden dimension, they are concatenated along the sequence dimension. This formulation handles Text-to-Image, Text-to-Video, and Image-to-Video tasks within a single framework. To resolve structural discrepancies, a 3D Multi-Modal RoPE mechanism places condition and visual tokens in non-overlapping temporal coordinate ranges.

The backbone is a streamlined single-stream diffusion transformer. All visual and condition tokens share the same transformer blocks, maximizing parameter reuse and facilitating dense cross-modal interactions. To stabilize attention in deep backbones, queries and keys are normalized with per-head RMSNorm. Modulation overhead is reduced via an AdaLN-Single design, computing a shared timestep modulation once before the transformer stack.

Scaling with Sparse Mixture-of-Experts To scale parameter capacity under a constant computational budget, the authors incorporate a sparse Mixture-of-Experts framework into the base generator. In each transformer block, the dense feed-forward computation is replaced with a token-choice sparse MoE layer. Refer to the framework diagram for the overall architecture.

The MoE design incorporates fine-grained expert segmentation and shared expert isolation. Given the modulated FFN input $\mathbf{u}_t$ut​ of token $t$t, the Sparse MoE layer computes a branch output from shared experts and routed experts: $m(\mathbf{u}_t) = \sum_{i=1}^{N_s} E_i^{(s)}(\mathbf{u}_t) + \sum_{j \in \mathcal{R}_b(\mathbf{u}_t)} g_{t,j} E_j^{(r)}(\mathbf{u}_t)$m(ut​)=∑i=1Ns​​Ei(s)​(ut​)+∑j∈Rb​(ut​)​gt,j​Ej(r)​(ut​) where $E_i^{(s)}$Ei(s)​ and $E_j^{(r)}$Ej(r)​ denote shared and routed expert functions, respectively. Shared experts provide a common pathway for general physical principles, while routed experts capture specialized features.

Token routing uses a sigmoid router to compute affinities. To control communication costs, a group-limited routing strategy is adopted. Load balancing is maintained via an auxiliary-loss-free strategy using dynamic correction bias updated online. Additionally, a sequence-wise auxiliary balance loss encourages balanced expert usage within each packed video sequence.

MoE Recipe Exploration The authors systematically explore the MoE design space. First, they evaluate scaling the expert pool size under a fixed active parameter scale. As shown in the figure below, scaling the expert count yields consistent improvements in training and validation losses. They choose an expert count of 128 as the default to balance performance and overhead.

Next, they compare fine-grained routing against coarse routing under a fixed total parameter budget. As shown in the figure below, the fine-grained routing recipe performs consistently better, highlighting the advantages of fine-grained expert specialization and a larger combinatorial routing space.

Cascaded Refiner To balance computational complexity and generation quality, a cascaded design is adopted. A high-capacity base generator models motion and scene layout at 480p, followed by a dedicated refiner that upsamples to 1080p.

During training, synthetic degradations such as Gaussian blur and compression are applied to construct degraded low-resolution inputs. The refiner learns a conditional rectified flow starting from the degraded condition $\mathbf{x}_{lr}$xlr​ toward the clean target latent $\mathbf{x}_0$x0​. The perturbed latent $\mathbf{x}_t$xt​ and target velocity $v_{ref}^*$vref∗​ are formulated as: $\mathbf{x}_t = \left(1 - \frac{t}{\tau}\right) \mathbf{x}_0 + \frac{t}{\tau} \mathbf{x}_\tau, \quad v_{ref}^* = \frac{\mathbf{x}_\tau - \mathbf{x}_0}{\tau}$xt​=(1−τt​)x0​+τt​xτ​,vref∗​=τxτ​−x0​​ The model is optimized using flow-matching loss restricted to timesteps $t \leq \tau$t≤τ. As shown in the figure below, the refiner significantly enhances face appearance and restores high-frequency details.

Data Profiling Engine The performance relies on a scalable data infrastructure. The Data Profiling Engine converts raw multimodal samples into structured, multi-dimensional records capturing structural, semantic, motion, camera, and quality attributes. Refer to the framework diagram for an overview of this pipeline.

These standardized records drive downstream processing stages, from filtering and sampling to captioning. Core annotations are generated by vision-language models and specialized scoring models.

Experiment

Extensive scaling experiments demonstrate that the proposed sparse single-stream diffusion transformer scales efficiently, with sparse models outperforming similarly sized dense baselines and achieving near-parity inference latency while offering significant speed advantages over larger dense counterparts. Post-training with multi-aspect rewards, including fine-grained physical plausibility, motion coherence, and human-motion consistency, guides the model toward more realistic and physically grounded generation. Internal and public benchmarks confirm LingBot-Video's state-of-the-art performance in embodied and physical domains, particularly under image-conditioned generation, and user studies further validate its strong open-source standing against both open and commercial models. Action-to-video post-training shows that the model generalizes beyond training trajectories, adhering more faithfully to physical laws and action specifications on out-of-distribution tasks.

LingBot-Video, an open-source model, achieves the highest average RBench score (0.620), surpassing all other open-source models and slightly edging out the closed-source Wan 2.6 (0.607). It excels in task-oriented categories like tasks and multi-entity, as well as dual-arm embodiment, while Wan 2.6 leads in reasoning, single-arm, and long-horizon dimensions. A separate Physics-IQ evaluation confirms LingBot-Video’s top open-source rank with a score of 40.4, just ahead of Cosmos 3 (39.5). LingBot-Video’s RBench average of 0.620 leads all open-source models, with Cosmos3 Super at 0.581 and Wan 2.2 A14B at 0.507, and it narrowly beats the closed-source Wan 2.6 (0.607). On task-oriented sub-scores, LingBot-Video scores highest in tasks (0.578) and multi-entity (0.634), while Wan 2.6 dominates reasoning (0.666) and long-horizon (0.531). In embodiment-specific metrics, LingBot-Video’s dual-arm score (0.758) is the strongest overall, whereas Wan 2.6 leads single-arm (0.681) and Cosmos3 Super closely matches LingBot-Video in quadruped (0.691 vs. 0.689) and humanoid (0.691 vs. 0.689). On a separate Physics-IQ benchmark, LingBot-Video attains a Verified score of 40.4, the highest among open-source models, narrowly surpassing Cosmos 3 (39.5) and with larger gaps to models like HunyuanVideo 1.5 (33.4).

Across RBench and Physics-IQ evaluations, LingBot-Video sets a new state-of-the-art for open-source models, outperforming the closed-source Wan 2.6 on average and excelling in task-oriented and dual-arm scenarios. Its top Physics-IQ score further demonstrates robust physical reasoning, while Wan 2.6 retains an edge in reasoning and single-arm sub-tasks.