One-sentence Summary

DAMO Academy, Alibaba Group, et al. propose RynnWorld-4D, a tri-branch diffusion model that co-produces future RGB frames, depth maps, and optical flow from a single RGB-D image and a language instruction using cross-modal attention and frame-wise 3D RoPE, and RynnWorld-4D-Policy, which directly outputs robot actions from the model's internal 4D representations in a single forward pass, achieving state-of-the-art on real-world dexterous bimanual manipulation tasks after training on Rynn4DDataset 1.0, a curated dataset of over 254.4 million frames.

Key Contributions

• RynnWorld-4D is introduced, a generative model that co-produces future RGB frames, depth maps, and optical flow from a single RGB-D image and a language instruction, using a tri-branch diffusion architecture with cross-modal attention and 3D rotary position embeddings to ensure consistent appearance, geometry, and motion evolution.
• Rynn4DDataset 1.0 is curated, a large-scale hybrid dataset of over 254.4 million frames sourced from egocentric human and robotic manipulation videos, with high-quality pseudo-labels for depth and optical flow, enabling training of scalable 4D world models.
• RynnWorld-4D-Policy is proposed, an inverse dynamics head that extracts internal 4D representations from the world model in a single forward pass, bypassing multi-step denoising, and achieves state-of-the-art closed-loop control performance on real-world dexterous bimanual manipulation tasks.

Introduction

Robotic manipulation in open-world settings requires visual world models that predict how scenes evolve under agent actions. Prior generative video models are limited to 2D pixel outputs, which discard critical 3D spatial relationships and suffer from geometric inconsistencies like fluctuating object scales. Existing 4D scene modeling methods either rely on expensive per-scene optimization, demand multi-view inputs, or lack the scalability of pretrained video diffusion priors. The authors propose a projective 4D representation that jointly generates synchronized RGB, depth, and optical flow sequences, preserving compatibility with large-scale 2D video diffusion backbones while making scene geometry and per-point 3D motion explicit. They introduce RynnWorld-4D, a tri-branch world model that co-generates these modalities through cross-modal attention, and a policy head that leverages the model’s internal 4D features to enable high-frequency, closed-loop robotic control.

Dataset

The authors introduce Rynn4DDataset 1.0, a large-scale hybrid dataset built from over 254.4 million video frames. It combines human-centric and robotic manipulation sources, and every frame is enriched with three types of pseudo-annotations: fine-grained captions, monocular depth, and dense optical flow.

Dataset composition and sources

• Human-centric datasets: Epic-Kitchens, EgoVid
• Robotic manipulation datasets: RoboMIND, RDT-1B, Galaxea, RoboCoin, AgiBot

No per-subset sizes or filtering rules are specified in the provided text; the total frame count is given as 254.4 million.

Annotation pipelines

• Video captioning: Qwen3-VL generates structured descriptions for 5‑second clips sampled at 1 FPS. The prompt asks for subject, action, environment, objects, interactions, and overall scene context. Captions (max 512 tokens, temperature 0.7) are stored as JSON.
• Optical flow: DPFlow estimates flow on sequential frame pairs at native resolution. Flow fields are color‑encoded and saved as MP4 videos at 25 FPS.
• Depth: Depth Anything 3 (DA3NESTED-GIANT-LARGE-1.1 checkpoint) predicts dense per‑frame depth and camera poses. Video is sampled at 30 FPS and processed at a working resolution of 392 pixels on the short side. Depth maps are clipped to [0, 5] meters, quantized to 8‑bit grayscale, and saved as RGB videos.

Usage in the model The dataset provides the pseudo‑annotations needed to train the authors’ model (presumably for 4D video understanding). The excerpt does not describe specific training splits, mixture ratios, or cropping strategies.

Method

To address the data scarcity in 4D generative modeling, the authors introduce Rynn4DDataset 1.0, a large-scale hybrid dataset specifically curated for training feed-forward 4D generative models. It comprises over 254.4 million video frames from human-centric and robotic manipulation datasets.

Each frame is enriched with high-quality 4D pseudo-annotations, including fine-grained instructions, monocular depth, and dense optical flow. The pipeline for multimodal annotation is illustrated in the figure below.

Video captioning uses Qwen3-VL to generate detailed, structured descriptions of video clips. Optical flow annotation employs DPFlow to compute dense per-frame motion fields. Depth annotation utilizes Depth Anything 3 to produce monocular depth predictions, which are upsampled to the original resolution and quantized to 8-bit grayscale.

A key advantage of the RGB-DF representation is its inherent geometric interpretability. By combining the co-generated depth and optical flow, the model can reconstruct a temporally consistent 3D scene and derive metric scene flow. Given the generated depth map $D_t$Dt​ at frame $t$t, each pixel $\mathbf{p}_t = [u, v, 1]^\top$pt​=[u,v,1]⊤ in homogeneous coordinates is unprojected into the 3D camera space as:

$\mathbf{P}_t = D_t(u, v) \cdot \mathbf{K}^{-1} \mathbf{p}_t$Pt​=Dt​(u,v)⋅K−1pt​

where $\mathbf{K}$K is the camera intrinsic matrix. To capture the underlying 4D dynamics, the dense optical flow $\mathbf{f}_{opt} = [\Delta u, \Delta v]^\top$fopt​=[Δu,Δv]⊤ is leveraged to establish temporal correspondences. A 3D point $\mathbf{P}_t$Pt​ is tracked to its position at $t+1$t+1 by:

$\mathbf{P}_{t+1} = D_{t+1}(u + \Delta u, v + \Delta v) \cdot \mathbf{K}^{-1} (\mathbf{p}_t + [\Delta u, \Delta v, 0]^\top)$Pt+1​=Dt+1​(u+Δu,v+Δv)⋅K−1(pt​+[Δu,Δv,0]⊤)

The 3D scene flow is then defined as $\mathbf{f}_{3D} = \mathbf{P}_{t+1} - \mathbf{P}_t$f3D​=Pt+1​−Pt​, representing the per-point metric displacement. As shown in the figure below, given an input RGB-D image and description, RynnWorld-4D generates RGB, depth, and optical flow videos synchronously, which can be further lifted into 3D scene flow.

To achieve synchronized generation of RGB-DF sequences, the authors extend a pretrained video generative model into a tri-branch architecture. Refer to the framework diagram for an overview of the RynnWorld-4D pipeline.

This decoupled design allows each modality to model its unique feature distributions, such as complex textures for RGB, spatial geometry for depth, and motion displacements for flow, mitigating representation interference among divergent modalities.

To enforce cross-modal consistency, a Joint Cross-Modal Attention (JA) module is introduced. It is inserted every three transformer blocks across all 30 layers. Before cross-modal mixing, each branch $m \in \{\text{rgb}, \text{depth}, \text{flow}\}$m∈{rgb,depth,flow} receives a learnable modality embedding $\mathbf{e}^m$em and is normalized by a per-modality LayerNorm $\mathrm{LN}^m$LNm to align numerical scales across branches:

$\tilde{\mathbf{z}}_l^m = \mathrm{LN}^m (\mathbf{z}_l^m + \mathbf{e}^m)$z~lm​=LNm(zlm​+em)

Each branch produces one query and one shared key/value pair that is reused by all other branches' queries, reducing parameter cost. Tokens are reshaped so that cross-modal attention is restricted to tokens of the same temporal frame across modalities, and 3D Rotary Positional Embeddings are applied. Each query attends only to the keys/values of the two complementary modalities:

$\mathbf{A}_l^m = \mathrm{Attn} (\mathrm{RoPE}(\mathbf{Q}_l^m), \mathrm{RoPE}(\mathbf{K}_l^{\text{cross}}), \mathbf{V}_l^{\text{cross}})$Alm​=Attn(RoPE(Qlm​),RoPE(Klcross​),Vlcross​)

Instead of double zero-initialization, the authors combine a zero-initialized output projection with a learnable gate $g_l^m$glm​ initialized to 1:

$\hat{\mathbf{z}}_l^m = \mathbf{z}_l^m + \tanh(g_l^m) \cdot \operatorname{OutProj}_l^m (\mathbf{A}_l^m)$z^lm​=zlm​+tanh(glm​)⋅OutProjlm​(Alm​)

At initialization, the zero-initialized output projection guarantees a smooth warm start, while $\tanh(g_l^m) \neq 0$tanh(glm​)=0 ensures non-zero gradients flow into the gate, preventing the joint pathway from being trapped at the origin.

To bridge the significant distribution gaps between modalities, a phased training paradigm is proposed. In Stage 1 (Modality Adaptation), the Joint Cross-Modal Attention is disabled, and the three branches are trained independently. In Stage 2 (Joint Attention Training), JA modules are inserted. The backbone and per-branch self-attention/FFN are frozen; only the JA projections, RMSNorms, per-modality LayerNorms, tanh gates, and modality embeddings are trainable. In Stage 3 (Full-Parameter Joint SFT), the entire model is unfrozen and continued on the full dataset.

In Stages 2 and 3, Branch Dropout is applied. With a certain probability, one of the depth or flow branches is randomly selected and its noisy latent is replaced with pure Gaussian noise, forcing the JA modules to reconstruct it from visible modalities. The RGB branch is never dropped, as it serves as the appearance anchor.

All three stages are optimized using the flow matching objective. For each modality $m \in \mathcal{M} = \{\text{rgb}, \text{depth}, \text{flow}\}$m∈M={rgb,depth,flow}, a velocity field $\mathbf{v}_\theta^m$vθm​ is learned to transport Gaussian noise $\boldsymbol{\epsilon}^m$ϵm to data $\mathbf{z}_0^m$z0m​. The total loss is:

$\mathcal{L}_{\text{total}} = \sum_{m \in \mathcal{M}} \lambda_m \mathbb{E}_{\mathbf{z}_0^m, \boldsymbol{\epsilon}^m, t, \mathbf{c}} \left[ \| \mathbf{v}_\theta^m (\mathbf{z}_t^m, t, \mathbf{c})_{[1:]} - (\boldsymbol{\epsilon}^m - \mathbf{z}_0^m)_{[1:]} \|_2^2 \right]$Ltotal​=∑m∈M​λm​Ez0m​,ϵm,t,c​[∥vθm​(ztm​,t,c)[1:]​−(ϵm−z0m​)[1:]​∥22​]

where $\mathbf{c}$c denotes the text prompt together with the initial RGB-D observation. A single Gaussian noise sample is shared across the three modalities so that their denoising trajectories stay temporally aligned.

The authors leverage RynnWorld-4D as a predictive 4D vision encoder. Given the current RGB-D observation and instruction, a forward pass through the frozen RynnWorld-4D yields a latent trajectory encoding future dynamics. Intermediate hidden states across all branches are extracted and concatenated. To compress 4D features, a Flow Former with learnable queries $\mathbf{Q}$Q is used. It processes the features via frame-wise spatial cross-attention followed by temporal self-attention:

$\mathbf{Q}_i' = \operatorname{Spat-CrossAttn}(\mathbf{Q}_i, F_p[i]), \quad \mathbf{Q}'' = \operatorname{FFN}(\text{Temp-SelfAttn}(\mathbf{Q}')), i \in \{1, \dots, T\}$Qi′​=Spat-CrossAttn(Qi​,Fp​[i]),Q′′=FFN(Temp-SelfAttn(Q′)),i∈{1,…,T}

Finally, a flow matching policy generates actions, conditioned on the predictive 4D tokens $\mathbf{Q}''$Q′′, text embedding, and proprioception. During inference, the action is derived via an ODE solver in 4 steps, enabling high-frequency closed-loop control through parallel action chunking.

Experiment

The experiments evaluate a 4D world model that jointly generates RGB, depth, and optical flow sequences, and a downstream policy that leverages its frozen predictive latents for robotic manipulation. The model is trained with a phased curriculum to adapt RGB priors and fuse cross-modal features, then tested on six real-world bimanual tasks with randomized initial states. Results show that the synchronized tri-branch generation significantly improves geometric and motion accuracy over separate training, and the policy consistently outperforms 2D-based baselines, especially on tasks requiring precise spatial coordination and dynamic object transfer. Ablations confirm that each component, including modality-specific adaptation, large-scale 4D pre-training, and the use of predictive 4D latents instead of static image encoders, is crucial for the model's success.

The inference pipeline is dominated by the RynnWorld-4D forward pass, which accounts for roughly 90% of the total latency. Depth estimation and VAE preparation contribute a small minority of the time, while the action generation head adds negligible overhead. The overall cycle time of about 1.1 seconds yields a control frequency near 9 Hz. The tri-branch Transformer inside RynnWorld-4D is the primary bottleneck, consuming 89.5% of the total inference time. Depth estimation adds only 7.7% and the combined action head (Flow Former and Flow Matching) takes just over 1%, making the policy head extremely lightweight. The full pipeline achieves a control frequency of approximately 9 Hz with a cycle time around 1.1 seconds.

The model is trained in three stages: first, all branches are trained independently with no cross-modal fusion; second, joint cross-modal attention modules and modality embeddings are trained while the backbone is frozen; finally, all parameters are jointly fine-tuned. The flow loss weight is increased from 0.5 to 1.0 after the first stage, and branch dropout is applied only in the joint training stages to encourage cross-modal reliance. Training progresses from independent branch training to full joint fine-tuning, with an intermediate stage that trains only the cross-modal attention and modality embeddings while keeping the backbone frozen. The optical flow loss coefficient is doubled from 0.5 to 1.0 after the initial independent stage, and branch dropout is introduced only during joint training stages, starting at 0.2 and decreasing to 0.1 in the final stage.

The real-world dataset contains 2,400 episodes for training the RynnWorld-4D world model and 1,200 episodes for the RynnWorld-4D-Policy, spanning six bimanual tasks. Tasks like Dual Picking and Block Pushing have the most world model data (500 episodes each), while Hand-over, Lid Placement, and Bowl Stacking have 300 episodes each. The policy set consistently uses 200 episodes per task. Dual Picking and Block Pushing are the most represented tasks in the world model training set, each with 500 episodes. The policy training set is uniformly half the size of the world model set, with 200 episodes per task across all six tasks.

RynnWorld-4D matches the visual quality of state-of-the-art video generation models while delivering significantly better reconstruction fidelity, and it outperforms existing 4D world models by a wide margin in both geometry and motion accuracy. Ablations reveal that jointly training the three modalities, using modality-specific adaptation, and injecting 3D positional information into cross-modal attention are essential for maintaining cross-modal consistency and achieving high-fidelity 4D generation. RynnWorld-4D achieves nearly double the depth accuracy of prior 4D models and uniquely supplies synchronized optical flow with low error, a capability absent in most baselines. Removing 3D RoPE from joint cross-modal attention causes a severe drop in geometric fidelity and motion precision, confirming that spatially-aware feature fusion is critical for aligning depth and flow with RGB content.

RynnWorld-4D-Policy outperforms all baselines on six robotic manipulation tasks, with the largest gains in tasks requiring spatial precision and dynamic object transfer. Foundation models π₀ and π₀.₅ achieve near‑zero success on the Hand‑over task, while the 4D policy reaches 28.57%. Ablations show that removing the predictive 4D encoder or using only RGB latents causes substantial drops, confirming that joint spatial and motion dynamics are critical for robust policy learning. Foundation models π₀ and π₀.₅ fail on the Hand‑over task (2.86% and 0% success), while RynnWorld‑4D‑Policy reaches 28.57%. Replacing the RynnWorld‑4D encoder with a static ResNet‑18 (w/o RynnWorld‑4D) drops Dual Picking success from 94.29% to 71.43%.

The evaluation examines RynnWorld-4D, a joint RGB-depth-flow generative model, and its derived policy on six bimanual manipulation tasks. RynnWorld-4D surpasses prior 4D models in reconstruction fidelity and motion accuracy, and ablations confirm that cross-modal fusion with 3D positional encoding is essential for maintaining geometric and motion consistency. The RynnWorld-4D-Policy, which leverages the predictive 4D encoder, outperforms all baselines, with the largest gains on tasks requiring spatial precision and dynamic coordination, while the inference pipeline reaches a 9 Hz control frequency with the world model forward pass as the primary computational bottleneck.