One-sentence Summary

DAMO Academy and Alibaba Group Hupan Lab introduce RynnUnify, a 16B unified embodied model combining embodied understanding and long-horizon visual planning through interleaved video-language reasoning, with agentic reinforcement-learning and world-action extensions that support anticipatory reasoning and robot control, and it achieves the highest average embodied understanding score among evaluated unified multimodal models while improving the Rynn-VP overall score by 9.75 points over Cosmos 3 Nano.

Key Contributions

• RynnUnify is introduced as a 16B unified embodied model that combines embodied understanding with long-horizon visual planning through interleaved video-language reasoning, using scene memory and self-forcing to improve visual consistency during long-horizon generation.
• Rynn-VP and Rynn-GEU are introduced as benchmarks for long-horizon visual planning and anticipatory reasoning; RynnUnify achieves the highest average embodied understanding score among the evaluated unified multimodal models and improves the Rynn-VP overall score by 9.75 points over Cosmos 3 Nano.
• An agentic extension learns to selectively use video predictions for reasoning about unobserved states, improving the Rynn-GEU average by 6.0 points over direct reasoning; a world-action extension translates the model's understanding and prediction capabilities into robot control, achieving 51.5% success on 20 selected composite RoboCasa365 tasks and 46.7% average success across three real-robot tasks.

Introduction

General robot brains require unified models that understand physical scenes, anticipate how actions change the world, and translate task intent into action. Prior unified multimodal models can jointly handle understanding and generation, but long-horizon coordination remains difficult because prediction errors accumulate and static visual outputs may omit intermediate dynamics. The authors introduce RynnUnify, a 16B embodied model based on Cosmos3's Mixture-of-Transformers architecture, pairing an 8B autoregressive reasoner with an 8B diffusion generator to perform interleaved video-language reasoning for long-horizon visual planning. They also add an agentic extension that decides when to generate videos for anticipatory reasoning, a World-Action Model extension for robot control, and two benchmarks, Rynn-VP and Rynn-GEU, that evaluate sustained visual planning and generation-enhanced embodied understanding.

Dataset

Dataset Overview

The authors use the data summarized in Table 1 to fine-tune two components: a reasoner that plans and produces world-modeling instructions, and a generator that predicts the corresponding videos. The training data combine general understanding, visual planning, and generation-enhanced understanding examples.

1. General Understanding Data

• Uses a 10% subset of the RynnBrain pretraining corpus.
• Covers general image and video understanding, visual question answering, temporal reasoning, instruction following, and embodied cognition tasks such as object understanding, spatial reasoning, counting, and egocentric task understanding.
• Includes spatially grounded supervision: object localization, area localization, affordance localization, trajectory prediction, and manipulation planning.
• This subset preserves broad embodied understanding while the model is fine-tuned on more specialized planning and perception data.

2. Visual Planning Data

• Sources: RynnBrain-Planning, HiFi-UMI, OpenAoE, HIW-500, AgiBot-Beta, RoboMIND, RoboCOIN, RoboTwin, RoboCasa365, plus egocentric loco-manipulation videos from the RynnUnify collection.
• Covers human demonstrations, UMI demonstrations, real-robot interactions, and simulated manipulation.
• Processing:
  • If explicit subtask annotations exist, the authors use them. Otherwise, Qwen3.6-27B performs temporal segmentation and relabeling.
  • Video clips are kept to 2 to 10 seconds where possible.
  • GPT-5.6-Terra rewrites each subtask instruction into a structured visual prompt describing the scene, relevant objects, interaction progression, and resulting state changes.
• Filtering:
  • An open-source MLLM scores individual clips and full trajectories on visual quality, operation completeness, and instruction-video alignment. Low-scoring samples are removed.
  • GPT-5.6-Terra analyzes base-model videos to identify capability gaps, and examples from similar tasks and scenes are combined with randomly sampled data.
• Usage:
  • Interleaved samples preserve the original task goal and temporal order.
  • Observations are interleaved with subtask instructions, structured visual prompts, and operation durations.
  • The reasoner learns to predict the next subtask and estimate duration.
  • Aligned prompt-video pairs supervise the generator, and consecutive clips support video continuation across subtask boundaries.

3. Generation Enhanced Understanding, or GEU, Data

• Each GEU example is an interleaved conversation: the model receives an observation and question or task instruction, first produces a world-modeling instruction, receives a video as visual context, and then returns a reasoning response ending with an answer or spatial output.

GEU Subsets

• RynnUnify-Phys

  • Built from rendered videos and simulation annotations in PhysInOne.
  • Each sequence is split into an observed prefix and a future continuation.
  • Question-answer pairs derive from object trajectories and event annotations in the continuation.
  • Each SFT example contains: observed video and question, world-modeling instruction, ground-truth continuation video, and a reasoning response ending with the answer.
  • GPT-5.6-Luna rewrites world-modeling instructions using timestamped frames from the observed prefix.
  • Audits check factual grounding, interaction coverage, and future-outcome leakage.
  • Contains 10,028 SFT examples and 1,553 numerical questions for reinforcement learning.

• RynnUnify-Mess

  • Built from cluttered tabletop scenes in RoboTwin with controlled interventions such as pulling or removing objects.
  • Records object poses, contact and support relations, and events including support loss, falling, and visibility changes.
  • GPT-5.6-Terra rewrites video-generation prompts; MiniMax-H3 generates videos conditioned on these prompts and simulated initial and final frames in FL2V mode.
  • Automatic motion and visibility criteria and GPT-based post-filtering remove artifacts such as teleportation, severe interpenetration, and unrelated instability.
  • Each example combines an initial observation and intervention question, a video-generation prompt, an intermediate video rollout, and an answer.

• RynnUnify-Spatial

  • Covers position, direction, distance, and size reasoning.
  • Position subset built from ScanNet++ using camera intrinsics and extrinsics, 3D bounding boxes, and per-view visibility.
  • Rule-based templates create four position question types: object-camera, camera-camera, camera motion, and object-object.
  • Position subset contains 100,000 examples: 99,200 training and 800 test, with 200 test examples per question type.
  • Direction, distance, and size examples are sampled from RynnBrain video-based spatial perception data and reformulated by Qwen3.8-27B.
  • MiniMax-H3 generates intermediate visual context in FL2V mode; some size examples use single-image I2V generation.
  • GPT-5.6-Luna produces chain-of-thought annotations.

• RynnUnify-ActionReasoning

  • Built from physics-aware planning demonstrations in RynnBrain, including AgiBot, Galaxea, RoboCasa, RoboCOIN, RoboMIND, RoboTwin, SoftFold, and self-captured data.
  • Episodes are split into adjacent observed-context and future-continuation clips of 2 to 6 seconds.
  • Covers seven capabilities: action-primitive selection, translation and reach, action-intent recognition, push direction, rotation and orientation, metric action, and affordance and feasibility.
  • Quality control combines multimodal judging and applicability scoring.
  • The authors retain the top 2,000 examples per category, giving 14,000 examples total.
  • Split: 13,300 training and 700 test examples, with 1,900 train and 100 test per category.

• RynnUnify-Affordance and Area

  • Built from a subset of RynnBrain-Planning with temporally ordered atomic subtasks.
  • Subtasks include language instructions and spatial annotations: target object bounding boxes, affordance points, and placement area points.
  • Videos are segmented into manipulation clips.
  • RynnUnify-Affordance asks the model to identify an interaction point on the target object.
  • RynnUnify-Area asks the model to localize a target region such as a placement destination.
  • GPT-5.6-Terra produces world-modeling descriptions from subtask clips.
  • Contains 20,039 affordance samples and 13,774 area samples, totaling 33,813 training examples.

• RynnUnify-Trajectory

  • Built from RH20T manipulation videos.
  • Each example pairs the initial frame of a manipulation clip with a task instruction.
  • The target is a 2D trajectory represented as at most ten points, with coordinates normalized to the range 0 to 1000.
  • Execution clips are at most 8 seconds long and sampled at 24 frames per second.
  • Qwen-VL-Max generates initial world-modeling descriptions from the clean initial frame; these are rewritten using four frames from the execution clip.
  • Examples with projected trajectory path length below 20 pixels are excluded.
  • Contains 9,892 examples: 9,592 training and 300 test.

4. Evaluation Benchmarks

• Rynn-VP

  • Contains 200 manually collected egocentric task scenarios for long-horizon visual planning.
  • Given an initial image and task instruction, the model generates a task-execution video or an ordered sequence of step images.
  • Reference demonstrations and timestamped subtask annotations are reserved for evaluation.
  • Scoring uses three dimensions: Planning Intelligence, Visual Realization, and Visual Quality.
  • Overall score: 0.5 Planning Intelligence plus 0.4 Visual Realization plus 0.1 Visual Quality.

• Rynn-GEU

  • Contains ten subsets covering spatial relationships, physical prediction, chain reactions, action reasoning, affordance grounding, area grounding, and trajectory prediction.
  • Construction follows the corresponding RynnUnify-GEU training pipelines, but evaluation scenes are separated from training scenes.
  • Mess uses RoboTwin2.0 scenes absent from training.
  • Spatial, Trajectory, and Action Reasoning exclude scenes that appear in or closely resemble training data.
  • Phys uses distinct PhysInOne assets and scenes.
  • Affordance and Area use AgiBot-Beta, a different source from the training data.
  • Future-oriented sample selection favors questions answered well with future visual evidence and poorly without it.
  • Evaluation uses task-specific metrics on a common 0 to 100 scale.

Method

RynnUnify adopts a Mixture-of-Transformers (MoT) architecture comprising two distinct branches: an autoregressive reasoner for multimodal understanding and language generation, and a diffusion generator for predicting continuous visual latents through rectified flow. The two branches utilize separate Transformer parameters and interact via joint attention, enabling the generator to attend to representations produced by the reasoner.

The reasoner processes multimodal observations and language context to support embodied understanding, spatial grounding, and task planning. For visual planning, it takes the task goal, initial and current observations, and preceding subtask history as input, predicting the next executable subtask along with a detailed description of its anticipated visual evolution and an estimated duration.

For video generation, visual inputs are encoded using a causal video VAE with spatial and temporal compression factors of 16 and 4, respectively. The resulting latents are spatially patchified into visual tokens. Both scene-memory frames and the main video follow this VAE pathway. When scene memory is enabled, the token sequence is organized as:

where $\mathbf{S}_{\mathrm{AR}}$SAR​ denotes the textual conditioning tokens processed by the reasoner, $\mathbf{Z}_{\mathrm{mem}}^{(i)}$Zmem(i)​ denotes the visual tokens of the $i$i-th reference frame, and $\mathbf{z}_t$zt​ denotes the main video tokens at denoising time $t$t.

To support long-horizon planning, the authors introduce cross-chunk scene memory. This consists of four independently encoded reference frames placed before the main video tokens, comprising a persistent initial observation, two keyframes from earlier history, and a reference from the most recent step. Additionally, a continuous video prefix from the preceding chunk occupies the beginning of the main video temporal sequence. The model supports three generation modes: image-to-video (I2V), first-and-last-frame-to-video (FL2V), and cross-chunk video continuation, where specific latent frames remain fixed throughout denoising.

The training procedure follows a multi-stage pipeline designed to develop complementary capabilities in the reasoner and generator.

In Stage 1, the authors fine-tune the reasoner to develop spatiotemporal understanding, physical-world grounding, subtask planning, and video-prompt formulation. The model is optimized using a standard autoregressive loss over assistant response tokens:

where $\mathcal{I}_{\mathrm{R}}$IR​ indexes the assistant response tokens. Long-horizon visual planning examples are constructed by interleaving visual observations with subtask instructions and structured visual prompts.

In Stage 2, the reasoner is frozen, and the generator is fine-tuned to translate structured visual prompts and scene context into video predictions. The authors employ self-forcing strategies to improve robustness during long-horizon rollout. These include condition degradation, where Gaussian noise and downsampling-upscaling blur are applied to starting condition frames; offline endpoint replacement, where the starting observation is replaced by the final frame of an offline-generated preceding clip; and joint history and endpoint replacement, which extends replacement to the preceding videos used for condition construction. The generator is optimized using a flow-matching objective applied only to the main video positions requiring prediction:

where $\Omega$Ω indexes the main video positions to be predicted.

Stage 3 introduces Branch-Conditional Policy Optimization (BCPO), an agentic reinforcement learning framework that jointly optimizes reasoning behavior and generator invocation. The reasoner chooses between direct reasoning and generation-assisted reasoning. Within-branch reward comparisons improve reasoning along each interaction path using a sequence-level GSPO objective, while between-branch reward differences teach the reasoner to select the path that benefits the task. The complete objective combines branch-specific reasoning, invocation learning, and policy regularization:

Finally, the authors extend RynnUnify to a World-Action Model (WAM) for robot control through post-training.

In this architecture, the reasoner takes the task instruction, current RGB observations, and visual history as input to produce a continuous subtask latent. This latent conditions the generator through layerwise attention to predict future video. An 18-block action head reads the generator video hidden states through cross-attention and uses flow matching to predict the velocity field for an action chunk. The WAM is trained in three stages: reasoner training on subtask annotations, generator training using the subtask latent, and action-head training where the generator and action head are jointly optimized while keeping the reasoner frozen. An auxiliary camera-pose head provides geometric supervision during the final stage.

Experiment

The introduced benchmarks evaluate long-horizon visual planning through simulator-free generated rollouts and anticipatory embodied understanding with generated visual context. Foundational embodied understanding experiments show that RynnUnify provides strong spatial, interaction, and trajectory reasoning, while agentic visual generation further improves reasoning when prediction is selective and grounded, though unreliable generated context can harm other models. Interleaved planning results indicate that retaining scene memory and adapting to imperfect predictions improves task progression and visual consistency. Simulation and real-robot evaluations on mobile manipulation and history-dependent tasks confirm that the post-trained model coordinates movement, precision, and memory-dependent execution better than baselines.

The fine-tuning data spans general understanding, visual planning, and embodied perception and reasoning. General understanding draws from a large sample pool but uses only a 10% subset. Visual planning is led by robot clips, closely followed by human clips and a smaller simulated set, while embodied reasoning samples outnumber perception samples by more than three to one. Robot visual planning clips form the largest planning collection, closely followed by human visual planning clips and a smaller RoboCasa365 contribution. Embodied reasoning data is more than three times larger than embodied perception data.

RynnUnify attains the highest average embodied understanding score among unified multimodal models and leads on six of ten evaluated metrics. It shows especially strong spatial understanding, with large gains over Cosmos 3 Nano on VSI-Bench and RoboSpatial, and records top results across several localization benchmarks. The improvements include more accurate grounding of task-relevant interaction points and better trajectory prediction. RynnUnify leads on six of ten embodied understanding metrics and achieves the highest average score among unified multimodal models. It outperforms Cosmos 3 Nano by double-digit margins on VSI-Bench and RoboSpatial, and achieves the best overall results on EmbSpatial, RefSpatial-Bench, and ShareRobot Trajectory.

RynnUnify achieves the strongest direct-answering performance on Rynn-GEU among the compared open-source understanding models, showing anticipatory reasoning about actions, physical changes, and spatial relationships. Adding the agentic visual-generation extension further lifts Rynn-GEU performance to the highest among all compared methods, with gains across perception, spatial, and physical tasks. Public reasoning benchmarks also improve under the agentic setup, including a notable MindCube gain. RynnUnify records the best direct-answering Rynn-GEU average among open-source understanding models. The agentic extension raises the Rynn-GEU average from 44.0 to 50.0, surpassing all compared methods including GPT-5.6-Terra. Agentic rollout improves seven of ten Rynn-GEU tasks over direct answering and outperforms other UMM rollout variants on all ten. Public benchmark average rises from 41.3 to 44.6, with a notable MindCube improvement.

Among unified multimodal models on Rynn-VP, RynnUnify achieves the highest planning intelligence, visual realization, and overall score. Its improvement over Cosmos 3 Nano is driven mainly by a large gain in planning intelligence and a moderate gain in visual realization, with visual quality slightly lower. A composed video generation system still reports the highest overall score across all methods shown. RynnUnify raises the overall score from 45.05 to 54.80 over Cosmos 3 Nano, with planning intelligence improving by more than 15 points. Visual realization improves from 54.58 to 60.54, while visual quality dips slightly from 79.69 to 77.53.

On RoboCasa365, earlier policies show much lower success rates across the composite splits and Selected-20. RynnUnify reaches roughly 50 percent on Composite-Seen and Composite-Unseen, ranking first on Composite-Unseen among compared methods. Its post-trained WAM leads Selected-20 mobile-manipulation tasks at 51.5 percent, ahead of CMA, Xiaomi-Robotics-1, and Cosmos3 by clear margins. RynnUnify achieves the best Composite-Unseen success rate among evaluated methods, outperforming CMA and Cosmos3 by notable percentage-point margins. The post-trained WAM records the highest Selected-20 success rate, exceeding the strongest baselines by roughly five to seven percentage points. Several baseline policies remain in single-digit or low-teen success rates on the full composite splits and Selected-20, far below the leading methods.

The fine-tuning data spans general understanding, visual planning, and embodied perception and reasoning, with robot visual planning clips and embodied reasoning samples forming the largest shares. RynnUnify is evaluated on embodied understanding, reasoning benchmarks, visual planning, and RoboCasa365 manipulation. It leads unified multimodal models on embodied understanding, achieves the best direct-answering Rynn-GEU score with further gains from agentic visual generation, improves visual planning over Cosmos 3 Nano, and attains strong RoboCasa365 success rates including the best Composite-Unseen and leading Selected-20 performance.