One-sentence Summary

Researchers from Peking University and JD.com propose PDRVLA, a vision-language-action model whose lightweight Physical Dynamics Module extracts physical tokens from vision-language features to condition action generation without online video generation, trains through direct prediction of future RGB, depth, and optical-flow features with auxiliary supervision from a frozen pretrained video world model, and achieves success rates of $99.3\%$99.3% on LIBERO, $89.9\%$89.9% on LIBERO-Plus, $93.6\%$93.6% on RoboTwin 2.0, and $81.5\%$81.5% on SimplerEnv.

Key Contributions

• PDRVLA introduces a lightweight Physical Dynamics Module that extracts compact physical tokens from intermediate vision-language features to condition action generation without online video generation. These representations are trained by directly predicting future RGB, depth, and optical-flow features with auxiliary supervision from a frozen pretrained video world model, and deployment uses the learned physical tokens without running the video generator.
• The training corpus is constructed entirely from publicly available datasets, combining real-robot, handheld UMI, and simulated demonstrations with general and robotics-oriented vision-language data. It is expanded through egocentric human-to-robot synthesis and augmented with subtask and embodied chain-of-thought annotations to support joint action and language learning.
• Evaluation on LIBERO, LIBERO-Plus, RoboTwin 2.0, and SimplerEnv yields success rates of 99.3%, 89.9%, 93.6%, and 81.5%, respectively. Ablations show improved future prediction with world-model supervision, and analyses characterize the effects of subtask conditioning, future object-pose supervision, and embodied chain-of-thought annotations on action success.

Introduction

Vision-language-action (VLA) models map visual observations and language instructions to robot actions, often relying on pretrained vision-language models and heterogeneous robot demonstrations, but their understanding of physical scene properties remains limited, including weak encoding of object mass and unreliable motion adaptation. Manipulation also requires anticipating how grasping, moving, and placing objects changes spatial configurations, not just recognizing the current scene. Pretrained video world models can learn physical dynamics by predicting future scenes, yet generating videos inside the control loop adds latency, so recent work uses future prediction only during training to build compact representations. The authors introduce PDRVLA, a VLA model with a lightweight Physical Dynamics Module that extracts physical tokens from intermediate VLM features and uses them alongside final-layer features for action generation. These tokens are trained with direct future RGB, depth, and optical-flow prediction plus auxiliary supervision from a frozen video world model, allowing the policy to benefit from physical anticipation without running a video generator at deployment.

Dataset

Corpus overview

• The authors build a training corpus that combines VLA data and VL data.
• The VLA portion pairs observations and task instructions with control targets to ground the model in robot behavior.
• The VL portion adds perceptual, spatial, and task-level reasoning beyond the coverage of action trajectories.
• The corpus is organized into five categories: real-robot trajectories, UMI demonstrations, simulated interactions, general image/video-language data, and spatial/embodied VL data.
• Exact mixture ratios and training split details are not stated in this section.

VLA subsets

• Real-robot data

  • Sources include RoboSet, DROID, BridgeData V2, BC-Z, and RH20T for single-arm tabletop and in-the-wild manipulation.
  • Bimanual and multi-platform coverage comes from RoboMIND V1/V2, RoboCOIN, and ABC-130K.
  • Humanoid, arm, and dexterous-hand data come from Humanoid Everyday and Unitree UniFoLM releases.
  • Multi-robot data includes AgiBot World Colosseo.
  • Additional manipulation trajectories come from REASSEMBLE, FuSe, MolmoAct, and selected LeRobot community subsets.
  • These sources cover pick-and-place, articulated-object interaction, tool use, contact-rich assembly, and other skills across different robots and environments.

• UMI data

  • Sources include UMI, FastUMI, FastUMI-100K, Hy-Embodied-0.5-VLA-Data, HiFi-UMI-2K, and RealOmni-Open Data.
  • After source-specific filtering, they contribute approximately 1.22 million episodes and 10,259 hours.
  • Because UMI data do not provide robot joint-space commands, the authors use temporally aligned end-effector poses directly in the unified Cartesian action representation.

• Simulation data

  • Standardized skills and long-horizon tasks come from ManiSkill2, Meta-World, and RoboCasa.
  • Large-scale cross-embodiment trajectories come from InternData-A1, GR00T-X-Embodiment-Sim, and InternVLA-M1.
  • Bimanual or dexterous manipulation data come from RoboTwin 2.0 and DexMimicGen.

VL subsets

• General VL data

  • Sources include Cambrian-1, COCO-Caption, VQAv2, and OneThinker.
  • These provide vision-centric instruction data, captioning, open-ended visual question answering, and image/video reasoning examples.

• Spatial and embodied VL data

  • Spatial-reasoning sources include SpaceThinker, OpenSpaces, SpaceOm, RefSpatial, data from Scaling Spatial Intelligence with Multimodal Foundation Models, and cross-view correspondence examples.
  • Embodied and procedural sources include RoboPoint, Robo2VLM, RoboVQA, EO-1, Cosmos-Reason1, and ProcCorpus-60M.
  • These cover object and affordance localization, manipulation video QA, interleaved perception and reasoning, physical commonsense, and procedural supervision.

Data processing and schema

• Action representation

  • Positions are converted to meters and orientations to rotation matrices.
  • Coordinate axes use a right-handed convention with +x forward, +y left, and +z up.
  • Poses remain relative to the arm base or world frame as specified by each source dataset.
  • The reference frame is recorded in metadata.
  • End-effector actions are represented as deltas from the current pose to a target pose.
  • The target is the dataset-provided absolute target if available, otherwise the next observed pose after resampling.
  • Gripper openings are normalized to 0 for fully closed and 1 for fully open. The gripper action is the normalized absolute target opening.

• Trajectory filtering

  • Entire episodes are rejected if they are too short, lack usable action targets or state sequences, contain values with unrecoverable units, cannot align image and state streams, or have task descriptions that conflict with source metadata.
  • Filtering thresholds are set per dataset or subset.
  • Frames with non-finite values, abnormal repeated observations, abrupt spikes, or out-of-range values are excluded from supervision but kept in their original temporal positions.
  • If corruption cannot be isolated to individual frames, the full episode is rejected.

• Embodiment metadata standardization

  • Robot, arm, and end-effector types are recorded separately as robot_type, arm_type, and hand_type.
  • Aliases such as panda and franka_panda are standardized while preserving the attached end-effector identity.
  • Missing or inconsistent metadata are checked against source documentation and recorded videos.

• Visual processing

  • Camera observations and robot states are aligned at the lowest rate among streams, capped at 30 Hz.
  • At most four camera views are retained per episode, prioritizing wrist and third-person views.
  • Videos are resized to 640 pixels on the longest side and encoded with a maximum GOP size of 2 frames for faster random access.
  • No separate cropping strategy is described.

Derived data and annotation

• Egocentric human-to-robot synthesis

  • The authors apply a human-to-robot pipeline from Qwen-RobotManip to EgoDex demonstrations.
  • Data are generated for 12 robot-arm models including Panda, UR5e, UR10e, FR3, Kinova Gen3, IIWA, ARX-L5, xArm7, ViperX, WidowX, Piper, and YAM.
  • Hand keypoints are mapped to end-effector pose trajectories and gripper commands, with inverse kinematics producing arm joint trajectories.
  • Visual adaptation uses SAM3 for segmentation, ProPainter for background reconstruction, robot rendering, and DA3 depth for occlusions.
  • After filtering, the synthesized corpus contains 896,832 bimanual robot trajectories and approximately 2,646 hours from 74,813 EgoDex demonstrations.

• Subtask annotation

  • Source-provided subtask labels are reused when available. Otherwise, Qwen3.8-27B annotates sparsely sampled observations.
  • Segment boundaries are merged into episode-level segmentation and propagated to every frame.
  • Annotation covers approximately 1.88 million episodes, 1.79 billion frames, and 20,101 hours.
  • About 78 million sampled frames are automatically annotated, representing 4.4% of total frames.
  • From roughly 294,000 annotated episodes, the authors construct 1.1 million single-turn VL examples.

• ECoT annotation

  • Keyframes are selected as the first and last valid frame in each subtask plus additional frames whose visual similarity falls below a threshold.
  • Qwen3.8-27B generates scene descriptions, progress assessments, and next-motion descriptions based on recorded poses.
  • The pipeline produces 2,564,851 annotated keyframes from selected real-robot and UMI datasets.
  • Released ProcCorpus-60M annotations are also used.

How the data is used

• The corpus is used as a combined VLA and VL training mixture.
• VLA examples provide supervision through observations, task instructions, and control targets.
• VL examples provide broader perceptual, spatial, embodied, and procedural reasoning supervision.
• Derived subtask and ECoT annotations add structured language targets on top of existing trajectories.
• Synthesized EgoDex data adds multi-embodiment VLA examples with robot-compatible observations and actions.
• No exact training split or mixture ratio is provided in the excerpt.

Method

The authors leverage a Qwen3.5-4B VLM backbone, a Physical Dynamics Module, and an action expert to construct the PDRVLA framework.

Given multi-view observations and a task instruction, the VLM backbone processes the input. The Physical Dynamics Module extracts physical tokens from intermediate VLM features, while the action expert utilizes the final-layer features along with the physical tokens and proprioceptive state to generate robot commands.

To capture visual information useful for predicting physical changes, the Physical Dynamics Module uses 50 learnable queries to extract a fixed set of physical tokens $C_t$Ct​ from the intermediate VLM features $H_t^{(l)}$Ht(l)​. Across 4 Transformer blocks, these queries retrieve information via cross-attention, exchange information through self-attention, and pass through a feed-forward layer. A final projection maps the outputs to the VLM hidden width, producing 50 physical tokens.

The action policy predicts a chunk of $K=8$K=8 robot commands. To handle heterogeneous robots, a fixed-dimensional 58-dimensional action vector is used, with separate slots for left and right manipulators. The action expert is a Diffusion Transformer trained with conditional flow matching. It receives a noisy action chunk, maps it to tokens, and alternates self-attention with cross-attention to the conditioning sequence $X_t = [H_t^{\text{last}}; C_t]$Xt​=[Htlast​;Ct​]. The proprioceptive state is encoded as a prefix token. At inference, the chunk is initialized with Gaussian noise and integrated using four Euler steps.

The training pipeline consists of physical-module pretraining and joint policy training.

During physical pretraining, the module is trained to encode future appearance, geometry, and motion. This combines direct prediction of future RGB, depth, and optical-flow latent features with auxiliary supervision from a frozen pretrained video model. The direct prediction loss uses a lightweight head with spatial queries, while the world-model-assisted loss guides representation learning by predicting latent velocities.

In the joint policy training stage, the video world model is removed. The VLM, Physical Dynamics Module, action expert, and direct prediction head are jointly trained on VLA and VL data. The intermediate VLM features entering the physical module are detached to prevent action losses from propagating into the VLM through this path. The total loss combines the masked velocity-matching action loss, the VL cross-entropy loss, and the direct prediction loss.

To enrich the training corpus, the authors employ human-to-robot synthesis and language annotation. Subtask annotation segments trajectories into ordered subtasks using a large language model on sparsely sampled observations. These annotations are converted into VL examples that query the model about the current, next, or remaining subtasks.

Furthermore, ECoT annotation augments keyframes with scene descriptions, progress assessments, and next-motion descriptions derived from recorded poses, providing detailed context for the action expert.

Experiment

PDRVLA is evaluated on four simulation benchmarks covering single-arm LIBERO and LIBERO-Plus perturbation tasks, bimanual RoboTwin 2.0, and WidowX SimplerEnv manipulation, with additional component studies for pretraining and policy supervision. Across these benchmarks, PDRVLA achieves top aggregate success rates and strong robustness, though some perturbation categories and individual tasks favor other methods. Component analyses show that world-model-assisted pretraining improves future RGB and depth prediction, subtask conditioning gives modest gains with periodic refreshing, ECoT training improves action success without requiring ECoT generation at inference, and auxiliary object-pose supervision provides only marginal benefit.

The core training corpus combines VLA data from real-robot trajectories, UMI demonstrations, and simulated interactions with VL data. Across VLA categories, simulation contributes the most trajectories but the least recorded hours, while real-robot data contributes the most hours. The VL portion is dominated by spatial and embodied examples, which outnumber general VL examples by roughly three to one. Simulation provides the largest VLA trajectory count but the fewest hours among VLA categories. Real-robot data provides the highest hour count among VLA categories despite having fewer trajectories than simulation. Spatial and embodied VL examples outnumber general VL examples by roughly three to one.

The synthesized and annotated training data add three large-scale components: egocentric conversion contributes nearly a million robot trajectories spanning thousands of hours, subtask annotation contributes more than a billion frame annotations and over a million VL examples, and ECoT annotation adds millions of annotated keyframes. These additions expand the corpus with behavior-adapted trajectories and visually grounded subtask and explanation-style supervision. Egocentric conversion yields on the order of a million robot trajectories and thousands of hours of adapted demonstration data. Subtask annotation is the largest frame-level contribution, with more than a billion annotated frames distilled into over a million VL examples. ECoT annotation adds millions of keyframe-level annotations for scene, task progress, and motion descriptions.

Subtask VL examples are heavily sourced from VLM-generated labels rather than source-provided native annotations. Among the listed groups, EgoDex, InternVLA-sim, and ABC-130K contribute the largest VLM-labeled shares, while RoboCOIN and GALAXEA are the only native-label sources and together provide about one fifth of examples. Many source groups contribute far fewer examples than the leading groups, indicating a long tail in dataset composition. VLM-labeled examples account for the majority of listed subtask VL data; native labels appear only for RoboCOIN and GALAXEA. EgoDex contributes the largest single share among all source groups, followed by InternVLA-sim and ABC-130K among VLM-labeled groups. RoboCOIN is the largest native-label source and ranks among the top overall contributors, while GALAXEA also contributes a substantial native-label share. A tail of smaller source groups contributes substantially fewer examples than the leading groups.

The ECoT annotation pipeline yields about 2.56 million annotated keyframes across six robotics datasets. Real-robot datasets provide the majority of these keyframes, while UMI datasets contribute a smaller but substantial share. Annotated keyframe counts vary widely by dataset, from roughly 84,000 to nearly 692,000. Real-robot data account for most annotated keyframes, with RoboCOIN contributing the largest single dataset total. Among UMI datasets, HiFi-UMI-2K contains far more annotated keyframes than FastUMI-100K, which has the smallest contribution overall. AgiBot World and ABC-130K contribute nearly equal annotated keyframe totals, each around 530,000.

PDRVLA achieves the highest aggregate success rates among the compared methods across the four simulation benchmarks, with particularly strong results on LIBERO and LIBERO-Plus. Baseline performance varies widely by benchmark, with most reported baselines clustering at high LIBERO success and lower SimplerEnv success. On LIBERO-Plus, PDRVLA leads overall but does not dominate every perturbation category, as Qwen-RobotManip performs better under robot-initial-state, sensor-noise, and object-layout changes. PDRVLA reports the top aggregate results on LIBERO, LIBERO-Plus, and SimplerEnv, and leads four of the seven LIBERO-Plus perturbation categories. Qwen-RobotManip outperforms PDRVLA under robot-initial-state, sensor-noise, and object-layout perturbations on LIBERO-Plus, while robot-initial-state is the hardest category for PDRVLA. Among baselines, LingBot-VLA 2.0 has the strongest reported RoboTwin 2.0 result, and π0 has the lowest reported SimplerEnv result. On SimplerEnv, PDRVLA's largest lead is on Eggplant to Basket, but it trails InternVLA-A1.5 on Carrot to Plate.

The experiments first characterize the training corpus and its synthesized supervision, then evaluate PDRVLA on four simulation benchmarks. The corpus is dominated by simulation trajectories in count and real-robot data in hours, with spatial and embodied VL examples outnumbering general VL data; added egocentric, subtask, and explanation annotations markedly expand behavior-adapted and visually grounded supervision. Subtask VL labels are mostly VLM-generated and concentrated in a few sources with a long tail, while real-robot datasets contribute most annotated keyframes. In robot manipulation benchmarks, PDRVLA achieves the highest aggregate success on LIBERO, LIBERO-Plus, and SimplerEnv, although Qwen-RobotManip outperforms it on several LIBERO-Plus perturbation categories.