One-sentence Summary

DeepCybo Team proposes PhysBrain 1.5, an 8B unified physical foundation model that extends vision-language models by encoding language responses, end-effector motion, and dense visual targets as discrete tokens optimized through autoregressive next-token prediction, is pretrained on embodied supervision from human interaction videos and fine-tuned on human demonstrations, robot trajectories, and simulated experience, and sets a new open-source embodied understanding state of the art with an average score of 72.5 across 28 benchmarks, best open-source results on 14 benchmarks while retaining general multimodal capabilities, and qualitative generation of end-effector trajectories and future RGB, depth, and robot-mask scenes.

Key Contributions

• PhysBrain 1.5 unifies embodied understanding, action generation, and future-state prediction by extending a vision-language model with discrete action and visual tokens, allowing language responses, end-effector trajectories, future RGB images, depth maps, and robot masks to be produced through a shared autoregressive next-token prediction backbone.
• The training recipe uses human interaction videos for embodied pretraining, organizing egocentric and synchronized ego-exocentric recordings into task-centered episodes that couple semantic and spatial annotations, recovered motion, and subsequent visual states, then applies supervised fine-tuning on human demonstrations, real-robot trajectories, and simulated interactions.
• The 8B model reaches an average score of 72.5 across 28 embodied understanding benchmarks, setting a new open-source state of the art, ranking first on 14 benchmarks among open-source models, and approaching proprietary systems such as GPT-6-Astra and Gemini 3.6 Flash while qualitatively producing spatially aligned future-state outputs.

Introduction

Physical intelligence depends on a continuous physical loop in which an agent observes its environment, understands spatial relationships and task goals, acts, and uses the resulting changes to guide future behavior. This matters for embodied AI because perception, reasoning, action generation, and future-state prediction are tightly coupled rather than separate capabilities.

Prior work has advanced these areas through general vision-language models, specialized spatial or planning models, robot action models, predictive world models, and system-level interfaces that connect language reasoning with robot skills. However, these capabilities remain fragmented across different model families, training objectives, data formats, and supervision structures, making joint learning difficult.

The authors address this with PhysBrain 1.5, a unified embodied foundation model that learns understanding, end-effector action generation, and multimodal future-state prediction through discrete tokens in a shared autoregressive backbone. They represent language outputs, action trajectories, RGB images, depth maps, and robot masks as tokens trained with a common next-token prediction objective, using task-specific loss masks for different outputs. The authors leverage human interaction experience as the main pre-training source, using egocentric and synchronized ego-exocentric videos to connect semantic annotations, spatial information, recovered motion, and subsequent visual states. Supervised fine-tuning then incorporates human, real-robot, and simulated demonstrations. Across 28 embodied benchmarks, the authors report strong results, positioning PhysBrain 1.5 as a step toward broader physical intelligence through interaction-centered learning.

Dataset

• The authors organize the dataset into two training stages: physical-aware pretraining and embodied supervised fine-tuning. Both stages provide supervision for perception and understanding, action generation, and future-state prediction.

• The pretraining corpus is built from human interaction videos. Sources include Xperience-10M, Egocentric-10K, Ego4D, EgoVerse, EgoDex, EgoLife, Ego-Exo4D, and two in-house corpora: PhysBrain-Human and PhysBrain-Ego360. These cover egocentric, exocentric, panoramic, daily activity, tool use, human object interaction, and long horizon tasks.

• The pretraining videos are segmented into task coherent episodes using task and action boundaries. Low quality episodes are filtered out, including those with severe motion blur, strong exposure problems, corrupted or missing frames, or key interactions that are obscured or outside the view. The curated source corpus is about 30,000 hours of video.

• Pretraining physical perception data contains 24.3M samples. These include structured captions, question answering pairs, and pseudo labels for object detection, segmentation, depth, pointing, counting, 3D object detection, temporal grounding, spatiotemporal grounding, and object tracking. Cross-view correspondence is supported by synchronized egocentric, exocentric, and panoramic views.

• Human action data contains 31.2M samples. The authors recover human motion and wrist end effector trajectories using the Human-as-Humanoid pipeline, then align trajectories with task descriptions and visual observations. They filter out unreliable pose recovery, missing temporal alignment, and implausible motion.

• Human interaction future-state data contains 26.8M samples. Samples pair a current observation with a future physical state. Each target includes an RGB frame, a depth map, and a human part segmentation mask for the interacting hands and arms. Filtering checks temporal offsets, annotation integrity, depth and mask validity, and cross-modal spatial alignment.

• General instruction data adds 14.9M language and vision language samples. Language sources include FLAN Collection, UltraChat, OpenAssistant Conversations, selected UltraData-SFT-2605 subsets, MetaMathQA, and Magicoder-OSS-Instruct. Vision language sources include FineVision, LLaVA-OneVision-1.5-Instruct-Data, Cambrian-7M, PixMo, ShareGPT4Video, and LLaVA-Video-178K. Cross-source deduplication is done before capability-aware sampling.

• The embodied supervised fine-tuning corpus combines high quality human interaction data, real robot trajectories, and simulated interactions. It is constructed for embodied understanding, end effector action generation, and future-state prediction. The mixture also includes 1M general language and vision language instruction samples to preserve broad multimodal capability during adaptation.

• Embodied understanding data is drawn from the datasets listed in Table 2 and grouped by annotation type. Native annotations are converted into instruction-formatted language or spatial targets for counting, depth and distance reasoning, spatial relations, and object localization. The authors preserve temporal, multi-view, and geometric evidence, standardize coordinates and output formats, and filter for semantic validity, referential clarity, geometric reliability, media integrity, and duplicate removal.

• Embodied action data is assembled from 17 real robot and simulation sources plus high quality human motion trajectories. The raw collection contains about 2,620 hours of robot and simulation trajectories and 500 hours of human motion data. The authors standardize action semantics, physical units, and gripper conventions while retaining each source’s native coordinate axes. Low-rate trajectories are resampled to improve temporal resolution without changing source sampling weights. Samples are filtered using pose and annotation validity checks plus the FrameSkip motion informativeness criterion, with uninformative segments downsampled, informative motion retained, and meaningful pauses selectively preserved. Each sample includes task description, visual observations, embodiment metadata, action frequency, and temporally aligned trajectories. Splits are constructed at the episode level.

• Embodied future-state data contains 1.2M samples built from robot and simulation trajectories. Each target frame is paired with the current observation and task context using timestamps and native frame rates. Target annotations include RGB images, metric z-depth maps from MoGe-2, and robot body masks from RoboEngine, extending the human hand and arm future-state format used in pretraining. Samples are filtered for missing data, temporal mismatch, and inconsistent cross-modal alignment.

• The text does not describe an explicit cropping strategy. Processing relies instead on temporal segmentation, task boundary selection, frame and clip sampling, timestamp alignment, coordinate normalization, cross-modal consistency checks, and quality filtering.

Method

The authors leverage the Qwen3-VL Instruct family to build PhysBrain 1.5, extending a pretrained vision-language model into a unified architecture capable of physical perception, interaction, and state transition. The central design principle is to express language, end-effector motion, and visual-state targets as discrete sequences and learn them through a shared autoregressive interface. This unification applies to the generated representations, which share the language-model backbone, token embedding, and output head.

As shown in the figure below:

To support this unified generation, the original language vocabulary is augmented with dedicated sets of action tokens and visual tokens, forming a combined vocabulary:

$$\mathcal{V} = \mathcal{V}_{\mathrm{lang}} \cup \mathcal{V}_{\mathrm{act}} \cup \mathcal{V}_{\mathrm{vis}}$$V=Vlang​∪Vact​∪Vvis​

This resulting vocabulary supports language, action, and visual-state generation through a shared embedding table and language-model output head. Let $x$x contain the visual input, instruction, and any task-dependent context, and let $y$y denote the target sequence. Across all task families, the model is optimized using masked next-token prediction:

$$p_{\theta}(y \mid x) = \prod_{i=1}^{|y|} p_{\theta}(y_i \mid x, y_{<i}), \qquad \mathcal{L}_{\mathrm{AR}} = - \sum_{i=1}^{|y|} m_i \log p_{\theta}(y_i \mid x, y_{<i})$$pθ​(y∣x)=i=1∏∣y∣​pθ​(yi​∣x,y<i​),LAR​=−i=1∑∣y∣​mi​logpθ​(yi​∣x,y<i​)

where $m_i$mi​ selects the supervised target tokens. Task formatting and loss masks specify whether the model should output natural language, an action trajectory, or a visual-state sequence, while the underlying prediction objective remains unchanged.

For embodied understanding, the model avoids a separate perception head. Instead, it retains the general image and video understanding interface of the base vision-language model and enriches it through embodied supervision. This enables the autoregressive decoder to produce natural-language answers or structured spatial targets for physical and spatial perception, multi-view understanding, and affordance reasoning.

In the embodied action generation module, human motion is converted into robot action representations via a Human-as-Humanoid pipeline, retaining the positions and orientations of both wrists. Each action is modeled as a short end-effector trajectory chunk of length $H = 16$H=16. The relative orientation is encoded using a continuous 6D representation, where for a rotation matrix $R$R, the encoding is defined as:

$$\rho_6(R) = \mathrm{vec}(R_{[:, 1:2]}) \in \mathbb{R}^6$$ρ6​(R)=vec(R[:,1:2]​)∈R6

The complete action target combines relative translation, relative rotation, and absolute gripper closure:

$$a_{t,k} = \left[ p_{t+k} - p_t, \rho_6(R_t^\top R_{t+k}), g_{t+k} \right] \in \mathbb{R}^{10}, \quad k = 1, \dots, H$$at,k​=[pt+k​−pt​,ρ6​(Rt⊤​Rt+k​),gt+k​]∈R10,k=1,…,H

These continuous trajectories are discretized using an ActionPiece tokenizer, which provides a 512-token action vocabulary and encodes a wrist trajectory into a sequence of 32 discrete action tokens. To handle heterogeneous coordinate frames across different data sources, the model employs homogeneous action continuation. It uses the immediately preceding action chunk as a local motion context, allowing the model to infer the local action convention and continue the trajectory naturally without requiring global canonicalization. The preceding and target action chunks are encoded as $z^- = \mathcal{T}_{\mathrm{AP}}(A^-)$z−=TAP​(A−) and $z^+ = \mathcal{T}_{\mathrm{AP}}(A^+)$z+=TAP​(A+), turning action prediction into conditional trajectory continuation.

The visual-foundation generation module models the future physical state as a spatially aligned combination of an RGB image, a depth map, and a robot mask. All three modalities are tokenized with a shared VQ-VAE at a target resolution of 128 by 128. Single-channel depth maps and masks are replicated across three channels before tokenization, producing a 16 by 16 grid of discrete codes for each modality. To provide local cross-modal context during autoregressive generation, the model employs spatially aligned joint serialization. It interleaves the three modalities at each spatial location in RGB-depth-mask order:

$$y^{\mathrm{vis}} = [b_{\mathrm{start}}, q_1^{\mathrm{rgb}}, q_1^{\mathrm{depth}}, q_1^{\mathrm{mask}}, \dots, q_N^{\mathrm{rgb}}, q_N^{\mathrm{depth}}, q_N^{\mathrm{mask}}, b_{\mathrm{end}}]$$yvis=[bstart​,q1rgb​,q1depth​,q1mask​,…,qNrgb​,qNdepth​,qNmask​,bend​]

where $b_{\mathrm{start}}$bstart​ and $b_{\mathrm{end}}$bend​ are boundary tokens. This sequence contains 768 VQ tokens and two boundary tokens. At inference time, the generated payload is de-interleaved and reconstructed separately using the frozen VQ decoder.

The training process consists of two stages: physical-aware pre-training followed by supervised fine-tuning, with the second stage initialized from the first-stage checkpoint. Both stages jointly optimize text generation, action-token generation, and future visual-state generation under the unified autoregressive objective. The authors utilize a distributed Adam optimizer with a learning rate that warms up over the first 3 percent of training steps to a peak of $2 \times 10^{-5}$2×10−5, followed by cosine decay. The maximum sequence length is set to 32,768 tokens with sequence packing enabled, and the gradient norm is clipped at 1.0.

Experiment

The experiments evaluate PhysBrain 1.5 across the perception, spatial understanding, planning, grounding, trajectory reasoning, general multimodal understanding, action generation, and future state prediction stages of an embodied physical loop. On 28 embodied benchmarks, the model shows broad strength and competitiveness among open-source and leading proprietary systems, while retaining general multimodal capabilities comparable to its base model. Additional action-trajectory and visual-prediction examples qualitatively validate its ability to produce structured motion continuations and plausible future scene, depth, and robot-mask states under task instructions.

The two-stage data design first uses large-scale human interaction videos for physical-aware pre-training and then fine-tunes on curated human, real-robot, and simulation data. Pre-training covers perception, action, and future-state supervision with large human-video sources, while embodied SFT uses a smaller multimodal and robot or simulation mix focused on understanding, action, and future state. Across the cited evaluations, action predictions broadly follow reference motion trends, and future-state predictions preserve scene layout while suggesting task-consistent changes. Pre-training relies mainly on human interaction videos, while embodied SFT broadens supervision with real-robot and simulated interaction data. Action and future-state supervision shrink in SFT, with future-state data forming the smallest supervised category. The data pipeline consistently covers perception or understanding, action generation, and future-state prediction to support physical-loop learning.

The embodied-understanding corpus combines diverse vision, spatial, egocentric, and robot interaction datasets into instruction-formatted supervision. It spans five capability groups, from foundational visual-spatial perception to spatial and multi-view understanding, embodied reasoning and planning, grounding and affordance, and visual trajectory reasoning. The composition is largest for spatial and multi-view understanding, while grounding and affordance, embodied reasoning, foundational perception, and visual trajectory reasoning cover complementary perception, action, and spatial tasks. Spatial and multi-view understanding is the largest capability group in the embodied-understanding data. Foundational perception and visual trajectory reasoning are the smallest capability groups among those listed. Embodied reasoning and planning supervision includes task state, progress, next action, procedural order, failure diagnosis, and corrective behavior. Grounding and affordance data cover localization, pointing, free-space grounding, placement, contact regions, and functional affordance. Visual trajectory reasoning uses image-space traces and waypoints for approach, contact, transport, and placement.

The supervised fine-tuning corpus combines large-scale robot and simulation trajectories with a substantial human motion component. The sources cover bimanual platforms, single-arm platforms, simulation benchmarks, and high-quality human interactions, creating diverse embodiments, viewpoints, tasks, and motion distributions. The data are processed to align action semantics, units, temporal sampling, and task context before constructing episode-level splits. Single-arm platform datasets contribute the largest fraction of the robot and simulation trajectories. Bimanual platforms provide the second largest source, while simulation and human motion data add complementary task and embodiment diversity. Samples are filtered for valid poses, motion informativeness, and temporal alignment with visual observations and task descriptions.

PhysBrain 1.5 is the strongest evaluated open-source model on embodied understanding, with the highest overall average and many benchmark wins. It outperforms its 8B base model on every benchmark and remains close to leading proprietary models overall. Its strength is broad, reaching the top two open-source models across perception, spatial reasoning, planning, grounding, and trajectory understanding. PhysBrain 1.5 leads the open-source embodied ranking by a clear margin It outperforms its Qwen3-VL base model on all embodied benchmarks It shows top two open-source strength across all capability categories

PhysBrain 1.5 preserves broad multimodal understanding close to the performance of Qwen3-VL-Instruct 8B. It shows small gains on several image and hallucination robustness benchmarks, while experiencing slight reductions on a few video understanding benchmarks. PhysBrain 1.5 improves relative to the base model on MMStar, RealWorldQA, and POPE, indicating stronger image understanding and reduced object hallucination. Video-oriented benchmarks such as VideoMME and MVBench show modest declines, but the overall suite remains comparable to the base model.

The experiments evaluate a two-stage embodied learning setup, where physical-aware pre-training on large human videos is followed by supervised fine-tuning on curated multimodal, robot, and simulation data. Data analyses show broad coverage of perception, action, future-state prediction, spatial reasoning, grounding, planning, trajectory reasoning, and heterogeneous robot motions. Model results indicate that PhysBrain 1.5 becomes the leading open-source model for embodied understanding, improves over its base model across benchmarks, and remains competitive with strong proprietary systems. Additional general multimodal tests show that embodied tuning largely preserves base capabilities, with stronger image understanding and hallucination robustness and only mild video understanding trade-offs.