One-sentence Summary

The DeepCybo Team presents PhysBrain 1.5, an $8\text{B}$8B unified physical foundation model that jointly optimizes language, end-effector motion, and dense visual targets as discrete sequences via autoregressive next-token prediction, pre-trained solely on human interaction videos and fine-tuned on demonstrations, robot trajectories, and simulations, achieving a $72.5$72.5 average across $28$28 embodied benchmarks and setting a new open-source state of the art on par with proprietary models like GPT-6-Astra and Gemini 3.6 Flash.

Key Contributions

• Introduces PhysBrain 1.5, a unified model that jointly handles embodied understanding, action generation, and future-state prediction by encoding language responses, end-effector trajectories, and dense visual targets as discrete tokens under a single autoregressive next-token prediction objective.
• Extends an understanding-first approach by deriving all embodied pre-training supervision from human interaction videos organized into task-centered episodes, then adapting the model via supervised fine-tuning on human, real-robot, and simulated data.
• Achieves an average score of 72.5 across 28 embodied understanding benchmarks with an 8B model, setting a new open-source state of the art, ranking first on 14 benchmarks, and performing on par with proprietary models GPT-6-Astra and Gemini 3.6 Flash, with qualitative demonstrations of end-effector trajectory generation and future RGB, depth, and robot-mask prediction.

Introduction

Physical intelligence requires an agent to continuously observe, interpret, act, and re-evaluate within its environment, a cycle the authors describe as the physical loop. Prior work has advanced this goal through specialized models for vision-language understanding, spatial reasoning, and robot control, but these capabilities are often trained separately and fail to share a unified learning objective. Joint training is difficult because supervision signals differ greatly: understanding tasks produce language and coordinates, action generation requires temporally structured motion, and future-state prediction demands dense visual outputs, all with varying data formats and granularities.

To address this, the authors introduce PhysBrain 1.5, a model that unifies embodied understanding, action generation, and future-state prediction through discrete tokens. They extend the language vocabulary with ActionPiece tokens for end-effector trajectories and visual tokens for future RGB images, depth maps, and robot masks, all processed by a shared autoregressive backbone under a common next-token prediction objective. Pre-training supervision comes entirely from human interaction videos, including egocentric and ego-exocentric recordings, while supervised fine-tuning adds human, real-robot, and simulated data. On 28 benchmarks, the 8B model achieves an average score of 72.5, setting a new open-source state of the art and closely approaching proprietary systems like GPT-6-Astra (73.3) and Gemini 3.6 Flash (73.0).

Dataset

The authors organize training data into two stages: physical-aware pre-training and embodied supervised fine-tuning (SFT). Both stages provide supervision for perception and understanding, action generation, and future-state prediction. Physical-aware pre-training derives these supervision types from large-scale human interaction videos. Embodied SFT reuses the pre-training data construction pipeline and adds real-robot and simulated environment interactions for fine-tuning on high-quality data.

Physical-Aware Pre-Training Data

• The pre-training corpus aggregates Xperience-10M, Egocentric-10K, Ego4D, EgoVerse, EgoDex, EgoLife, and Ego-Exo4D, plus two in-house corpora: PhysBrain-Human (egocentric recordings with synchronized exocentric views) and PhysBrain-Ego360 (panoramic videos with full-body poses, hand motion, and task-level speech).
• Sources cover daily activities, tool use, human-object interaction, and long-horizon tasks across egocentric, exocentric, and panoramic viewpoints.
• Long recordings are segmented at task and action boundaries to create task-coherent episodes. Low-quality episodes with motion blur, exposure issues, corrupted frames, or obscured interactions are discarded.
• The curated corpus contains approximately 30,000 hours of video, from which three training data types are constructed.

Physical Perception Data

• Structured annotations, captions, and question-answer pairs are generated from sampled frames and clips using open-source and in-house models.
• Image-level supervision covers object detection, segmentation, depth estimation, pointing, counting, and 3D object detection. Video-level supervision covers temporal grounding, spatiotemporal grounding, and object tracking.
• Synchronized egocentric, exocentric, and panoramic recordings support cross-view object correspondence and referring grounding under viewpoint changes.
• Captions are built at frame, segment, and episode levels. Frame-level captions describe visible objects and scene properties; segment-level captions align actions with manipulated objects and state changes; episode-level captions summarize environment, task goal, and progress.
• For PhysBrain-Ego360, speech transcripts are combined with video evidence to derive task descriptions and step-level captions.
• Physical VQA examples probe task recognition, progress assessment, action understanding, human-object interaction, temporal order, and state changes.
• Inconsistent pseudo-labels, captions, and answers are rejected. The resulting corpus contains 24.3M training samples.

Human Action Data

• Human motion is recovered using the Human-as-Humanoid pipeline, deriving wrist end-effector trajectories from the kinematic chain.
• Temporal correspondences associate trajectories with task descriptions and visual observations.
• Videos are divided into short clips, with each sample preserving temporal relationships among an action segment, the subsequent visual observation, and the action continuation.
• Samples with unreliable pose recovery, missing temporal correspondences, or implausible motion are rejected. The corpus contains 31.2M training samples.

Human-Interaction Future-State Data

• Each sample pairs the scene context before an interaction with a subsequent physical state, using short video clips.
• Future-state targets comprise a spatially aligned RGB image, depth map, and human-part segmentation mask. The RGB image is taken from the video frame at the target timestamp; depth and segmentation masks are generated by perception pipelines.
• Temporal offsets, image integrity, depth validity, and cross-modal spatial alignment are validated. Samples with mismatches or missing data are discarded. The corpus contains 26.8M training samples.

General Instruction Data

• Pre-training also includes 14.9M general language and vision-language instruction samples.
• Language sources include FLAN Collection, UltraChat, OpenAssistant Conversations, subsets of UltraData-SFT-2605, MetaMathQA, and Magicoder-OSS-Instruct.
• Vision-language sources include FineVision, LLaVA-OneVision-1.5-Instruct-Data, Cambrian-7M, PixMo, ShareGPT4Video, and LLaVA-Video-178K.
• Cross-source deduplication is performed before capability-aware sampling.

Embodied Supervised Fine-Tuning Data

• The SFT corpus combines high-quality human interaction data, real-robot trajectories, and simulated interactions.
• It includes 1M high-quality general language and vision-language instruction samples to retain broad capabilities during embodied adaptation.

Embodied Understanding Data

• Datasets from Table 2 are grouped by annotation type, with native annotations converted into instruction-formatted language or spatial targets.
• Examples cover counting, depth and distance judgments, spatial relations, and object localization, in question answering, captioning, or structured spatial prediction formats.
• Spatial and multi-view examples retain original image, video, camera, and scene organization.
• Grounding and visual-trajectory data use a shared normalized spatial format for points, boxes, regions, and image-space waypoints.

Embodied Action Data

• Trajectories come from 17 real-robot and simulation sources plus high-quality human motion data, covering diverse embodiments, viewpoints, tasks, and motion distributions.
• The raw collection contains approximately 2,620 hours of real-robot and simulation trajectories and 500 hours of human motion data.
• Sources differ in action semantics, physical units, and temporal sampling. The authors standardize units and gripper conventions while retaining audited native coordinate axes.
• Human data reuses the Human-as-Humanoid pipeline, retaining sequences with reliable pose recovery and alignment.
• Low sampling rate sources are resampled to improve temporal resolution without changing sample anchors or source weights.
• Trajectories are filtered using the motion-informativeness criterion from FrameSkip. Uninformative segments are downsampled, new motion segments are retained, and stationary segments are selectively kept.
• Each sample includes task description, visual observations, embodiment metadata, action frequency, and temporally aligned motion trajectories. Splits are constructed at the episode level.

Embodied Future-State Data

• Built from real-robot and simulation trajectories used in the action corpus, associating task descriptions and current observations with subsequent target frames.
• Timestamps and native frame rates establish temporal correspondences; target frames come from the same continuous interaction sequence.
• Each target includes a spatially aligned RGB image, a metric z-depth map generated with MoGe-2, and a robot-body mask obtained with RoboEngine. Segmentation covers the robot body, extending the human hand-and-arm annotations used in pre-training.
• Completeness, temporal offsets, depth validity, and spatial correspondences are checked. The corpus contains 1.2M training samples.

Method

The authors introduce PhysBrain 1.5, a unified model built upon the Qwen3-VL Instruct family that extends a pretrained vision-language model to jointly handle physical perception, interaction, and state transition. The central architectural design expresses language, end-effector motion, and visual-state targets as discrete sequences, learning them through a single autoregressive interface.

As illustrated in the framework diagram, the model unifies generated representations by sharing the language-model backbone, token embedding, and output head. The original language vocabulary is augmented with dedicated sets of action and visual tokens, forming a unified vocabulary defined as:

$$\mathcal{V} = \mathcal{V}_{\mathrm{lang}} \cup \mathcal{V}_{\mathrm{act}} \cup \mathcal{V}_{\mathrm{vis}}$$V=Vlang​∪Vact​∪Vvis​

This formulation enables the joint training of understanding and generation. Perceptual supervision provides context for predicting interactions and their consequences, while action and state-transition supervision introduces physical priors that support spatial grounding and trajectory reasoning. Given visual input, instructions, and task-dependent context $x$x, and a target sequence $y$y, the model is optimized using masked next-token prediction:

$$p_{\theta}(y \mid x) = \prod_{i=1}^{|y|} p_{\theta}(y_i \mid x, y_{<i}), \quad \mathcal{L}_{\mathrm{AR}} = -\sum_{i=1}^{|y|} m_i \log p_{\theta}(y_i \mid x, y_{<i})$$pθ​(y∣x)=i=1∏∣y∣​pθ​(yi​∣x,y<i​),LAR​=−i=1∑∣y∣​mi​logpθ​(yi​∣x,y<i​)

where $m_i$mi​ selects the supervised target tokens. Task formatting and loss masks dictate whether the model outputs natural language, action trajectories, or visual-state sequences, while the underlying prediction objective remains unchanged.

For embodied action generation, the authors convert human motion into robot action representations via a Human-as-Humanoid pipeline, retaining wrist positions and orientations. Each action is modeled as a short end-effector trajectory chunk of length $H=16$H=16. Relative orientation is encoded using a continuous 6D representation, and the action target combines relative translation, relative rotation, and absolute gripper closure. These continuous trajectories are discretized using an ActionPiece tokenizer, which encodes a wrist trajectory into a sequence of discrete action tokens. To handle heterogeneous coordinate frames and motion conventions across different datasets, the model utilizes the immediately preceding action chunk as a local motion context. This allows the autoregressive decoder to infer the local action convention and continue the trajectory naturally without requiring globally canonicalized coordinate frames.

For visual-foundation generation, the future physical state is modeled as a spatially aligned combination of an RGB image, a depth map, and a robot mask. All three modalities are tokenized using a shared VQ-VAE at a target resolution of $128 \times 128$128×128, producing discrete codes that map one-to-one to atomic VLM tokens. The modalities are interleaved at each spatial location in an RGB-depth-mask order, placing tokens from corresponding spatial locations adjacent to one another to provide local cross-modal context during autoregressive generation. At inference, the generated payload is de-interleaved and reconstructed into the respective modalities using a frozen VQ decoder.

The training process proceeds in two stages: physical-aware pre-training followed by supervised fine-tuning. Both stages jointly optimize text generation, action-token generation, and future visual-state generation under the unified autoregressive objective. The authors employ the distributed Adam optimizer with a cosine learning rate schedule, training for one epoch in each stage while mixing data categories in proportion to their sample counts.

Experiment

PhysBrain 1.5 is evaluated across 28 embodied benchmarks spanning perception, spatial reasoning, planning, grounding, and trajectory understanding, where it achieves the highest average among open-source embodied models (72.5) and outperforms its base model on all benchmarks while approaching proprietary leaders. General multimodal understanding is assessed on 12 standard benchmarks, showing performance comparable to the base model and confirming that embodied specialization does not compromise broad perception and reasoning. Beyond understanding, the model demonstrates action trajectory token prediction that follows reference motion trends in both in-distribution and out-of-distribution scenes, and it predicts future visual states across RGB, depth, and robot masks, preserving scene layout while reflecting plausible task progress.

The training data is organized into two stages, physical-aware pre-training and embodied supervised fine-tuning, with each stage covering perception, action, and future-state supervision. Pre-training relies on large-scale human interaction videos, while embodied SFT incorporates additional real-robot and simulation data alongside curated human data. Both stages also include general language and vision-language instruction data. Pre-training uses human videos for all three supervision types, with action data being the largest component. Embodied SFT includes multimodal understanding data as the largest component, followed by action data from human, robot, and simulation sources. Future-state supervision in embodied SFT is sourced only from robot and simulation data, unlike pre-training which uses human videos. General instruction data is included in pre-training but not listed for embodied SFT, which instead emphasizes embodied-specific understanding and action data.

The embodied-understanding dataset is organized into five capability groups, each with distinct supervision types and representative sources, totaling over 6.6 million training samples. The largest group is spatial and multi-view understanding, while visual trajectory reasoning is the smallest, highlighting a deliberate balance across perception, reasoning, grounding, and action-related tasks. Spatial and multi-view understanding is the largest capability group, with 2.36M samples, focusing on geometric relations and viewpoint reasoning. Foundational visual-spatial perception includes counting, depth, and spatial relations, sourced from datasets like COCO and ScanNet, with 500K samples. Grounding and affordance data, at 1.8M samples, covers object localization and functional affordance using sources like RoboRefIt and Visual Genome. Visual trajectory reasoning is the smallest group at 300K, emphasizing image-space traces for approach and placement actions. All annotations are converted into instruction-formatted language or spatial targets, with filtering for semantic and geometric reliability.

The embodied-action corpus combines real-robot and simulation trajectories with high-quality human motion data, totaling over 3,000 hours. The data span multiple platform types, including bimanual, single-arm, simulation, and human interaction sources, with varied scales and semantics. The corpus is used for supervised fine-tuning to support diverse embodiments and tasks. The corpus includes 2,620 hours of real-robot and simulation trajectories and 500 hours of human motion data. Bimanual platforms contribute 987 hours, while single-arm platforms contribute 1,339 hours, making them the largest source family. Simulation benchmarks add 294 hours, and human interaction data add 500 hours, covering diverse embodiments and tasks. Source datasets differ in action semantics, physical units, and temporal conventions, requiring standardization and filtering.

PhysBrain 1.5, an 8B open-source model, achieves the highest overall average among open-source models on embodied understanding benchmarks, surpassing the strongest open-source baseline by 6.5 points. It ranks first on 14 benchmarks and among the top two on 24, while outperforming its base model on all 28 benchmarks. Across five capability categories, it consistently ranks among the top two open-source models, approaching the performance of leading proprietary models. PhysBrain 1.5 achieves the highest overall average among open-source models, with a score of 72.5. It outperforms the strongest open-source baseline by 6.5 points and ranks first on 14 benchmarks. The model ranks among the top two open-source models in all five capability categories, including perception, spatial reasoning, planning, grounding, and trajectory understanding. Despite its compact 8B scale, it approaches the performance of leading proprietary models like Gemini 3.6 Flash and GPT-6-Astra.

PhysBrain 1.5, an 8B model, maintains general multimodal understanding comparable to its base model Qwen3-VL-Instruct (8B) across image, video, and hallucination-robustness benchmarks. It shows slight improvements on MMStar, RealWorldQA, and POPE, while trailing slightly on MME, VideoMME, and MVBench. PhysBrain 1.5 scores higher than the base model on MMStar, RealWorldQA, and POPE, indicating stronger visual reasoning and lower object hallucination. The model's MME, VideoMME, and MVBench scores are slightly lower than the base model, but the overall performance remains comparable. The results suggest that embodied post-training does not sacrifice broad multimodal understanding, as the model retains strong performance across general benchmarks.

The evaluation covers a two-stage training pipeline (physical-aware pre-training and embodied supervised fine-tuning) using diverse data sources, including human videos, real-robot trajectories, and simulation data. The embodied-understanding dataset spans five capability groups with over 6.6 million samples, and the action corpus totals over 3,000 hours across multiple platforms. Results show that PhysBrain 1.5, an 8B model, leads open-source models on embodied understanding benchmarks (average 72.5, up 6.5 points over the strongest baseline) while maintaining general multimodal performance comparable to its base model, with slight gains in visual reasoning and hallucination robustness. Overall, the experiments validate that embodied post-training enhances task-specific capabilities without sacrificing broad multimodal understanding.