One-sentence Summary

Researchers from NLPR, Institute of Automation, Chinese Academy of Sciences introduce PHIZERO, a physical world model that, unlike prior pixel-space predictors, learns a compact discrete physical language from in-the-wild videos via self-supervision to explicitly reason about world-state transitions before rendering them into videos, demonstrating coherent generation, interactive simulation, and zero-shot motion transfer.

Key Contributions

• The paper introduces physical language, a compact discrete representation of state-transition patterns learned at scale from in-the-wild videos through self-supervised learning.
• It develops PHIZERO, a physical world model that adopts a reason-then-render paradigm: future world evolution is first inferred as a physical-language sequence, then rendered into videos.
• Extensive experiments validate that PHIZERO generates physically coherent outcomes and detects implausible events, and demonstrate its potential for interactive rollouts, action-conditioned simulation, and zero-shot motion transfer.

Introduction

Video world models are increasingly viewed as simulators for Physical AI, but mainstream approaches that generate future frames through direct pixel-space prediction often yield physically inconsistent results. The underlying dynamics remain implicit in high-dimensional visual features, making it difficult to reason about how the world evolves in a physically principled way. The authors address this by introducing PHIZERO, a physical world model that learns a compact discrete representation called physical language. This representation captures state-transition patterns from unlabeled in-the-wild videos via self-supervised reconstruction, explicitly decoupling dynamics from visual appearance. By adopting a reason-then-render paradigm, PHIZERO first predicts future world evolution in the learned physical-language space before rendering the outcomes into video, enabling more physically coherent generation, interactive control, and zero-shot motion transfer.

Dataset

The authors construct the Physical Language Reasoner dataset in two specialized tiers, reusing video clips originally curated for training the Physical Language Tokenizer.

• General pretraining corpus (~5M clips): Four-second video clips drawn from both real-world and simulation sources. Each clip is accompanied by a VLM-generated caption that deliberately describes only the high-level initiating action or interaction, never the fine-grained outcome (ensuring the text does not leak the future state). The target for each sample is the full physical language token sequence produced offline by the frozen Physical Language Tokenizer. Inputs are the first frame and the caption; the model must predict how the state will evolve.

• Motion-rich SFT corpus (~1M clips): A filtered subset of the general corpus, augmented with curated simulator-generated samples. The authors apply two VLM-based filters—a rich-motion filter and a physical filter—to retain clips that exhibit salient state changes and physically informative interactions. The resulting ~1M four-second clips serve as a high-quality specialization set.

• How the data is used: Training proceeds in two stages. In Stage 1, the model undergoes continued pretraining on the full 5M general corpus; this adapts the pretrained VLM to autoregressive physical language prediction. In Stage 2, the model is fine-tuned on the 1M SFT corpus, focusing learning on motion-rich, physically meaningful transitions. No mixing ratios are applied; each stage uses its entire respective dataset.

• Key processing details:

  • Captions are generated with a prompt that restricts the description to the initial action, preventing the text from revealing the clip's outcome.
  • The frozen tokenizer encodes each 4-second video into a discrete physical language sequence, which becomes the ground-truth target.
  • Only the first frame is provided as visual input to the reasoner; no additional frame cropping or temporal sampling is performed beyond that single initial state.

Method

The authors introduce PHIZERO, a framework for future video prediction that decomposes the task into physical-language reasoning and video rendering. Formally, let $V$V denote the future video, $I^0$I0 the first frame representing the current world state, $c$c the textual action intent, and $z$z the discrete physical language describing the transition from $I^0$I0 to $V$V. The joint modeling factorizes as:

$$p_{\theta, \psi}(V, z \mid I^0, c) = p_{\theta}(z \mid I^0, c) p_{\psi}(V \mid I^0, z)$$pθ,ψ​(V,z∣I0,c)=pθ​(z∣I0,c)pψ​(V∣I0,z)

where the Physical Language Reasoner infers the state evolution in the physical-language space, and the diffusion decoder renders the inferred transition into a future video. This reason-then-render decomposition separates state-transition reasoning from pixel-level synthesis.

Refer to the framework diagram:

The Physical Language Tokenizer compresses state transitions into a discrete sequence. Given a video $V \in \mathbb{R}^{B \times 3 \times T \times H \times W}$V∈RB×3×T×H×W, a spatiotemporal encoder extracts latent features $x \in \mathbb{R}^{B \times C \times t \times h \times w}$x∈RB×C×t×h×w. Instead of extracting a global representation, the authors explicitly model transitions between adjacent latent states to introduce a local temporal inductive bias. For each pair $(x^i, x^{i+1})$(xi,xi+1), a shared Q-Former extracts a transition representation:

$$q_i = \text{QFormer}(Q; x^i, x^{i+1})$$qi​=QFormer(Q;xi,xi+1)

where $Q \in \mathbb{R}^{M \times D_q}$Q∈RM×Dq​ contains $M$M shared learnable transition queries. The resulting features are concatenated in temporal order and discretized using finite scalar quantization (FSQ):

$$z = \text{FSQ}(\text{Proj}_{\text{down}}(q))$$z=FSQ(Projdown​(q))

The quantized representation $z$z is projected to the hidden dimension $d$d of the diffusion transformer, yielding the physical-language context $P_c$Pc​.

To reconstruct the video, the authors employ a pretrained video diffusion model as the decoder. The original architecture is retained, but the text condition is replaced with $P_c$Pc​. The first frame $I^0$I0 is provided as a clean visual condition to supply static appearance information directly. Conditioned on $P_c$Pc​, $I^0$I0, the clean latent $x_0$x0​, and the diffusion timestep $\tau \sim \mathcal{U}(0, 1)$τ∼U(0,1), the decoder optimizes the standard flow-matching objective:

$$\mathcal{L}_{\text{FM}} = \mathbb{E}_{x, \epsilon, \tau} \left[ \| v_{\psi}(x_{\tau}, \tau; I^0, P_c) - (\epsilon - x_0) \|_2^2 \right], \quad x_{\tau} = (1 - \tau)x_0 + \tau \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$LFM​=Ex,ϵ,τ​[∥vψ​(xτ​,τ;I0,Pc​)−(ϵ−x0​)∥22​],xτ​=(1−τ)x0​+τϵ,ϵ∼N(0,I)

To prevent the pretrained decoder from ignoring the physical-language context by relying on its existing denoising prior, a pure-noise warm-up stage initializes all future-frame latents from pure noise. This forces the decoder to rely on $P_c$Pc​ and $I^0$I0 to reconstruct the future video before restoring the standard flow-matching noise schedule.

The training of the Physical Language Tokenizer follows a two-stage scheme coupled with a joint temporal and spatial curriculum.

As shown in the figure below:

For pretraining, the authors filter a 50K-hour real-world video pool to yield 10K hours of unlabeled videos. Videos are processed at 256x448 resolution, with clip duration progressively increased from 1 to 4 seconds to allow the tokenizer to first learn local state changes. For the subsequent SFT stage, stricter filtering based on aesthetic quality, motion magnitude, and observability of state transitions is applied. This is combined with 1K hours of simulated videos to produce 5M four-second clips. The reconstruction resolution is increased to 512x896 for SFT. Finally, the tokenizer is frozen, and only the diffusion decoder is refined to improve reconstruction quality.

The Physical Language Reasoner predicts the physical-language sequence from the first frame and textual intent. Initialized from a pretrained VLM, its vocabulary is extended with a distinct atomic symbol for each FSQ index. Given $I^0$I0 and $c$c, it autoregressively predicts a length-$N$N sequence:

$$p_{\theta}(z \mid I^0, c) = \prod_{j=1}^{N} p_{\theta}(z_j \mid I^0, c, z_{<j})$$pθ​(z∣I0,c)=j=1∏N​pθ​(zj​∣I0,c,z<j​)

Supervision is generated offline by encoding training videos with the frozen tokenizer. The reasoner is optimized with the autoregressive cross-entropy objective:

$$\mathcal{L}_{\text{VLM}} = - \sum_{j=1}^{N} \log p_{\theta}(z_j \mid I^0, c, z_{<j})$$LVLM​=−j=1∑N​logpθ​(zj​∣I0,c,z<j​)

Training proceeds in two stages with progressively specialized data distributions. Stage 1 performs continued pretraining on the 5M-clip general corpus to adapt the VLM to physical-language prediction and establish the correspondence among textual intent, current visual state, and resulting state transition. Stage 2 performs SFT on a curated corpus of 1M motion-rich and physically informative clips, filtered using a VLM-based rich-motion filter and a physical filter, to improve the physical plausibility and precision of inferred state transitions.

Experiment

PHIZERO is evaluated on video generation benchmarks that assess physical outcome fidelity and law adherence, and on video understanding tasks that require discriminating physically valid from invalid scenarios, where it demonstrates strong physical coherence and competitive understanding. Ablation studies confirm that the diffusion decoder, transition-level Q-Former, and pure-noise warm-up are critical for the tokenizer, while prompt enhancement, simulation data, and two-stage training improve the reasoner's transition prediction. Physical language is shown to disentangle state transitions from visual appearance, enabling zero-shot transfer across embodiments and visual domains, and supporting controllable, interactive world models that capture fine-grained dynamics.

On the Physics-IQ Verified benchmark, PHIZERO achieves the highest physical outcome fidelity, outperforming all compared models. The largest models, such as Grok-Video and Cosmos3-Super, score above 34 IQ-Score, while smaller models like Wan2.2-5B reach only 21.2. The results show a trend where larger video generation models tend to produce more physically coherent outcomes. PHIZERO attains the top IQ-Score, surpassing Cosmos3-Super (39.5) and Grok-Video (34.8). Larger models generally improve physical fidelity: Wan2.2-14B (32.2) and Hunyuan-Video (33.4) outperform Wan2.2-5B (21.2).

On PhyGround, larger video generation models consistently attain higher general quality and physics scores, with Veo3.1 and Wan2.2-14B leading the evaluated baselines. The paper reports that PHIZERO surpasses all listed models, securing the top physics score and overall score on this benchmark. Baseline performance improves with model scale: LTX-Video-22B outperforms LTX-Video-19B and Wan2.2-14B surpasses Wan2.2-5B across all metrics. Veo3.1 achieves the highest general quality among baselines, while Wan2.2-14B attains the highest physics score and overall score in the table. PHIZERO exceeds the strongest baselines, ranking first in both physics score and overall score on PhyGround.

On the WorldModelBench general world modeling benchmark, Runway achieves the highest total score and common sense rating, while Wan2.2-5B leads in physics adherence. Luma and Mochi also perform competitively, and earlier models Pandora, OpenSora-Plan, and CogVideoX trail behind in both dimensions. Runway attains the top overall score and common sense rating, outperforming all other evaluated models. Wan2.2-5B records the highest physics adherence, surpassing every other model in physical coherence. Luma and Mochi show strong balanced performance with total scores above 7.6, while Pandora, OpenSora-Plan, and CogVideoX remain below 6.8.

On the IntPhys2 benchmark for intuitive physics understanding, Gemini-2.5 Flash achieves the highest overall accuracy at 55.63%, followed by a group of models tied at 53.75%. Performance varies by difficulty level: Gemini-2.5 Flash excels on easy examples, while V-JEPA leads on hard examples. Gemini-2.5 Flash obtains the best overall score, driven by a strong 64.42% on easy instances. V-JEPA achieves the highest hard-instance accuracy (57.42%), outperforming all other models on that subset.

On the LikePhys physical-plausibility discrimination benchmark, more recent video generation models achieve lower average error rates. Wan2.1-1.3B attains the lowest overall average error at 48.0, driven by a very low rigid error but a relatively high fluid error, while CogVideoX-2B performs best on fluid and optical categories. Earlier models such as AnimateDiff-SDXL show substantially higher errors across all physical domains. Wan2.1-1.3B achieves the lowest rigid error (44.66) and the best average error (48.0), but its fluid error (57.10) is the highest among recent models. CogVideoX-2B records the lowest fluid error (42.00) and optical error (25.15), resulting in a near-best average error of 48.2.

The evaluation spans multiple benchmarks that probe different facets of physical understanding in video generation, including physical fidelity and overall quality (Physics-IQ Verified, PhyGround), world modeling with common sense and physics adherence (WorldModelBench), intuitive physics accuracy (IntPhys2), and physical-plausibility discrimination (LikePhys). Larger models generally produce more physically coherent outcomes, with PHIZERO leading on physics-oriented benchmarks, yet model strengths are domain specific: some smaller models achieve top scores in fluid error, physics adherence, or hard-instance physics understanding, highlighting that scale complements architectural design in capturing physical realism.