One-sentence Summary

Nanyang Technological University and The Chinese University of Hong Kong introduce Apple-π, a benchmark for evaluating video generation models on physical law grounding that combines a curated dataset with a three-stage scientific reasoning protocol (Perception, Formulation, Deduction) treating generated videos as reasoning traces, and a hybrid physics-law-grounded evaluation suite to reveal current models’ failures and guide progress toward law-grounded world simulators.

Key Contributions

• Orchard is a dataset of 400 videos covering ten canonical classical mechanics tasks, organized into single-law and multi-law splits to enable confounder-free diagnosis and generalization testing.
• A three-stage benchmark protocol (Perception, Formulation, and Deduction) employs chain-of-frames prompting on infographic-annotated first frames, treating the generated video as a visible reasoning trace.
• A hybrid evaluation suite combines MLLM-based subjective scoring with physics-law-grounded objective measures, enabling stage-resolved diagnosis of where a model's reasoning breaks down.

Introduction

The authors investigate whether video generation models can reason about the physical world through explicit laws, rather than merely reproducing visually plausible patterns. Prior benchmarks evaluate only the final output, making it impossible to distinguish law-grounded deduction from intuitive imitation. To address this, the authors present Apple-π, a benchmark that decomposes physical reasoning into three auditable stages: Perception of physical quantities, Formulation of the governing law, and Deduction of law-consistent dynamics. Using a curated dataset of classical mechanics scenarios and a suite of hybrid evaluation metrics, it reveals that current video models possess useful physical priors but fall short of reliable law-grounded world simulation, particularly when composition and real-world transfer are required.

Dataset

The authors introduce Orchard, a curated video dataset for physical-reasoning evaluation, with a law-first design: each case is built around explicit classical-mechanics laws, specified physical conditions, and law-predicted motion. The dataset contains 400 cases drawn from three sources.

• Simulated videos (243 cases) – generated with NVIDIA Isaac Sim, providing exact physical parameters and pixel-accurate trajectories. These serve as the objective anchor for the evaluation suite.
• Self-recorded real-world videos (121 cases) – captured under controlled lab conditions that approximate isolation of the governing law while introducing real optical and material cues.
• Internet-sourced real-world videos (36 cases) – curated from physics-education YouTube channels to broaden visual diversity beyond the first two sources.

Across all sources, the object vocabulary is standardized to four primitive solids: sphere, cube, cylinder, and cone. This constrains object identity as a nuisance variable, ensures well-defined centers, masks, contact surfaces, and size parameters, and makes physical states consistently annotatable. The design reduces reliance on object-specific semantic priors, keeping the focus on law-grounded motion reasoning.

Orchard follows a two-level task taxonomy:

• Single-law branch – tasks are organized into three pillars: (i) law of universal gravitation (free fall, projectile motion, inclined-plane motion, circular motion on a gravity-driven track); (ii) conservation of momentum (elastic, perfectly inelastic, and inelastic collisions parameterized by coefficient of restitution); (iii) Newton’s first law (objects at rest, uniform linear motion).
• Multi-law branch – composes tasks across pillars (e.g., inclined plane feeding into projectile motion, or circular motion followed by a collision) to test compositional generalization.

The dataset is used for stage-resolved evaluation, not for training. The authors leverage the single-law cases for controlled diagnosis and the multi-law cases to assess compositional reasoning. No training split, mixing ratios, or cropping strategies are described; the data is prepared as a fixed benchmark with consistent physical-state annotations and metadata that include object geometries, physical parameters (where available), and motion trajectories.

Method

The authors introduce Apple-π, a diagnostic framework designed to assess whether video models can progress from surface-level physical plausibility to law-grounded physical intelligence. Rather than proposing a new model architecture, Apple-π operates as a structured evaluation protocol built around three tightly coupled components: a curated dataset of physically specified scenarios, a multi-stage reasoning benchmark, and a hybrid evaluation suite. Together, these components probe a model’s ability to reason about physics through explicit perception, mathematical formulation, and logical deduction.

The first component is Orchard, a dataset of law-derived, physically specified scenarios. Each scenario is constructed from explicit physical laws, ensuring that every interaction in a video sequence is governed by well-defined mechanics rather than heuristic appearance. This design forces models to rely on underlying physical principles instead of superficial visual cues. The dataset supplies the raw material for the benchmark, providing diverse motion and collision events that demand precise reasoning.

The second component is a benchmark protocol that structures the reasoning task into three sequential stages: Perception, Formulation, and Deduction. In the Perception stage, the model must extract relevant physical quantities from the visual input (e.g., object positions, velocities, masses). The Formulation stage then requires translating these observations into the appropriate governing equations or physical laws. Finally, the Deduction stage tasks the model with applying those laws to predict future states, answer counterfactual queries, or explain observed outcomes. This pipeline moves beyond simple outcome prediction and tests whether a model can produce a coherent, principled reasoning chain that mirrors how a physicist would approach a problem.

The third component is an evaluation suite that combines two complementary scoring mechanisms. Subjective metrics leverage an MLLM-based judge to assess the fluency, coherence, and logical soundness of open-ended reasoning traces. In parallel, objective metrics are grounded directly in the physical laws that define each scenario, enabling a precise, rule-based verification of correctness. By diagnosing where models fail -- whether at perceptual grounding, law formulation, or deductive inference -- the suite provides fine-grained insight into the robustness and limitations of video-based physical reasoning.

Experiment

Apple-π evaluates video generation and unified understanding-generation models on a three-stage physical reasoning benchmark, revealing that strong video synthesis alone does not imply law-grounded physical intelligence; unified models with explicit understanding before generation substantially outperform video models, yet temporal deduction remains the central bottleneck across all architectures. The benchmark uncovers a progressive reasoning funnel where perception and formulation successes do not reliably transfer to law-consistent dynamics, and models struggle to generalize across law transitions and from simulated to real-world visuals. Ablations confirm that the infographic input format is not a limiting factor and that the protocol primarily standardizes output format, while qualitative failures show that copying annotations does not equate to binding physical meaning.

Existing physical intelligence benchmarks either evaluate understanding or generation, but none combine explicit physical-law grounding with multi-stage diagnostic reasoning and chain-of-frames evaluation. While a few benchmarks partially support law-grounded or objective metrics, they all lack the decomposition into perception, formulation, and deduction stages that would make the reasoning process auditable. All listed understanding benchmarks (PhysBench, QuantiPhy, PhyX) and generation benchmarks (VideoPhy-2, PhyWorldBench) do not support multi-stage evaluation or chain-of-frames reasoning. Only QuantiPhy offers an objective physical metric, and it provides only partial support for law-grounded evaluation, limited to kinematics. Video generation benchmarks like VideoPhy-2 and PhyWorldBench assess physical plausibility but do not verify whether models internally ground outputs in physical laws, leaving the reasoning process opaque.

Video generation models reach only moderate performance on Apple-π, with the best video model averaging 0.473, while unified image generation models achieve around 0.70, showing that explicit understanding modules improve physical reasoning. Scores drop sharply from Perception to Formulation and especially Deduction, indicating that models can perceive physical cues but struggle to convert them into law-conditioned temporal dynamics. Multi-law and real-world cases remain substantially harder, exposing weaknesses in compositional state transfer and visual generalization. Unified models like GPT Image 2 and Nano Banana 2 substantially outperform all video models across Perception and Formulation, yet even their Deduction scores remain around 0.40, leaving the temporal reasoning bottleneck unsolved. Video models read annotation text far better than they ground quantities to object regions, and their scores on multi-law scenarios are consistently lower because they fail to carry physical states across law transitions.

Ablating the way physical quantities are presented shows that switching from infographic annotations to a structured text prompt has a negligible effect on performance, with all track-level score changes staying within ±0.05. Using a concise prompt that omits detailed output specifications slightly reduces overall scores, primarily on graphic-output tracks, but it consistently improves formula-text selection, indicating that the full prompt mainly standardizes visual answer formats rather than conveying physics solutions. Specifying physical quantities via a structured text prompt yields nearly neutral paired deltas, with every available track-level change contained within a narrow ±0.05 band, confirming that infographic annotation reading is not a performance bottleneck. A concise prompt lowers average scores, with the decline concentrated on graphic-output tracks (P-G and F-G), while the formula-text track (F-T) improves by +0.051 for both models, showing that the prompt details primarily enforce visual output constraints rather than aiding physics reasoning.

the paper present Apple-π, a benchmark that evaluates physical reasoning across perception, formulation, and deduction stages with explicit law grounding. Unified image generation models with understanding modules outperform video generation models, yet all models suffer a sharp decline from perception to deduction, revealing a bottleneck in temporal reasoning. Multi-law and real-world cases remain especially difficult due to failures in compositional state transfer. Ablation experiments show that infographic annotations are not a performance bottleneck and that elaborate prompts primarily enforce visual output format rather than boost physics reasoning.