HyperAIHyperAI

Command Palette

Search for a command to run...

Videoerzeugung
LLM

PhysLoop: Nutzung des Physikbewusstseins von LLMs für selbstverbessernde Videogenerierungsmodelle

Zusammenfassung

Videogenerierungsmodelle erzeugen visuell realistische Videos, haben jedoch weiterhin Schwierigkeiten mit physikalischer Dynamik und kausaler Konsistenz. Das Lernen aus generierten Videos erfordert feinkörnige physikalische Supervision, doch zuverlässige Qualitätsurteile, Korrekturpaare und kontrollierte Variationen sind schwer zu erhalten. Große Sprachmodelle (LLMs) verfügen über physikalisches Wissen, haben aber Schwierigkeiten, Videogenerierungsmodelle zuverlässig bei der Erzeugung ihrer Trainingsvideos anzuleiten. Wir präsentieren PhysLoop, ein einheitliches Rahmenwerk zur Unterstützung physikalisch fundierter Selbstverbesserung von Videogenerierungsmodellen. Das Rahmenwerk leitet und beschränkt LLMs während der Videogenerierung, -auswahl und -korrektur und destilliert ihr physikalisches Wissen in physikalisch konsistente Videos. Das Framework unterstützt die kontrollierbare Konstruktion von Trainingsdaten sowie die Videoreparatur zur Inferenzzeit. Durch die Identifikation und Korrektur von Fehlern markiert PhysLoop einen wichtigen Meilenstein auf dem Weg zur rekursiven Selbstverbesserung (RSI) von Videogenerierungsmodellen mit begrenztem menschlichem Eingreifen. Wir veröffentlichen 100.000 Videos mit vielfältigen, realistischen Innenund Außenszenen sowie Generierungswerkzeuge und zwei Evaluierungsdatensätze: PhysLoop-Gen und PhysLoop-Repair. Wir evaluieren kontrollierbare Generierung und lokalisierte Reparatur auf diesen Datensätzen und bewerten das physikalische Verhalten anhand des externen Physics-IQ-Verified-Benchmarks.

One-sentence Summary

Researchers from Peking University, ByteDance Seed, UC Santa Barbara, MIT, et al. propose PhysLoop, a unified harness that guides and constrains LLMs throughout video generation, selection, and correction to distill physical knowledge into physically consistent videos, enabling controllable training-data construction, inference-time video repair, and recursive self-improvement, and they release 100K diverse indoor and outdoor videos along with the PhysLoop-Gen and PhysLoop-Repair evaluation datasets while assessing physical behavior on the external Physics-IQ Verified benchmark.

Key Contributions

  • PhysLoop is a unified harness that guides and constrains LLMs throughout video generation, selection, and correction, distilling their physical knowledge into physically consistent videos and supporting controllable training-data construction and inference-time video repair.
  • PhysLoop provides a shared workflow for failure identification, correction, and verification, connecting diagnosis to simulation-grounded correction. It evaluates both the resolution of the target physical error and the preservation of unaffected content.
  • The release includes 100K videos spanning diverse, realistic indoor and outdoor scenes, generation tools, and two evaluation datasets, PhysLoop-Gen and PhysLoop-Repair. Evaluations are reported for controllable generation, localized repair, and physical behavior on the external Physics-IQ Verified benchmark.

Introduction

Video generation models can produce temporally coherent and viewpoint-consistent scenes, but visual realism does not guarantee physical or causal consistency; objects may appear or disappear, contact may have no plausible response, and motion can conflict with gravity, inertia, or contact forces. Collecting real-world videos that systematically vary physical conditions is difficult to scale while preserving controlled setups and reliable annotations, and fine-tuning data for specific physical failures is similarly hard to obtain. Current self-improvement pipelines also struggle because general-purpose multimodal evaluators cannot reliably judge physical validity, so generating more videos does not by itself produce dependable learning signals. The authors introduce PhysLoop, a unified harness that guides LLMs to apply their existing physical knowledge inside a simulator, using physical assessment and iterative correction to construct training data and repair generated videos. They release 100K videos covering common physical phenomena, the PhysLoop-Gen and PhysLoop-Repair evaluation datasets, and generation tools, and they evaluate controllable generation, localized repair, and physical behavior.

Dataset

Dataset overview

The authors describe a physics-grounded data pipeline and three resources:

  • PhysLoop-100K: a training set with video-text pairs and simulation annotations.
  • PhysLoop-Gen: an evaluation set for generation.
  • PhysLoop-Repair: an evaluation set for physical-error repair.

PhysLoop-100K

  • Scale: 100K accepted videos from 2.5K distinct physical events.
  • Structure: each event has about 40 videos on average, captured from 3 to 5 perspectives, with 2 to 4 key physical parameters set to multiple levels.
  • Covered categories: kinematics, dynamics, rigid-body dynamics, magnetism, fluid mechanics, flexible and elastic materials, fragmentation, and fracture.
  • Source and variation: the system automatically varies physical parameters such as mass and initial velocity, along with assets, scenes, lighting, and camera views. It includes all valid combinations defined for each case.
  • Annotations: refined video-text pairs, generating configurations, original simulation videos, depth maps, instance segmentation maps, and object states. Depth, segmentation, and object-state annotations are not pixel-aligned with refined videos.
  • Split protocol: groups each source event with its camera views and parameter variants.

PhysLoop-Gen

  • Size: 100 generation tasks.
  • Input: each instance has a text request, and some cases also include image references.
  • Coverage: rigid-body, cloth, and fluid phenomena, from simple bouncing or sliding to coupled interactions and multistage mechanical chains.
  • Difficulty definition: based on input properties such as interaction count, event ordering, and observable conditions, not on a specific model’s performance.
  • Use: supports comparison between agentic frameworks such as PhysLoop and direct video generation models.

PhysLoop-Repair

  • Size: 50 repair tasks.
  • Source videos: generated by different video models, including Seedance series, MiniMax H3, and Gemini Omni 1.1 Flash.
  • Input: a source video with a physical error and an optional repair request.
  • Annotations: include error description and temporal interval. The primary evaluation protocol provides only the source video and request, while error and interval annotations remain evaluation-only.
  • Inclusion criteria: requires an observable error, a specified correction, and source content to preserve.
  • Evaluation checks: target-error removal, connection to surrounding events, and preservation of unrelated content.

Processing and usage

  • The authors use an agent to expose task-specific parameters in generated scene configurations, such as release height and initial velocity.
  • Selected parameters are varied while other conditions are held fixed. Scene and camera settings can be edited, and multiple views can be rendered from the same simulation.
  • Task descriptions and visual references specify the appearance to be produced. Videos that pass final verification are paired with their task descriptions as training examples.
  • Each sample retains its generating configuration for inspection, expansion, and regeneration.
  • Downstream training uses the video-text pairs to update the video generation model.
  • Raw simulation renders retain synchronized depth, segmentation, and object-state annotations. Because refinement may alter geometry or object positions, these annotations are kept separate from the refined video-text pairs and are not treated as pixel-aligned supervision.
  • The provided text does not specify a cropping strategy or training mixture ratios.

Method

The authors present PhysLoop, a harness that guides large language models to identify physical errors and generate physically consistent videos through simulators and supporting tools. The system processes a text request, optionally accompanied by image or video references, to produce a generated or repaired video alongside its task specification and execution record.

The workflow is orchestrated by a code agent that leverages a suite of specialized modules. First, a Planner Agent expands the user request into a detailed task specification, recording the target event, construction assumptions, and verification conditions. The Asset Manager Agent then retrieves or generates 3D assets that meet visual and interaction requirements, checking asset scale and collision geometry against the target interaction. Visual meshes and physical representations are handled separately to pair detailed appearance with appropriate collision proxies. The agent writes a scene program combining retrieved assets and procedural geometry, and renders low-resolution preview frames to inspect placement before full simulation.

Next, the Executive Agent drives the physics execution. The harness integrates Unreal Engine as the primary simulator, with Taichi and Genesis providing additional support. Simulator adapters translate scene configurations into the inputs of each simulator and transfer the resulting motion with aligned coordinates and timing. Alongside RGB, the system captures depth, instance segmentation, and object states.

Before appearance refinement, a Verifier Agent compares the simulation states and visual observations with the predefined event requirements. If checks fail, the verifier returns a failure description with supporting evidence, prompting the agent to revise the configuration. Once simulation checks pass, a Refiner Agent adapts the rendered scene to the requested visual appearance using a video generation model, taking the verified simulation video as a motion reference. A Final Verifier Agent then evaluates whether the visible behavior preserves the target event and physical relationships after refinement. If either verification stage fails, the system iterates until all checks pass or the resource budget is exhausted.

For video diagnosis and repair, the agent identifies physical or event-level errors in an existing video and specifies the segment to replace. The repair process replaces only the selected interval [a,b)[a, b)[a,b) of a source video VVV with TTT frames. The corrected segment V^a:b\hat{V}_{a:b}V^a:b​ matches the source resolution and frame rate, and the final output is assembled as:

V′=V0:a∥V^a:b∥Vb:TV' = V_{0:a} \parallel \hat{V}_{a:b} \parallel V_{b:T}V′=V0:a​∥V^a:b​∥Vb:T​

where ∥\parallel∥ denotes concatenation. The verifier evaluates the replacement within the complete video to ensure consistency with preceding and following frames, guiding further revision if necessary.

Experiment

The evaluation covers controllable video generation on PhysLoop-Gen, diagnosis and repair on PhysLoop-Repair, and continuation matching on Physics-IQ Verified, comparing PhysLoop configurations with standalone generators and alternative agent backbones or refiners. Human and automatic metrics assess physical plausibility, causal consistency, visual quality, and task success. PhysLoop improves generation and repair success substantially, with higher human ratings for physical and causal consistency, although its visual realism can trail standalone generators. Case analysis shows that plausible local motion is insufficient because baseline models often fail on event outcomes and scene consistency, while scene construction, simulation, verification, and appearance refinement serve complementary roles.

PhysLoop improves generation and repair success over standalone video generators, with especially large gains in repair. Adding refinement raises task success and human-rated visual realism, while the full configuration leads human ratings for physical plausibility and causal consistency. Physics-IQ rankings diverge from task success rankings, showing that refiner and backbone choices affect different aspects of performance. PhysLoop converts repair from a weak area for standalone generators into a clear strength. Refinement improves generation and repair success and visual realism, while slightly improving physical plausibility and causal consistency. Human ratings favor the full PhysLoop configuration, but Physics-IQ performance depends on the refiner and backbone combination.

The experiments compare PhysLoop with standalone video generators on generation and repair tasks, measuring task success, human ratings of visual realism, physical plausibility, and causal consistency, along with Physics-IQ. PhysLoop substantially improves generation and especially repair performance, turning repair into a strength, while refinement further raises task success and visual realism and yields modest gains in physical plausibility and causal consistency. Human ratings favor the full PhysLoop configuration, but Physics-IQ rankings diverge from task success rankings, indicating that refiner and backbone choices affect different aspects of performance.


KI mit KI entwickeln

Von der Idee bis zum Launch – beschleunigen Sie Ihre KI-Entwicklung mit kostenlosem KI-Co-Coding, sofort einsatzbereiter Umgebung und bestem GPU-Preis.

KI-gestütztes kollaboratives Programmieren
Sofort einsatzbereite GPUs
Die besten Preise

HyperAI Newsletters

Abonnieren Sie unsere neuesten Updates
Wir werden die neuesten Updates der Woche in Ihren Posteingang liefern um neun Uhr jeden Montagmorgen
Unterstützt von MailChimp