HyperAIHyperAI

Command Palette

Search for a command to run...

PhysLoop: 비디오 생성 모델의 자기 개선을 위한 LLM의 물리 인식 활용

초록

비디오 생성 모델은 시각적으로 사실적인 비디오를 생성하지만, 여전히 물리적 역학과 인과적 일관성에 어려움을 겪는다. 생성된 비디오로부터 학습하려면 세밀한 물리적 지도가 필요하지만, 신뢰할 수 있는 품질 판단, 교정 쌍, 통제된 변형을 확보하기는 어렵다. 대규모 언어 모델(LLM)은 물리적 지식을 보유하고 있으나, 비디오 생성 모델이 학습용 비디오를 생성하도록 안정적으로 안내하는 데에는 한계가 있다. 본 논문에서는 비디오 생성 모델의 물리적 기반 자기 개선을 지원하도록 설계된 통합 하네스인 PhysLoop을 제시한다. 이 하네스는 비디오 생성, 선별, 교정의 전 과정에서 LLM을 안내하고 제약함으로써, LLM의 물리적 지식을 물리적으로 일관된 비디오로 증류한다. 이 프레임워크는 통제 가능한 학습 데이터 구축과 추론 시점의 비디오 복구를 지원한다. 실패 사례를 식별하고 교정함으로써, PhysLoop은 인간 개입을 최소화하면서 비디오 생성 모델의 재귀적 자기 개선(RSI)을 향한 중요한 이정표가 된다. 다양한 사실적인 실내 및 실외 장면을 아우르는 100K 규모의 비디오와 생성 도구, 그리고 PhysLoop-Gen과 PhysLoop-Repair라는 두 개의 평가 데이터셋을 공개한다. 각 데이터셋에서 통제 가능한 생성과 국소적 복구를 평가하고, 외부의 Physics-IQ Verified 벤치마크에서 물리적 행동을 평가한다.

One-sentence Summary

Researchers from Peking University, ByteDance Seed, UC Santa Barbara, MIT, et al. propose PhysLoop, a unified harness that guides and constrains LLMs throughout video generation, selection, and correction to distill physical knowledge into physically consistent videos, enabling controllable training-data construction, inference-time video repair, and recursive self-improvement, and they release 100K diverse indoor and outdoor videos along with the PhysLoop-Gen and PhysLoop-Repair evaluation datasets while assessing physical behavior on the external Physics-IQ Verified benchmark.

Key Contributions

  • PhysLoop is a unified harness that guides and constrains LLMs throughout video generation, selection, and correction, distilling their physical knowledge into physically consistent videos and supporting controllable training-data construction and inference-time video repair.
  • PhysLoop provides a shared workflow for failure identification, correction, and verification, connecting diagnosis to simulation-grounded correction. It evaluates both the resolution of the target physical error and the preservation of unaffected content.
  • The release includes 100K videos spanning diverse, realistic indoor and outdoor scenes, generation tools, and two evaluation datasets, PhysLoop-Gen and PhysLoop-Repair. Evaluations are reported for controllable generation, localized repair, and physical behavior on the external Physics-IQ Verified benchmark.

Introduction

Video generation models can produce temporally coherent and viewpoint-consistent scenes, but visual realism does not guarantee physical or causal consistency; objects may appear or disappear, contact may have no plausible response, and motion can conflict with gravity, inertia, or contact forces. Collecting real-world videos that systematically vary physical conditions is difficult to scale while preserving controlled setups and reliable annotations, and fine-tuning data for specific physical failures is similarly hard to obtain. Current self-improvement pipelines also struggle because general-purpose multimodal evaluators cannot reliably judge physical validity, so generating more videos does not by itself produce dependable learning signals. The authors introduce PhysLoop, a unified harness that guides LLMs to apply their existing physical knowledge inside a simulator, using physical assessment and iterative correction to construct training data and repair generated videos. They release 100K videos covering common physical phenomena, the PhysLoop-Gen and PhysLoop-Repair evaluation datasets, and generation tools, and they evaluate controllable generation, localized repair, and physical behavior.

Dataset

Dataset overview

The authors describe a physics-grounded data pipeline and three resources:

  • PhysLoop-100K: a training set with video-text pairs and simulation annotations.
  • PhysLoop-Gen: an evaluation set for generation.
  • PhysLoop-Repair: an evaluation set for physical-error repair.

PhysLoop-100K

  • Scale: 100K accepted videos from 2.5K distinct physical events.
  • Structure: each event has about 40 videos on average, captured from 3 to 5 perspectives, with 2 to 4 key physical parameters set to multiple levels.
  • Covered categories: kinematics, dynamics, rigid-body dynamics, magnetism, fluid mechanics, flexible and elastic materials, fragmentation, and fracture.
  • Source and variation: the system automatically varies physical parameters such as mass and initial velocity, along with assets, scenes, lighting, and camera views. It includes all valid combinations defined for each case.
  • Annotations: refined video-text pairs, generating configurations, original simulation videos, depth maps, instance segmentation maps, and object states. Depth, segmentation, and object-state annotations are not pixel-aligned with refined videos.
  • Split protocol: groups each source event with its camera views and parameter variants.

PhysLoop-Gen

  • Size: 100 generation tasks.
  • Input: each instance has a text request, and some cases also include image references.
  • Coverage: rigid-body, cloth, and fluid phenomena, from simple bouncing or sliding to coupled interactions and multistage mechanical chains.
  • Difficulty definition: based on input properties such as interaction count, event ordering, and observable conditions, not on a specific model’s performance.
  • Use: supports comparison between agentic frameworks such as PhysLoop and direct video generation models.

PhysLoop-Repair

  • Size: 50 repair tasks.
  • Source videos: generated by different video models, including Seedance series, MiniMax H3, and Gemini Omni 1.1 Flash.
  • Input: a source video with a physical error and an optional repair request.
  • Annotations: include error description and temporal interval. The primary evaluation protocol provides only the source video and request, while error and interval annotations remain evaluation-only.
  • Inclusion criteria: requires an observable error, a specified correction, and source content to preserve.
  • Evaluation checks: target-error removal, connection to surrounding events, and preservation of unrelated content.

Processing and usage

  • The authors use an agent to expose task-specific parameters in generated scene configurations, such as release height and initial velocity.
  • Selected parameters are varied while other conditions are held fixed. Scene and camera settings can be edited, and multiple views can be rendered from the same simulation.
  • Task descriptions and visual references specify the appearance to be produced. Videos that pass final verification are paired with their task descriptions as training examples.
  • Each sample retains its generating configuration for inspection, expansion, and regeneration.
  • Downstream training uses the video-text pairs to update the video generation model.
  • Raw simulation renders retain synchronized depth, segmentation, and object-state annotations. Because refinement may alter geometry or object positions, these annotations are kept separate from the refined video-text pairs and are not treated as pixel-aligned supervision.
  • The provided text does not specify a cropping strategy or training mixture ratios.

Method

The authors present PhysLoop, a harness that guides large language models to identify physical errors and generate physically consistent videos through simulators and supporting tools. The system processes a text request, optionally accompanied by image or video references, to produce a generated or repaired video alongside its task specification and execution record.

The workflow is orchestrated by a code agent that leverages a suite of specialized modules. First, a Planner Agent expands the user request into a detailed task specification, recording the target event, construction assumptions, and verification conditions. The Asset Manager Agent then retrieves or generates 3D assets that meet visual and interaction requirements, checking asset scale and collision geometry against the target interaction. Visual meshes and physical representations are handled separately to pair detailed appearance with appropriate collision proxies. The agent writes a scene program combining retrieved assets and procedural geometry, and renders low-resolution preview frames to inspect placement before full simulation.

Next, the Executive Agent drives the physics execution. The harness integrates Unreal Engine as the primary simulator, with Taichi and Genesis providing additional support. Simulator adapters translate scene configurations into the inputs of each simulator and transfer the resulting motion with aligned coordinates and timing. Alongside RGB, the system captures depth, instance segmentation, and object states.

Before appearance refinement, a Verifier Agent compares the simulation states and visual observations with the predefined event requirements. If checks fail, the verifier returns a failure description with supporting evidence, prompting the agent to revise the configuration. Once simulation checks pass, a Refiner Agent adapts the rendered scene to the requested visual appearance using a video generation model, taking the verified simulation video as a motion reference. A Final Verifier Agent then evaluates whether the visible behavior preserves the target event and physical relationships after refinement. If either verification stage fails, the system iterates until all checks pass or the resource budget is exhausted.

For video diagnosis and repair, the agent identifies physical or event-level errors in an existing video and specifies the segment to replace. The repair process replaces only the selected interval [a,b)[a, b)[a,b) of a source video VVV with TTT frames. The corrected segment V^a:b\hat{V}_{a:b}V^a:b​ matches the source resolution and frame rate, and the final output is assembled as:

V′=V0:a∥V^a:b∥Vb:TV' = V_{0:a} \parallel \hat{V}_{a:b} \parallel V_{b:T}V′=V0:a​∥V^a:b​∥Vb:T​

where ∥\parallel∥ denotes concatenation. The verifier evaluates the replacement within the complete video to ensure consistency with preceding and following frames, guiding further revision if necessary.

Experiment

The evaluation covers controllable video generation on PhysLoop-Gen, diagnosis and repair on PhysLoop-Repair, and continuation matching on Physics-IQ Verified, comparing PhysLoop configurations with standalone generators and alternative agent backbones or refiners. Human and automatic metrics assess physical plausibility, causal consistency, visual quality, and task success. PhysLoop improves generation and repair success substantially, with higher human ratings for physical and causal consistency, although its visual realism can trail standalone generators. Case analysis shows that plausible local motion is insufficient because baseline models often fail on event outcomes and scene consistency, while scene construction, simulation, verification, and appearance refinement serve complementary roles.

PhysLoop improves generation and repair success over standalone video generators, with especially large gains in repair. Adding refinement raises task success and human-rated visual realism, while the full configuration leads human ratings for physical plausibility and causal consistency. Physics-IQ rankings diverge from task success rankings, showing that refiner and backbone choices affect different aspects of performance. PhysLoop converts repair from a weak area for standalone generators into a clear strength. Refinement improves generation and repair success and visual realism, while slightly improving physical plausibility and causal consistency. Human ratings favor the full PhysLoop configuration, but Physics-IQ performance depends on the refiner and backbone combination.

The experiments compare PhysLoop with standalone video generators on generation and repair tasks, measuring task success, human ratings of visual realism, physical plausibility, and causal consistency, along with Physics-IQ. PhysLoop substantially improves generation and especially repair performance, turning repair into a strength, while refinement further raises task success and visual realism and yields modest gains in physical plausibility and causal consistency. Human ratings favor the full PhysLoop configuration, but Physics-IQ rankings diverge from task success rankings, indicating that refiner and backbone choices affect different aspects of performance.


AI로 AI 구축

아이디어에서 출시까지 — 무료 AI 코코딩, 즉시 사용 가능한 환경, 최적의 GPU 가격으로 AI 개발을 가속화하세요.

AI 협업 코딩
바로 사용 가능한 GPU
최적의 가격

HyperAI Newsletters

최신 정보 구독하기
한국 시간 매주 월요일 오전 9시 에 이번 주의 최신 업데이트를 메일로 발송합니다
이메일 서비스 제공: MailChimp