One-sentence Summary

Researchers from CUHK, USTC, SCUT, and others propose VideoCoCo, an agentic dual-engine framework where a coding agent generates executable Blender code as a chain of thought to simulate scene physics and produce a deterministic spatiotemporal draft, which a generative engine then refines into a photorealistic video, achieving state-of-the-art physically consistent generation on PhyGenBench and VBench-2.0.

Key Contributions

• VideoCoCo is a dual-engine framework that uses executable Blender code as a process-level chain of thought to generate a deterministic spatiotemporal draft, which a generative editor then refines into a photorealistic video, decoupling physical reasoning from visual fidelity.
• A curated dataset, VideoCoCo-3K, composed of draft–instruction–target triplets, adapts the video editor to simulated drafts.
• On PhyGenBench and VBench-2.0, VideoCoCo raises the OmniWeaving baseline from 0.475 to 0.558 and from 52.18 to 77.88, respectively, achieving the highest average scores and demonstrating that executable code is an effective intermediate representation for physically consistent video generation.

Introduction

Text-to-video generation is a promising pathway toward world modeling for AGI, yet it faces a fundamental mismatch: highly compressed text prompts leave the underlying physical dynamics unspecified, a problem the authors call Causal Opacity. Prior chain-of-thought approaches for video use descriptive or selective intermediates (text plans, keyframes, or learned reward signals) that do not instantiate a complete, executable spatiotemporal process. The authors introduce VideoCoCo, an agentic dual-engine framework that resolves this gap by treating executable Blender code as a process-level chain of thought. A coding agent synthesizes a program that explicitly simulates the scene's evolution into a deterministic draft video; a generative video engine then conditions on that draft to realize the process as a photorealistic video. To train the engine, they construct VideoCoCo-3K, a dataset of draft–instruction–target triplets that aligns simulated drafts with realistic visuals. This design decouples physical reasoning from visual synthesis, yielding substantial gains in physical consistency on benchmarks such as PhyGenBench and VBench-2.0.

Dataset

The authors construct VideoCoCo‑3K, a synthetic dataset of 3,000 triplets designed to learn draft‑conditioned video editing. Each triplet contains three aligned components:

• Draft – a rendered white‑clay video produced by running the executable simulation engine on a sampled prompt.
• Instruction – a natural‑language editing instruction composed by the instruction agent for that draft.
• Target – a photorealistic video obtained by feeding the draft and instruction into a teacher editor (Seedance 2.0). The teacher is chosen because it preserves the draft’s motion while yielding photorealistic frames.

Construction pipeline. Starting from prompts collected from the source distribution, the simulation engine and instruction agent generate a draft–instruction pair. The teacher editor then produces the corresponding photorealistic target, creating a supervised triplet that links a physically grounded motion to its realistic realization. All prompts used in evaluation benchmarks and near‑duplicates are excluded. The dataset retains the original prompt and Blender program as metadata for inspection and regeneration.

Usage in the model. VideoCoCo‑3K is used to adapt the draft‑conditioned editing engine $G_{\theta}$Gθ​ by providing the draft‑instruction‑target supervision that public video‑editing datasets lack. The full set of 3,000 triplets serves as training data; no further training split or mixing ratios are mentioned.

Method

The authors instantiate the Code-as-CoT idea as VideoCoCo, an agentic dual-engine framework that separates process-level reasoning from visual realization. As illustrated in the framework diagram, the two engines play complementary roles to decouple faithful dynamics from photorealistic appearance.

The first component is the Executable Simulation Engine. A text prompt compresses a physical event into a few words, forcing a text-to-video model to reconstruct the entire spatiotemporal process from language alone. Instead of asking the video model to bear this burden implicitly, the authors let a coding agent $A_{\mathrm{code}}$Acode​ synthesize a self-contained Blender Python program $c = A_{\mathrm{code}}(p)$c=Acode​(p) that specifies the scene, objects, physical properties, and temporal evolution. This code is executed in an isolated Blender environment $\mathcal{B}$B to obtain a rendered draft $d = \mathcal{B}(c)$d=B(c). The sandbox provides standardized primitives and enforces deterministic behavior. The resulting draft is a low-fidelity white-clay simulation that is temporally dense, fixing what happens and when, while leaving the visual appearance to the next engine.

The second component is the Generative Video Engine, which translates the instantiated process into a photorealistic video. Because the original prompt and the rendered draft describe the target at different levels of granularity, passing the prompt directly to a video editor would leave the appearance underspecified. To reconcile this, an instruction agent $A_{\mathrm{edit}}$Aedit​ reads both signals and composes an appearance-focused editing instruction $e = A_{\mathrm{edit}}(p, d)$e=Aedit​(p,d). This instruction describes target subjects, materials, lighting, and cinematic style without redefining the motion. The two conditions are then fed jointly into a draft-conditioned video editor $G_{\theta}$Gθ​, which produces the final video $\hat{v} = G_{\theta}(d, e)$v^=Gθ​(d,e). The draft anchors the spatiotemporal structure, while the instruction specifies the photorealistic appearance.

To adapt the editor $G_{\theta}$Gθ​ to consume simulated drafts, the authors construct VideoCoCo-3K, a dataset of draft, instruction, and target triplets. For each prompt $p_i$pi​, the executable simulation engine produces a draft $d_i$di​ and the instruction agent composes an editing instruction $e_i$ei​. A high-fidelity teacher editor $G_{\mathrm{T}}$GT​ generates the photorealistic target $y_i = G_{\mathrm{T}}(d_i, e_i)$yi​=GT​(di​,ei​), yielding the dataset $\mathcal{D}_{\mathrm{VideoCoCo-3K}} = \{(d_i, e_i, y_i)\}_{i=1}^{3000}$DVideoCoCo−3K​={(di​,ei​,yi​)}i=13000​.

The editor is adapted on these triplets using a standard conditional denoising objective. Let $z_0$z0​ denote the latent of the target $y$y and $z_t$zt​ its noised version at diffusion timestep $t$t. The training loss is defined as:

$$\mathcal{L}(\theta) = \mathbb{E}_{(d, e, y), t, \epsilon} \left[ \| \epsilon - \epsilon_{\theta}(z_t, t, d, e) \|_2^2 \right]$$L(θ)=E(d,e,y),t,ϵ​[∥ϵ−ϵθ​(zt​,t,d,e)∥22​]

where $\epsilon \sim \mathcal{N}(0, I)$ϵ∼N(0,I). The authors initialize $G_{\theta}$Gθ​ from a base generator and apply parameter-efficient LoRA adaptation to learn the draft-to-realistic mapping while retaining strong visual priors.

At inference time, VideoCoCo operates end-to-end from a text prompt alone. The coding agent synthesizes the Blender program, the sandbox renders the deterministic draft, the instruction agent composes the editing instruction, and the adapted editor produces the final video. No ground-truth draft or human annotation is required, making the entire pipeline fully automatic, inspectable, and reproducible.

Experiment

The evaluation uses two physics-focused benchmarks, PhyGenBench and VBench-2.0, comparing VideoCoCo against closed- and open-source generators. Experiments show that generating a code-based executable draft and refining it with a video editor improves physical consistency, with the largest gains in categories where purely appearance-driven models struggle most. Ablation confirms that the simulation draft provides the essential physical dynamics, while lightweight LoRA tuning of the editor refines photorealism without overwriting those dynamics, outperforming full fine-tuning.

Adding VideoCoCo to the OmniWeaving video generator substantially raises physical consistency and plausibility across mechanics, optics, thermal, and material dynamics. The largest improvements appear in material and thermal categories where appearance-driven models struggle most, confirming that the executable draft supplies genuine physical dynamics rather than mere surface realism. The combined system achieves top or near-top overall scores on both PhyGenBench and VBench-2.0, outperforming strong open and closed baselines. VideoCoCo lifts OmniWeaving's average consistency score on PhyGenBench to the best overall, ahead of the strongest open baseline, with the largest gains in material and thermal dynamics. On VBench-2.0, VideoCoCo improves the base model's average plausibility by a wide margin, attaining the highest mechanics and thermotics scores and ranking second in material.

Using an executable draft without editor tuning already improves physical plausibility over the base generator, especially in thermal and material categories. Adding editor adaptation yields further complementary gains, with LoRA tuning achieving the best overall average and the top mechanics score while updating far fewer parameters than full fine-tuning. This confirms that the draft supplies physical dynamics and that a lightweight adapter is sufficient to refine appearance without overfitting. The tuning-free variant raises the average from 0.48 to 0.51, driven by marked improvements in thermal (0.43 to 0.48) and material (0.39 to 0.51). LoRA-Tune delivers the highest average (0.56) and the best mechanics score (0.56), outperforming full fine-tuning on both metrics while updating far fewer parameters. Full-Tune boosts optics to 0.61 and matches LoRA on optics, but LoRA achieves a higher material score (0.53 vs. 0.49) and better overall average. The tuning-free gain isolates the draft’s contribution to physical dynamics; subsequent editor tuning, especially with LoRA, adds a complementary improvement in appearance.

Integrating VideoCoCo's executable physics draft into a video generator substantially improves physical consistency, with the largest gains in material and thermal dynamics where appearance-driven models are weakest. Even without tuning, the draft alone lifts plausibility, and adding a lightweight LoRA adapter delivers complementary appearance refinements to achieve top overall scores on PhyGenBench and VBench-2.0, outperforming strong baselines and full fine-tuning. These results confirm that the draft supplies genuine physical dynamics beyond surface realism and that efficient adapters suffice to bridge the remaining appearance gap.