One-sentence Summary

Researchers at Tencent Video, LIX, and École Polytechnique propose Factor-Aware Predictive Learning (FAPL), which grounds a frozen multimodal large language model (MLLM) and connector into a Video Diffusion Transformer (DiT) through latent-only training, then uses factor-span replacement and DiT-frozen optimization to achieve factor-aware control and temporal inference for sparse-to-video, factor-aware editing, and multi-shot generation.

Key Contributions

• Factor-Aware Predictive Learning (FAPL) first grounds a Video Diffusion Transformer in frozen MLLM latents through latent-only generator training, strengthening the generator’s use of multimodal representations for cinematic video generation.
• Factor-span replacement on paired videos supervises a target-factor change while preserving non-target content, enabling factor-aware control over character, scene, action, and camera.
• FAPL trains the MLLM and connector through a frozen generator using shared video targets from text-only and text-plus-complete-video observations, producing generation-compatible limited-input representations; experiments report improved performance over baselines on sparse-to-video, factor-aware editing, and multi-shot generation.

Introduction

Cinematic video generation helps creators preview content, performances, and camera choices before production, but it requires models to infer plausible action and camera motion from sparse inputs while independently controlling character, scene, action, and camera. Prior MLLM-conditioned diffusion models often fail to make multimodal features effective: text and VAE reference pathways can bypass MLLM latents, and video-level supervision does not explicitly associate specific factor interventions with intended changes and preservation or train the MLLM to produce temporally informative representations from limited evidence. The authors propose Factor-Aware Predictive Learning (FAPL), a three-stage training strategy that first grounds the video DiT solely in MLLM latents, then uses factor-span replacement between paired videos to supervise selective editing and preservation, and finally trains the MLLM and connector through a fixed generator with mixed sparse and complete observations. This improves temporal inference and factor-aware control in cinematic video generation without requiring fully disentangled representations or added architectural complexity.

Dataset

The authors describe two paired-data components, $D_{\mathrm{obs}}$Dobs​ and $D_{\mathrm{edit}}$Dedit​. The excerpt does not specify underlying video dataset names, total scale, or source filtering rules.

$D_{\mathrm{obs}}$Dobs​: observation paired data

• For each training video $x$x and description $p$p, the authors create two inputs:
  • $u^{t} = p$ut=p: text-only branch.
  • $u^{tv} = (p, x)$utv=(p,x): text-plus-video branch.
• Both inputs are supervised with the same target video $x$x.
• The text-only branch never receives the full target video, so it must infer from the description; the text-plus-video branch receives direct visual evidence.

$D_{\mathrm{edit}}$Dedit​: factor-specific editing data

• Each tuple is $(x^{a}, p^{a}, x^{b}, p^{b}, k)$(xa,pa,xb,pb,k), containing two videos and two descriptions.
• $k$k belongs to $\{${character, scene, action, camera$\}$}; only the selected factor is changed from source to target.
• Captions are rewritten as structured prompts with four ordered fields: character, scene, action, camera.
• $p^{b}$pb is produced by updating only the edited factor field, while the other fields stay the same.
• Field labels and boundaries locate token spans in MLLM hidden states and projected latents, giving each factor an explicit intervention address.
• Span lengths follow the text and can differ across examples; the authors do not assume the contextualized spans are fully disentangled.

Processing and schema

• Paired examples are built around videos, structured descriptions, factor labels, and token span boundaries.
• No cropping, filtering, or dataset statistics are described in this section.

How the data is used

• $D_{\mathrm{obs}}$Dobs​ supports two observation modes with a shared target video.
• $D_{\mathrm{edit}}$Dedit​ supports factor-specific editing and provides token-level factor locations for later intervention.

Method

The authors propose Factor-Aware Predictive Learning (FAPL) to train Multimodal Large Language Model representations for limited-input temporal inference and factor-aware control of character, scene, action, and camera. The framework connects an MLLM to a pretrained Video Diffusion Transformer through a connector. The DiT is trained via flow matching and receives concatenated conditions comprising text-encoder features, MLLM features, and VAE-encoded visual features. The MLLM condition is produced by passing the MLLM hidden states $H_{\theta}(u)$Hθ​(u) through a connector $P_{\eta}$Pη​ to map them into the DiT conditioning space:

$$C^{\mathrm{mllm}} = Z_{\theta, \eta}(u) = P_{\eta}(H_{\theta}(u))$$Cmllm=Zθ,η​(u)=Pη​(Hθ​(u))

For a target video $x$x with VAE latent $z_0$z0​, flow matching constructs $z_t = \alpha_t z_0 + \sigma_t \epsilon$zt​=αt​z0​+σt​ϵ, and the DiT $G_{\phi}$Gϕ​ predicts the target velocity $v^* = \partial z_t / \partial t$v∗=∂zt​/∂t using the flow matching objective:

$$\mathcal{L}_{\mathrm{FM}}(x \mid C; \phi) = \mathbb{E}_{t, \epsilon} \left[ \| G_{\phi}(z_t, t \mid C) - v^* \|_2^2 \right]$$LFM​(x∣C;ϕ)=Et,ϵ​[∥Gϕ​(zt​,t∣C)−v∗∥22​]

The training process proceeds in three stages, utilizing MLLM latents as the sole content conditions to enforce reliance on the MLLM pathway.

Stage I: MLLM-Only Generator Grounding To ensure the DiT reliably decodes and uses MLLM latents rather than relying on native text-encoder or VAE-reference pathways, the authors isolate the MLLM pathway. They freeze the pretrained MLLM and connector, updating only the DiT parameters $\phi$ϕ. For each video and description, they use both text-only and text-plus-complete-video inputs. The mixed-observation objective minimizes the flow matching loss across both modes:

$$\mathcal{L}_{\mathrm{ground}}(\phi) = \mathbb{E}_{(x, p) \sim \mathcal{D}_{\mathrm{obs}}} \left[ \sum_{m \in \{\mathrm{t}, \mathrm{tv}\}} \rho_m \mathcal{L}_{\mathrm{FM}}(x \mid Z_{\theta_0, \eta_0}(u^m); \phi) \right]$$Lground​(ϕ)=E(x,p)∼Dobs​​​m∈{t,tv}∑​ρm​LFM​(x∣Zθ0​,η0​​(um);ϕ)​

This forces the DiT to learn from MLLM latents across both limited and dense observation modes.

Stage II: Factor-Aware Control Improvement To teach the DiT which output changes correspond to specific condition changes, the authors introduce factor-level latent augmentation. They construct paired data where only one factor is modified between two descriptions. Let $Z^a$Za and $Z^b$Zb be the projected latents for the two descriptions. They define a replacement operation $\mathcal{R}_k(A, B)$Rk​(A,B) that swaps the factor-$k$k span in $A$A with the corresponding span from $B$B. The augmented conditions are:

$$Z_k^{a \leftarrow b} = \mathcal{R}_k(Z^a, Z^b), \quad Z_k^{b \leftarrow a} = \mathcal{R}_k(Z^b, Z^a)$$Zka←b​=Rk​(Za,Zb),Zkb←a​=Rk​(Zb,Za)

The DiT is updated using the editing objective:

$$\mathcal{L}_{\mathrm{edit}}(\phi) = \mathbb{E}_{(x^a, p^a, x^b, p^b, k) \sim \mathcal{D}_{\mathrm{edit}}} \left[ \mathcal{L}_{\mathrm{FM}}(x^b \mid Z_k^{a \leftarrow b}; \phi) + \mathcal{L}_{\mathrm{FM}}(x^a \mid Z_k^{b \leftarrow a}; \phi) \right]$$Ledit​(ϕ)=E(xa,pa,xb,pb,k)∼Dedit​​[LFM​(xb∣Zka←b​;ϕ)+LFM​(xa∣Zkb←a​;ϕ)]

This supervises the generator's response to controlled replacements, ensuring target-factor changes while preserving other content.

Stage III: Prediction From Limited Inputs With the DiT fixed at parameters $\phi^*$ϕ∗, the authors jointly fine-tune the MLLM and connector. The goal is to produce generation-compatible representations from limited observations. Using the same mixed-observation setup as Stage I, they minimize:

$$\mathcal{L}_{\mathrm{obs}}(\theta, \eta) = \mathbb{E}_{(x, p) \sim \mathcal{D}_{\mathrm{obs}}} \left[ \sum_{m \in \{\mathrm{t}, \mathrm{tv}\}} \pi_m \mathcal{L}_{\mathrm{FM}}(x \mid Z_{\theta, \eta}(u^m); \phi^*) \right]$$Lobs​(θ,η)=E(x,p)∼Dobs​​​m∈{t,tv}∑​πm​LFM​(x∣Zθ,η​(um);ϕ∗)​

Gradients pass through the frozen DiT to adapt the MLLM pathway, enabling it to reconstruct semantic, motion, and temporal content from limited evidence while maintaining compatibility with complete-video observations.

At inference, the framework supports generation from scripts and references, factor-aware control via span replacement in the structured prompt, and multi-shot generation with shared character and scene references.

Experiment

FAPL is trained on a 13B DiT and evaluated against Omni-Video 2 and UniVideo on held-out CineFactor-10K cases covering sparse-to-video generation, factor-aware editing, and multi-shot generation. The comparisons show that FAPL follows shot-specific action and camera instructions more faithfully, applies single-factor edits without disturbing unrelated attributes, and maintains stronger character, scene, and facial identity consistency across shots. Ablations confirm that each training stage contributes distinct benefits: Stage I improves temporal and camera control from limited inputs, Stage II with factor swapping strengthens factor-aware editing, and Stage III with mixed complete-video observations enhances prompt-only action and camera performance while preserving editing accuracy.

FAPL consistently outperforms Omni-Video 2 and UniVideo across sparse-to-video generation, factor-aware editing, and multi-shot consistency. It achieves better action ordering and camera-trajectory fidelity from text alone, higher accuracy for requested factor changes, and stronger subject, scene, and facial identity consistency across shots. These results indicate improved instruction following and cross-shot consistency. FAPL surpasses the compared models on sparse-to-video generation, with the highest motion order score and the lowest translation and rotation errors. FAPL leads in factor-aware editing and multi-shot consistency, with higher ∩-Acc, subject consistency, scene consistency, and FaceSim-Arc than Omni-Video 2 and UniVideo.

Each FAPL training stage yields clear improvements on UniVideo. Stage I improves motion and camera quality, Stage II mainly strengthens factor-aware editing through factor swapping, and Stage III mixed-observation training further improves action ordering and camera control while keeping editing accuracy stable. Stage I training improves action ordering and reduces translation and rotation errors relative to the pretrained model. Factor swapping in Stage II substantially improves editing accuracy over training without factor swapping while motion and camera metrics do not clearly improve. Stage III mixed-observation training further improves motion order and camera errors, with editing accuracy staying comparable to the Stage II checkpoint.

The experiments compare FAPL with Omni-Video 2 and UniVideo on sparse-to-video generation, factor-aware editing, and multi-shot consistency. FAPL achieves better action ordering and camera trajectory fidelity from text, more accurate factor editing, and stronger subject, scene, and facial identity consistency across shots. Ablation results show that Stage I improves motion and camera quality, Stage II mainly strengthens editing accuracy through factor swapping, and Stage III mixed-observation training further refines motion and camera control while preserving editing performance. Overall, the findings validate FAPL's improved instruction following, cross-shot consistency, and the complementary benefits of each training stage.