One-sentence Summary

Researchers from the California Institute of Technology and Netflix propose Vera, a layered video diffusion model that preserves content by generating an edit layer and alpha matte for compositing onto the source video, employing a Mixture-of-Transformers architecture with joint self-attention across separate DiTs, trained on a 486K-frame layered dataset, and outperforms leading open-source video editing models in content preservation while remaining competitive in edit quality.

Key Contributions

• Vera introduces a layered diffusion framework that generates an edit layer and an alpha matte for compositing with the source video, separating creative editing from content preservation.
• The text-to-video DiT is extended to a Mixture-of-Transformers architecture, where separate DiTs for each layer interact through joint self-attention to encourage coherent composition.
• A high-quality layered dataset with accurate alpha mattes, diverse scenes, and visual effects is constructed using 486K frames of training data; Vera outperforms leading open-source video editing models in content preservation while remaining competitive in edit quality.

Introduction

Recent breakthroughs in text-to-video diffusion models have driven rapid progress in controllable video editing, but a persistent barrier is content-preserving editing: even small unintended changes to background regions can ruin an edit and undermine user trust. Existing methods such as regional constraints and mask conditioning still operate inside an end-to-end diffusion paradigm, which regenerates the entire video and can introduce subtle alterations to areas that should be strictly preserved, while also failing to provide the layered assets needed for iterative production workflows. A layered approach that produces the edit as a separate RGBA layer would preserve the original video by construction, but applying this to video editing introduces new challenges: the edit layer and alpha matte must be precisely aligned, must match the source video’s camera motion, lighting, and spatial layout, and must handle complex cross-layer interactions like shadows and reflections, which prior layered generation work has not addressed. The authors introduce Vera, a layered diffusion framework that jointly generates an edit layer, an alpha matte, and a composite video, using a Mixture-of-Transformers architecture with cross-layer joint self-attention and a carefully curated layered dataset to achieve faithful content preservation and natural compositing.

Dataset

The authors construct a layered video editing dataset from open-source videos, combining manual annotation and automated generation tools. The dataset is designed to train a model for tasks like background change and object addition.

Overall composition and scale

• 486K frames in total, all at 832 × 480 resolution.
• About 6K samples, each a tuple of four videos (input, edit layer, alpha matte, composite) of 81 frames.
• Approximately 60–70% of filtered source videos produced usable samples through the pipeline.

Subset breakdown

1. Synthetic composites (Object Addition and Background Change)

   • Source: VideoMatte240K, which supplies high-quality alpha mattes with fine details (e.g., hair).
   • Processing: Synthetic backgrounds are generated via an inpainting model and composited with the foreground mattes.
   • Characteristics: Provides accurate alpha supervision; limited to human subjects filmed with static, centered cameras.

2. Realistic single-object videos (Object Addition and Background Change)

   • Source: Real-world videos from Pexels and Mixkit.
   • Processing: A multi-stage pipeline that chains segmentation, video matting, video inpainting, generation, and human annotation/filtering.
   • Characteristics: Introduces diverse scenes and dynamic camera motion; typically contains one prominent subject with few visual effects.

3. Realistic multi-object videos with effects (Object Addition only)

   • Source: Built by extending the single-object pipeline with an additional omnimatte optimization step and alpha matte curation.
   • Characteristics: Features multiple objects together with associated effects (shadows, reflections) in complex scenes with rich dynamics. Provides data solely for the object addition task.

How the data is used

• The dataset serves as the training data for the model proposed in the paper.
• The synthetic subset offers precise alpha supervision for both tasks.
• The realistic single-object subset supplies training data for both background change and object addition.
• The multi-object subset is used exclusively for object addition training.
• No explicit training split or mixture ratios are detailed; the authors rely on these complementary subsets to cover increasing complexity levels.

Method

The authors address the content-preserving video editing problem by formulating the source video $V_{\text{src}}$Vsrc​ as a composition of preserved content $V_{\text{preserved}}$Vpreserved​ and non-preserved content, governed by an alpha matte $A_{\text{edit}}$Aedit​. Given the source video and a conditioning signal $C$C, the goal is to generate an edit layer $V_{\text{edit}}$Vedit​ and its corresponding alpha matte. The final output video is composited as:

In regions with minimal semi-transparency, the preserved content is well approximated by the source video ($V_{\text{preserved}} \approx V_{\text{src}}$Vpreserved​≈Vsrc​). This layered formulation explicitly separates creative edit generation from content preservation.

To model this, the authors leverage a diffusion model to learn the joint conditional distribution $p(V_{\text{edit}}, A_{\text{edit}}, V_{\text{composite}} \mid V_{\text{src}}, C)$p(Vedit​,Aedit​,Vcomposite​∣Vsrc​,C). They choose to generate the composite video rather than the preserved content directly, as the composite shares the distribution of natural videos, aligning better with pretrained video generation models. The preserved content can subsequently be recovered. The model is implemented as a latent diffusion model where all videos are encoded into a latent space using the Wan2.1 VAE. The alpha matte sequence is treated as a video with identical RGB channels. Using flow matching parameterization, the neural network takes the source latent $Z_{\text{src}}$Zsrc​ and conditioning $C$C as inputs to jointly predict the velocity fields for all layers: $u_{\theta} = [u_{\theta;\text{edit}}, u_{\theta;\text{alpha}}, u_{\theta;\text{composite}}]$uθ​=[uθ;edit​,uθ;alpha​,uθ;composite​]. The training objective is defined as:

where $Z_0 \sim \mathcal{N}(0, I)$Z0​∼N(0,I), $Z_1 = [Z_{\text{edit}}, Z_{\text{alpha}}, Z_{\text{composite}}]$Z1​=[Zedit​,Zalpha​,Zcomposite​], $t \in [0, 1]$t∈[0,1], and $Z_t = t Z_1 + (1 - t) Z_0$Zt​=tZ1​+(1−t)Z0​.

A key design challenge is that the edit layer, alpha matte, and composite video possess substantially different distributions. A single shared transformer would struggle to reconcile these differences efficiently. To overcome this, the authors adopt a Mixture-of-Transformers framework, utilizing three separate DiTs that interact through joint self-attention.

As shown in the figure above, unlike standard fine-tuning or adapter-based approaches, the proposed architecture assigns a dedicated DiT block to each output layer. While each DiT maintains its own QKV projections and feed-forward network weights, tokens from all three branches are concatenated into a single sequence for the self-attention operation. This enables cross-layer interaction while allowing each branch to specialize in its specific distribution. All three DiTs are initialized from the same pretrained text-to-video model. To incorporate conditioning inputs, two additional patch embedding layers are introduced for the input video and an optional mask video. The source video tokens are added to the composite tokens, and mask tokens are added to the noisy alpha tokens. All layer tokens share the same positional encoding, with zero-initialized learnable embeddings added to the alpha and composite tokens to distinguish between layers.

Since no public dataset provides suitable layered video editing data, the authors construct a custom layered dataset from open-source videos.

The dataset comprises roughly 6,000 samples, each containing an input video along with the three output layers at 832x480 resolution and 81 frames. The data is categorized into complementary subsets of increasing complexity, including synthetic composites, realistic single-object videos, and realistic multi-object videos with visual effects like shadows and reflections.

The data construction pipelines, illustrated above, involve multi-stage processes including segmentation, video matting, inpainting, and human annotation. For object addition, the pipeline extracts individual objects and their effects using omnimatte optimization. For background change, it utilizes inpainting and recomposition techniques to generate diverse synthetic backgrounds or remove existing objects to create clean edit layers. This curated data enables the model to learn the complex interactions required for high-quality layered video editing.

Experiment

The evaluation of Vera on background change and object addition demonstrates that its layered editing design substantially improves content preservation over existing video-to-video methods, while remaining competitive in composition quality and instruction compliance. Ablations attribute this to the layered paradigm, composite branch, Mixture-of-Transformers architecture, and the use of diverse training data. Qualitatively, Vera avoids common failure modes such as distorting regions that should remain unchanged, and a human preference study confirms strong user preference for its outputs.

Vera substantially outperforms all existing methods in content preservation on both background change and object addition tasks, with PSNR gains exceeding 3 dB and large reductions in structural and perceptual errors. Traditional video quality metrics OC and TF are nearly saturated and provide little distinction between methods. VLM-based composition metrics reveal a more nuanced ranking, with VACE leading on background change and Vera ranking second. Vera-1.3B surpasses the strongest baseline by 6.3 dB PSNR on object addition and reduces LPIPS and structural error by more than half. Vera-14B further extends the PSNR margin to 7.1 dB on object addition. On background change, VACE achieves the highest VLM-judged composition scores (CS, CT, IS), while Vera ranks second. OC and TF metrics are near-identical across most models, failing to capture differences in video quality.

Layered editing with Vera-no-comp dramatically improves content preservation over standard video-to-video models, raising PSNR by over 4 dB and SSIM by 0.08, while reducing LPIPS by 0.13 on object addition. Removing the composite branch causes composition and instruction-following metrics to fall below the 1.3B V2V baselines; the full Vera model retains the composite branch and keeps these scores competitive. Vera-no-comp achieves 25.4 dB PSNR and 0.942 SSIM, substantially outperforming the 1.3B V2V baselines (21.1 dB, 0.861) on content preservation. Without the composite branch, composition score (CS) drops to 2.85, below the 1.3B V2V models (3.29–3.54), and instruction score (IS) falls to 3.54 versus 3.90–4.09. The full Vera model, with the composite branch, maintains competitive CS (3.46) and IS (3.97) compared to the best V2V result (CS 3.54, IS 4.09).

Ablation on architecture choices shows that a mixture-of-transformers (MoT) design with separate diffusion transformers per layer substantially outperforms a single dense DiT for object addition, improving content preservation and instruction compliance. Channel concatenation for input video conditioning yields better results than sequence concatenation on this task, with zero-initialized patch embeddings delivering the highest fidelity and copy-initialized embeddings achieving the best instruction following. The MoT design also enhances alignment between the edit layer and alpha matte, leading to better compositing. MoT architecture significantly boosts content preservation (PSNR, SSIM, LPIPS) and instruction compliance (CS, CT, IS) over a dense DiT. Channel concatenation with zero-initialized input embedding gives the best content fidelity, while copy-initialized embedding excels in instruction compliance metrics. Sequence concatenation for input conditioning is less effective for object addition, with lower VLM-based scores and higher perceptual loss, and incurs higher computational cost.

Starting from synthetic-only training data, adding realistic single-object videos substantially improves background change preservation and composition quality, while further including multi-object videos yields a large gain in object addition performance at the cost of a slight regression on background change metrics. The results show that diverse real-world data is crucial for generalization across tasks, but balancing the data mixture remains important to avoid trade-offs. Adding realistic single-object data boosts background change PSNR from 33.8 to 35.6 dB and lifts composition scores and instruction compliance on both tasks. Multi-object data dramatically improves object addition content preservation (PSNR from 19.1 to 24.8) but causes a small drop in background change preservation, indicating a need for task-balanced data mixing.

Assigning different learning rates to the three DiT branches reveals that the alpha and composite branches benefit from faster adaptation than the edit branch. Increasing the alpha branch learning rate improves content preservation on background change and instruction compliance on object addition, while further raising the composite branch rate yields large gains in object addition content preservation. Reducing the edit branch learning rate maintains comparable overall performance, indicating that the model can tolerate slower updates for the edit layer. Increasing the alpha branch learning rate to 10× base rate improves background change PSNR by 2.5 dB and raises object addition instruction score (IS) from 3.00 to 4.00. Further raising the composite branch learning rate to 10× base rate adds a 3.2 dB PSNR gain on object addition over Config-A, along with improved SSIM and LPIPS. Reducing the edit branch learning rate to 0.1× base rate (Vera config) achieves overall performance comparable to the best Config-B, with no clear winner between them. Uniform learning rates across all branches are suboptimal; the alpha and composite branches require faster adaptation than the edit branch.

the paper evaluate Vera on background change and object addition tasks, comparing its layered video editing approach with existing methods and examining architecture, data, and training design choices. The model significantly improves content preservation over baselines, with the composite branch critical for maintaining composition quality alongside the edit and alpha layers. Ablations reveal that a mixture-of-transformers architecture, channel conditioning, and a balanced mix of synthetic and diverse real-world data enhance performance, while differential learning rates across the edit, alpha, and composite branches are essential for optimal results.