One-sentence Summary

Researchers from THU and HKUST propose LiveEdit, a real-time streaming video editor that distills a bidirectional diffusion model into a unidirectional causal editor via a three-stage pipeline and employs an AR-oriented mask cache to achieve 12.66 FPS inference, enabling stable long-horizon editing for interactive and augmented reality applications.

Key Contributions

• A three-stage distillation pipeline comprised of Foundation Tuning, Teacher Forcing, and Distribution Matching Distillation transfers bidirectional diffusion editing priors into a causal unidirectional streaming editor, compressing the inference loop to 4 steps while preserving long-horizon visual fidelity.
• An AR-oriented mask cache reuses region-related computation across frames, eliminating redundant processing in non-edited areas and reducing latency without visible degradation in those regions.
• A dedicated benchmark for streaming video editing is established; experiments achieve state-of-the-art visual quality among streaming baselines at 12.66 FPS, enabling real-time interactive and augmented reality applications.

Introduction

The authors tackle real-time streaming video editing for augmented reality and livestreaming, where frames must be processed chunk by chunk without future information. Prior video diffusion models rely on offline bidirectional attention, and directly adapting them to causal streaming causes flickering and a forgetting effect due to attention distribution shift. Moreover, standard pipelines redundantly reprocess static background regions across frames, causing prohibitive latency on edge devices. The authors introduce LiveEdit, a framework that distills a bidirectional teacher into a 4-step causal student via a three-stage pipeline (foundation tuning, teacher forcing, and distribution matching distillation). They also propose an AR-oriented mask cache that selectively recomputes only edited regions, dramatically reducing per-frame computation while preserving visual quality, achieving state-of-the-art 12.66 FPS with high fidelity and temporal consistency.

Method

The authors propose a streamingvideo editing framework designed to bridge the gap between offline bidirectional priors and online causal execution. They identify two primary inefficiencies in adapting state-of-the-art video diffusion models to streaming tasks: attention distribution shift and spatial-temporal token redundancy. When bidirectional models are abruptly truncated for causal execution, the loss of future context forces attention weights to flatten and distribute uniformly across historical frames, disrupting pre-trained structural priors.

To resolve these issues, the authors introduce a progressive three-stage distillation pipeline alongside an AR-oriented Mask Cache mechanism. An overview of this comprehensive framework is presented below.

The three-stage distillation pipeline effectively transfers high-fidelity editing capabilities into an ultra-fast, unidirectional streaming editor. In Stage 1, the authors establish a robust multimodal video-to-video editing baseline using a Bidirectional Diffusion Transformer (DiT). The network processes the channel-wise concatenation of the original video latent $z_0 \in \mathbb{R}^{F \times C \times H \times W}$z0​∈RF×C×H×W and the noisy latent $z_t$zt​ at timestep $t$t, supervised via the standard noise-matching objective: $\mathcal{L}_{MSE}^{bid} = \mathbb{E}_{z_0, \epsilon \sim \mathcal{N}(0, I), t, c} \left[ \left\| \epsilon - \epsilon_{\theta}^{bid} (z_t, t, c) \right\|_2^2 \right]$LMSEbid​=Ez0​,ϵ∼N(0,I),t,c​[​ϵ−ϵθbid​(zt​,t,c)​22​] This yields a powerful offline editing prior. In Stage 2, to enable streaming functionality, the architecture transitions to an autoregressive paradigm using a Teacher Forcing mechanism with chunk-wise causal attention. A causal mask $M_{causal}$Mcausal​ restricts temporal tokens from attending to future chunks, optimizing the causal DiT $\epsilon_{\theta}^{causal}$ϵθcausal​ to predict noise while adhering to causal constraints: $\mathcal{L}_{MSE}^{causal} = \mathbb{E}_{z_0, \epsilon, t, c} \left[ \left\| \epsilon - \epsilon_{\theta}^{causal} (z_t, t, c \mid M_{causal}) \right\|_2^2 \right]$LMSEcausal​=Ez0​,ϵ,t,c​[​ϵ−ϵθcausal​(zt​,t,c∣Mcausal​)​22​] This aligns the causal output distribution with the bidirectional representations, preventing structural collapse. In Stage 3, the authors apply Distribution Matching Distillation (DMD) to achieve ultra-low latency. They initialize a 4-step DMD generator $G_{\theta}$Gθ​ using the AR-based model parameters from Stage 2. The distillation process is jointly optimized by $\mathcal{L}_{MSE}$LMSE​ and the DMD gradient $\nabla_{\theta} \mathcal{L}_{DMD}$∇θ​LDMD​, computed between a real score model $\epsilon_{\phi}^{real}$ϵϕreal​ and a trainable fake score model $\epsilon_{\psi}^{fake}$ϵψfake​: $\nabla_{\theta} \mathcal{L}_{DMD} = \mathbb{E}_{z_T, c} \left[ w(t) \left( \epsilon_{\phi}^{real}(z_t, t, c) - \epsilon_{\psi}^{fake}(z_t, t, c) \right) \nabla_{\theta} G_{\theta}(z_T, c) \right]$∇θ​LDMD​=EzT​,c​[w(t)(ϵϕreal​(zt​,t,c)−ϵψfake​(zt​,t,c))∇θ​Gθ​(zT​,c)] where $z_t$zt​ is the intermediate latent simulated from the generated output, and $w(t)$w(t) is a timestep-dependent weighting function.

To address spatial-temporal token redundancy during real-time inference, the authors introduce the AR-oriented Mask Cache mechanism. During streaming inference, the pipeline processes video in sequential chunks. For an incoming chunk $k$k, a binary spatial editing mask $M^k \in \{0, 1\}^{H \times W}$Mk∈{0,1}H×W is derived by computing the $L_2$L2​ distance between the source representation $z_{src}^{k-1}$zsrck−1​ and the edited output latent $z_{edit}^{k-1}$zeditk−1​ of the previous chunk: $M_{u, v}^k = \mathbb{I} \left( \left\| z_{edit, u, v}^{k-1} - z_{src, u, v}^{k-1} \right\|_2 > \tau \right)$Mu,vk​=I(​zedit,u,vk−1​−zsrc,u,vk−1​​2​>τ) where $\mathbb{I}(\cdot)$I(⋅) is the indicator function and $\tau$τ is a dynamically determined threshold. This mask separates the spatial layout into active editing regions and unedited regions. The visualization below demonstrates the high structural stability of this mask generation process.

Instead of executing a complete network forward pass globally, the mechanism implements a spatial routing strategy guided by $M^k$Mk. Tokens within the active mask undergo full calculation through Self-Attention, Cross-Attention, and Feed-Forward Network modules. Conversely, tokens in unedited regions bypass these expensive layers, and their intermediate feature representations are directly retrieved from a Token Cache populated by the preceding chunk. The output token feature $f_{u, v}^k$fu,vk​ is determined by: $f_{u, v}^k = \begin{cases} \mathcal{F}(z_{u, v}^k) & \text{if } M_{u, v}^k = 1 \\ f_{u, v}^{k-1} & \text{if } M_{u, v}^k = 0 \end{cases}$fu,vk​={F(zu,vk​)fu,vk−1​​if Mu,vk​=1if Mu,vk​=0​ This dynamic, inter-chunk spatial decoupling dramatically reduces per-frame computational complexity while strictly guaranteeing absolute visual consistency in unedited background areas.

Experiment

The evaluation pipeline builds on a three-stage distillation process that transforms a bidirectional foundation model into a real-time streaming editor, using causal attention and distribution matching distillation to enable autoregressive 4-step generation. Qualitative comparisons against state-of-the-art offline and streaming baselines reveal that the proposed approach uniquely combines precise local editing with strict preservation of unedited regions, avoiding common failures like color bleeding, structural collapse, or identity corruption. Quantitative assessments confirm that the unidirectional streaming model not only reaches competitive aesthetic quality and motion smoothness but even surpasses bidirectional methods in text alignment. Ablation studies validate the necessity of each training stage and demonstrate that caching self-attention features, guided by a dynamic pruning threshold, optimally retains spatial-temporal redundancy to achieve low-latency performance without sacrificing visual fidelity.

The proposed streaming video editing framework, even without its AR-oriented cache, achieves the highest scores for Text Alignment, Background Consistency, Dynamic Degree, and Imaging Quality among all compared methods. It surpasses the offline bidirectional model InsV2V in text alignment while maintaining competitive Motion Smoothness and Aesthetic Quality against both streaming and offline baselines. The method records the best Text Alignment (0.265) and Dynamic Degree (0.282), outperforming all baselines including the offline InsV2V. Background Consistency remains nearly perfect (0.956), matching the top bidirectional model and exceeding typical streaming performance.

The three-stage distillation pipeline sequentially enables streaming and drastically cuts inference latency while preserving text alignment quality. Starting from a high-cost offline foundation model, the process transitions to chunk-wise autoregressive streaming, then compresses generation to 4 NFEs without classifier-free guidance, achieving real-time performance with minimal accuracy drop. Streaming capability is introduced in Stage 2 by switching from full-sequence processing to autoregressive generation in 3-frame chunks. Stage 3 reduces per-video latency from roughly 200 seconds to under 8 seconds while text alignment scores remain nearly identical across all stages.

Applying the mask cache to self-attention layers maintains editing quality close to the no-cache baseline, with only minor reductions in detail and image quality metrics. In contrast, caching feed-forward network features causes severe degradation, collapsing detail preservation and producing blurred, structurally unstable outputs. This divergence is explained by high temporal redundancy in self-attention tokens versus low similarity in feed-forward representations. Caching self-attention features yields performance on par with no caching, showing only slight drops in detail preservation and image quality. Caching feed-forward network features drastically reduces detail preservation and image quality, leading to severe blurring and structural instability. Self-attention tokens exhibit high temporal redundancy across consecutive steps, making them ideal for efficient caching. Feed-forward representations show notably low temporal similarity, explaining their unsuitability for direct temporal reuse.

The proposed streaming video editing framework, even without its AR-oriented cache, outperforms offline and streaming baselines with the highest text alignment and nearly perfect background consistency. A three-stage distillation pipeline introduces chunk-wise autoregressive streaming and compresses inference latency from roughly 200 seconds to under 8 seconds while text alignment scores remain unchanged. Caching only self-attention features preserves editing quality close to that of a no-cache baseline, whereas caching feed-forward network features causes severe blurring and instability, explained by high temporal redundancy in self-attention tokens and low similarity in feed-forward representations.