One-sentence Summary

JoyAI-Video-Edit, a 16B-parameter autoregressive diffusion framework proposed by researchers from Joy Future Academy, JD, combines chunkwise autoregressive adaptation, Source-Anchored Distribution Matching Distillation (SA-DMD), and Long-Horizon Autoregressive Distillation to achieve real-time, open-ended video editing at 720p and approximately 30 FPS on a single Nvidia B200 GPU, substantially outperforming existing streaming editors while remaining competitive with offline systems.

Key Contributions

• JoyAI Video-Edit is a 16B-parameter autoregressive diffusion framework that enables real-time, open-ended streaming video editing without requiring future frames or a predefined video length.
• The framework combines chunkwise autoregressive adaptation, Source-Anchored Distribution Matching Distillation (SA-DMD), and Long-Horizon Autoregressive Distillation to reduce train-inference mismatch, preserve source fidelity during two-step generation, and mitigate accumulated temporal drift.
• Extensive automatic and human evaluations demonstrate substantial improvements over existing streaming editors, competitive performance with strong offline baselines on both short and long videos, and end-to-end 720p editing at approximately 30 FPS on a single Nvidia B200 GPU.

Introduction

The authors address streaming video editing for applications like live broadcasting and interactive entertainment, where edited frames must be output causally and in real time as source video arrives. Existing high-quality editors are offline, rely on bidirectional temporal processing that prevents causal output and grows memory with length, and when applied naively to clips produce discontinuities. Simple causal adaptation of offline models introduces a train-inference mismatch that causes error accumulation and appearance drift over long streams. The authors propose JoyAI-Video-Edit, a 16B-parameter autoregressive diffusion framework that converts a strong bidirectional editor into a chunk-wise causal model with bounded-state sliding windows and resampling forcing, aligning training and inference history distributions. To enable real-time two-step generation, they introduce Source-Anchored Distribution Matching Distillation (SA-DMD), which distills iterative denoising into a generator that counteracts drift by anchoring to the temporally aligned source chunk, absorbing guidance into a single branch, and they further apply Long-Horizon Autoregressive Distillation to explicitly optimize against accumulated errors. The complete system processes 720p video editing at approximately 30 FPS on a single NVIDIA B200 GPU.

Dataset

The authors construct two complementary datasets: a text-to-video generation corpus and a paired video-editing dataset. Both are tailored for training a unified video model.

• Text-to-Video Data

  • Composition and Sources: Images and videos drawn from diverse domains (people, lifestyle, entertainment, nature, objects, urban scenes).
  • Image filtering: Sharpness and quality checks, black-border detection, and deduplication via perceptual hashes and embeddings.
  • Video filtering: Visual quality, aesthetic score, motion magnitude, camera stability, and temporal validity; blurred, rotated, presentation-style, and corrupted videos are discarded.
  • Concept balancing: Filtered data is clustered; dominant categories are downsampled, while long-tail concepts are preserved.
  • Motion bias correction: Additional motion-based filtering prevents aesthetic models from over-selecting static videos, retaining clips with meaningful but stable dynamics.
  • High-quality fine-tuning subset: Automatic filtering is followed by manual inspection to exclude videos containing prominent text, broken structures, implausible motion, or physical inconsistencies.

• Video Editing Data

  • Source: Transferred from JoyAI-Image’s image editing supervision (image-to-image and reference-to-image pairs).
  • Construction Pipeline 1 – Keyframe propagation: A representative keyframe is selected from a source video, edited per instruction, then processed by an image-and-video-to-video model that propagates the edit while preserving original motion and unchanged regions.
  • Construction Pipeline 2 – Latent-shared generation: An original image and its edited counterpart are used for latent-shared image-to-video generation; early denoising latents are shared to maintain consistent motion and composition, while later stages are conditioned on the different images to introduce the edit.
  • Filtering: Generated pairs are filtered by visual quality, editing correctness, content preservation, and temporal consistency.
  • Instruction refinement: A multimodal large language model (MLLM) compares source and edited videos and refines the editing instructions.
  • Edit types: Covers local edits (subjects, backgrounds, specific regions), global edits (style, tone, motion transformations), and subject additions, replacements, or removals.

These datasets are used together to train the model: the text-to-video corpus provides foundational generation skills, while the video editing pairs enable instruction-driven video manipulation.

Method

The authors propose JoyAI-Video-Edit, a unified autoregressive diffusion model designed for video editing. As shown in the framework diagram, the architecture comprises a multimodal large language model (MLLM), a causal video variational autoencoder (VAE), and a multimodal diffusion transformer (MM-DiT).

The MLLM processes the first frame of the source video and the corresponding editing instruction to extract condition tokens, encoding both source-aware visual information and editing intent. The causal video VAE encodes the source video into latent sequences and maps optional reference images into the same latent space, utilizing a spatiotemporal compression ratio of $8 \times 24 \times 24$8×24×24. The MM-DiT jointly models the condition tokens and latent visual tokens to generate edited-video latents, which are subsequently decoded by the VAE into the final output video.

Training begins with progressive text-to-image pretraining to establish text-visual alignment, followed by text-to-video training where temporal modeling is introduced. The curriculum progresses from low-resolution videos to higher resolutions and frame rates, concluding with supervised fine-tuning and continual training. To enable bidirectional video-to-video editing, the authors introduce image-to-image supervision to learn instruction-conditioned transformation. Since high-quality paired video editing data is scarce, they construct it using two complementary pipelines, as illustrated in the data pipeline diagram.

First, a representative keyframe is edited according to the instruction, and the change is propagated across the video using an image-and-video-to-video model. Second, paired videos are generated from original and edited images using latent-shared image-to-video generation, where branches share early denoising latents to maintain consistent motion and composition. The resulting pairs are filtered and recaptioned. The bidirectional editor is trained using a flow-matching objective. Given a source video, editing instruction, optional reference image, and edited target, the causal video VAE produces the corresponding latents. For a sampled noise level $\sigma$σ and Gaussian noise $\epsilon$ϵ, the noisy target and flow target are constructed as $z _ { \sigma } = ( 1 - \sigma ) z _ { 0 } + \sigma \epsilon$zσ​=(1−σ)z0​+σϵ and $v ^ { \star } = \epsilon - z _ { 0 }$v⋆=ϵ−z0​. The model is optimized to minimize the difference between the predicted and target velocities.

The bidirectional model is then adapted into a causal paradigm for streaming generation. The video is split into chunks, and a chunk-wise attention mechanism is adopted: bidirectional attention within each chunk and causal attention across chunks. To bound computation, cross-chunk attention is restricted to a sliding window of recent history chunks and a global sink (the first chunk). During training, Resampling Forcing is employed, replacing clean history with on-policy estimates regenerated via single-step denoising, thereby alleviating the train-test distribution mismatch.

To achieve real-time throughput, the model is distilled into a few-step generator using Distribution Matching Distillation. To prevent source drift and hallucinations during extended rollouts, the authors introduce Source-Anchored Distillation. This method anchors the teacher to the temporally aligned source chunk by applying classifier-free guidance along text condition and source fidelity axes:

where the source-free prediction omits the aligned source latent. This formulation creates a source-sharpened posterior, distilling source-fidelity control directly into the generator. To expose distillation to compounded errors in long-horizon inference without memory overflow, segmented optimization is performed over extended rollouts. The sequence is divided into shorter clips, gradients are accumulated, and the graph is cleared between clips. For rollouts exceeding source video length, a dynamic mirror looping strategy extends conditioning to preserve temporal continuity.

At deployment, the incoming video stream is processed in consecutive eight-frame chunks. The runtime breakdown for this pipeline is shown in the figure below.

The pipeline employs FP8 quantization, operator fusion, and computation-graph compilation. Each chunk is encoded, edited by the few-step DiT, and decoded immediately. Clean key-value states are cached for subsequent chunks, retaining the first chunk as a global sink and a sliding window of recent chunks. This optimized pipeline, combined with bounded KV reuse, allows the model to support real-time 720p editing at approximately 30 FPS on a single GPU.

Experiment

The evaluation compares JoyAI-Video-Edit against streaming and offline video editors on short-video benchmarks, a newly constructed one-minute long-video benchmark, human preference studies, and deployment efficiency profiling. The experiments validate that the model substantially outperforms existing streaming methods, achieves editing quality competitive with strong offline systems, and sustains coherent editing over unbounded video streams with high throughput. Ablation studies confirm that source-anchored distillation and long-horizon alignment are complementary, and the deployment optimizations enable continuous, real-time editing workflows that extend video editing beyond traditional post-production.

The foundation model is trained in a multi-stage curriculum that begins with text-to-image pretraining at increasing resolutions before transitioning to text-to-video training. Video stages mix image and video data, with the number of video-only samples decreasing from 370M to 30M as resolution and temporal fidelity increase. Learning rates are halved for higher-resolution video phases, and a final supervised fine-tuning stage uses only 30M high-quality samples. Image pretraining used 4.3B samples at 256×256, then 793M samples at 512×512. Video training started at 256p with 12/24 fps and 370M video samples, then moved to 360p at 24 fps with 108M samples. The supervised fine-tuning stage employed 480p resolution, 24 fps, and only 30M video samples, with a reduced learning rate of 5×10⁻⁵.

JoyAI-Video-Edit achieves the highest overall score among streaming video editors on OpenVE-Bench, substantially outperforming prior streaming methods and rivaling strong offline systems. It ranks first in four of five editing categories, with particularly large improvements in local change and local removal, while maintaining causal inference. This performance narrows the quality gap between streaming and offline editing, and open-source alternatives remain far behind. JoyAI-Video-Edit attains an overall score of 3.60, exceeding the next-best streaming method by 0.98 points. It leads streaming methods in four of five categories, with the largest margins in local change and local removal. Its local-removal score surpasses all offline and streaming methods, and it ties the best local-change score among all evaluated systems. The open-source VACE model scores only 1.57 overall, illustrating the difficulty of high-fidelity streaming editing without proprietary optimizations.

On one-minute video editing tasks, JoyAI-Video-Edit achieves the highest overall editing quality and the fastest throughput among all evaluated streaming methods. It leads in every editing category, surpassing the strongest prior method by a wide margin in overall score while also processing over 30 frames per second. JoyAI-Video-Edit attains an overall score of 3.30, outperforming the next best streaming editor by 1.59 points. It achieves a throughput of 30.19 FPS, making it the fastest method in the comparison while also delivering the best local-change and global-style scores.

Among streaming video editing methods, JoyAI-Video-Edit achieves the lowest full-pipeline latency and highest throughput, processing 81-frame inputs at 30.19 FPS at 720×1280 resolution. Its video autoencoder runs at 200 FPS, greatly reducing the encoding/decoding bottleneck. Although SANA-Streaming has a faster standalone diffusion transformer, JoyAI-Video-Edit's full pipeline is over twice as fast as SANA-Streaming's and all other compared methods. JoyAI-Video-Edit's end-to-end latency of 2.68 seconds is the lowest, and its 30.19 FPS throughput is the highest, outperforming methods operating at lower resolutions. The VAE component reaches 200 FPS, far surpassing the next-fastest VAE at 37.26 FPS, effectively eliminating the autoencoding bottleneck. While SANA-Streaming's DiT is the fastest individually (54.36 FPS), JoyAI-Video-Edit's full pipeline provides 2.08x the overall throughput of SANA-Streaming.

Source-anchored distillation (SA-DMD) and long-horizon autoregressive distillation (LHAD) each strengthen video editing in complementary ways. SA-DMD delivers the largest single-component gain, especially for global style and local change, while LHAD improves background change and local removal by stabilizing error accumulation in extended sequences. Their combination yields the best overall score and the highest results on all three local editing tasks, confirming that source anchoring and long-horizon optimization work together effectively. SA-DMD alone gives the largest single performance boost, sharply raising global style and local change scores. LHAD alone raises the overall score and particularly benefits background change and local removal, where long-sequence stability matters most. Enabling both components achieves the best overall score and leads across all local editing metrics, demonstrating strong complementarity.

JoyAI-Video-Edit is evaluated on the OpenVE-Bench streaming video editing benchmark, where it sets a new state of the art among streaming methods, substantially narrowing the quality gap with offline editors. It delivers the highest overall editing quality on full one-minute sequences while also achieving the lowest end-to-end latency and highest throughput, exceeding 30 FPS at 720p resolution. Ablation studies confirm that source-anchored distillation and long-horizon autoregressive distillation provide complementary gains, and their combination yields the best results across all local editing categories.