One-sentence Summary

Tsinghua University and Shengshu Technology present Vidu S2, comprising Vidu S2-Avatar and Vidu S2-Editing, which achieve real-time $720\text{p}$720p interactive digital-character generation with dynamic references and stronger instruction following, plus real-time video editing for style transfer, virtual try-on, and character or background replacement, outperforming all baselines with a playable demo at https://vidu.com/vidu-stream.

Key Contributions

• Introduces Vidu S2-Avatar, a real-time interactive digital-character model that achieves 720p video generation, supports dynamic references updated at any moment during a stream, and improves instruction following for actions such as dancing.
• Presents Vidu S2-Editing, a real-time video editing model that edits incoming video streams on the fly, covering style transfer, virtual try-on, character replacement, and background replacement, with support for both monocular and stereoscopic inputs.
• Demonstrates real-time spatial video generation and editing for VR head-mounted displays, converting avatar-generated or edited monocular streams into synchronized left- and right-eye views, and reports that Vidu S2 outperforms all baselines while meeting real-time inference requirements.

Introduction

Current video generation models like Sora, Veo, and Wan produce high-quality content but rely on an offline, one-shot paradigm where users wait minutes for a complete video and cannot interact during generation. This limitation stems from the diffusion-based approach, which denoises the entire video synchronously over many steps. While suitable for pre-generated content, it fails to meet the demand for interactive visual experiences such as live streaming, gaming, or face-to-face communication, where content must respond instantly to user input. The authors argue that real-time interactive generation has far greater demand potential than offline generation, since offline videos can be replayed and shared, reducing the need for new generations per user.

Prior work, such as Vidu S1, made initial strides by enabling real-time, voice-controlled video generation with infinite length and up to 42 FPS on consumer GPUs. However, it was limited to talking-head-centric characters, capped at 540p resolution, fixed the character reference once a stream started, struggled with large body motions like dancing, and could not edit incoming video streams. These constraints hindered broader applicability and user control.

The authors introduce Vidu S2, a suite that addresses these gaps with two key models. Vidu S2-Avatar improves on S1 by supporting 720p real-time generation, allowing users to update reference images at any moment during a stream, and following a wider range of instructions, including dancing. It uses a novel Self-Replay Forcing technique to improve training stability and data efficiency. Vidu S2-Editing enables real-time editing of video streams, covering style transfer, virtual try-on, character replacement, and background replacement, using frame-aligned attention to preserve motion and timing. The authors also explore real-time spatial video generation and editing for VR headsets, converting monocular streams into synchronized left- and right-eye views or directly editing stereoscopic input. An efficient inference stack, leveraging attention optimizations, low-bit GEMM, and multi-GPU parallelism, makes these capabilities practical on low-cost hardware.

Dataset

Dataset Description

The authors construct training data for two model variants, Vidu S2-Avatar and Vidu S2-Editing, with distinct pipelines.

Vidu S2-Avatar Data

• Sources and Composition: The dataset builds on the Vidu S1 framework and expands sources to include livestream/talking-head videos, film and television content, high-quality solo dance videos, and 2D/3D animation data. This mix enriches facial expressions and body movements.
• Pipeline Stages: The five-stage pipeline (Clipping, Filtering, Speech Processing, Captioning, Embedding) is retained, with refined clipping and filtering methods.
• Video Clipping: The single-shot clipping approach is kept, but a vision-language model is used for a second-stage assessment of candidate cuts. This reduces false positives from subtle edits in vlogs and unboxing videos, keeping the false rejection rate below 2%.
• Video Filtering: The six-dimensional taxonomy from Vidu S1 is retained (subject detection, frame cleanliness, visual quality, content safety, shot stability, interactivity). A new high-clarity selection operator is added, and the hard criterion for shot stability is relaxed.
• High-Clarity Video Selection: A multidimensional hybrid framework evaluates videos by resolution, frame rate, codec, pixel format, bit depth, and bitrate. Expert models assess technical quality, texture detail, edge sharpness, and compression artifacts. A weighted quality score determines final selection.
• Background Stabilization: Videos with small or smooth camera movements are retained and processed by a background stabilization operator. Foreground subjects are masked, camera transformations are estimated from background regions, and geometric correction with cropping yields stable backgrounds. A second filtering pass ensures the subject remains in frame.
• Data Captioning: Captions use temporally ordered dense descriptions instead of structured fields. Events are described chronologically with temporal boundaries, actions, and outcomes. Chunk-level captions are segmented at event boundaries, not fixed intervals. The annotation pipeline uses a hierarchical, multi-agent design with expert models for object detection and speech recognition, followed by dedicated agents for recognition, description, verification, and filtering.

Vidu S2-Editing Data

• Source and Scale: Training data is derived from videos filtered through the Vidu S2-Avatar pipeline. Stricter filtering and label balancing yield 800,000 videos. From this, four disjoint subsets of 200,000 videos each are sampled for four tasks: style transfer, subject replacement, background replacement, and virtual try-on.
• Style Transfer Data Construction: A conditional video generation model is trained to reconstruct a raw video from a surface-normal video (estimated with a model like NormalCrafter) and a reference image. For training data, the reference image is replaced with a stylized image, and the model generates a stylized video following the original spatial structure and motion. This generalizes to over 50 styles.
• Other Editing Tasks: Similar procedures use task-specific video editing models. Open-source models (Bernini, SCAIL-2, Wan2.1 VACE, SAMA-14B, CoinVE-Edit) supplement the data. Post-filtering and comparative selection retain the best source-edited video pairs, as no single model consistently outperforms others.
• Data Captioning: Each caption identifies the editing task and describes the reference image attributes relevant to that task (e.g., visual style for style transfer, clothing attributes for virtual try-on). Captions explicitly require preservation of all source-video content outside the intended edits. They are generated without feeding the source video to the VLM, preventing content leakage through text conditioning.

Evaluation Benchmarks

• Digital Character Generation: Public evaluation uses StreamAV-Bench (160 scenarios per track, with an Interactive Track receiving updates every 30 seconds for up to 180 seconds). An internal benchmark covers streaming quality, long-horizon stability, runtime updates, interruption and recovery, audio-video synchronization, responsiveness, and cost.
• Video Editing: Public benchmarks include OpenVE-Bench and Sparkle-Bench for instruction-guided editing, RefVIE-Bench for reference-conditioned edits, and the ViViD test set for virtual try-on. An internal benchmark covers all four editing tasks, plus a long-horizon set with 10-minute clips to test identity preservation, edited-region stability, and temporal background stability.

Method

The authors leverage a comprehensive data preparation pipeline to construct high-quality training datasets for both Vidu S2-Avatar and Vidu S2-Editing. As shown in the framework diagram, the pipeline for Vidu S2-Avatar processes diverse video sources, including talking heads, film and TV content, solo dance, and 2D and 3D animations. The workflow consists of five core stages: clipping, filtering, speech processing, temporal dense captioning, and embedding. To enhance data quality, the authors introduce operator enhancements such as cut-point detection, high-clarity selection, and background stabilization. For Vidu S2-Editing, the pipeline further curates the filtered Avatar data through stricter filtering and label balancing to yield 800,000 selected clips. These clips are divided into four disjoint task subsets for style transfer, subject replacement, background replacement, and virtual try-on. The editing data construction involves building reconstruction data to train a video-to-video model, which is then used to generate paired editing data alongside other task-specific editors, followed by post-filtering and comparative selection.

Vidu S2-Avatar employs an audio-visual joint Diffusion Transformer for reference-conditioned video-audio generation. Given a reference image $r$r and a conditioning sequence $c^{1:\infty}$c1:∞, the model jointly predicts the clean video-audio latents for the first $N$N segments:

where $\pmb{x}_{t}^{1:N}$xt1:N​ denotes the noisy joint video-audio latents at diffusion timestep $t$t. To support both image-to-video and reference-to-video generation within a single model, the authors jointly train a bidirectional model with segment-wise conditional supervision.

To enable streaming generation, the bidirectional temporal attention is replaced with a block-wise causal attention mask. The causal denoising process for the $i$i-th segment is formulated as:

where $\boldsymbol{x}_{\tau_{i}}^{<i}$xτi​<i​ denotes the historical video-audio states at noise level $\tau_{i}$τi​. The training strategy combines Teacher Forcing and Diffusion Forcing to improve robustness to accumulated errors. Furthermore, the authors introduce Self-Replay Forcing, an on-policy distribution matching distillation that aligns the student's autoregressive rollout distribution with the teacher distribution while preserving cross-block gradient flow. The loss function for Self-Replay Forcing is defined as:

Preference optimization is applied at both bidirectional and streaming stages to mitigate visual quality degradation. A one-step super-resolution refiner operates in the latent space to recover fine-grained spatial details, utilizing asymmetric noise levels for historical caches to separate temporal propagation from spatial detail restoration.

Vidu S2-Editing employs a similar Diffusion Transformer architecture for instruction-guided video editing. The model predicts clean video latents conditioned on a source video $s^{1:N}$s1:N, a reference image $r$r, and editing instructions $c^{1:\infty}$c1:∞:

The training follows a bidirectional video editing phase with frame-aligned attention, followed by causal streaming training using the same hybrid forcing and Self-Replay Forcing framework.

To achieve real-time and low-latency inference, the authors adopt an end-to-end inference acceleration framework. They utilize a layer-wise hybrid attention strategy, selecting appropriate approximate attention methods based on layer sensitivity. Linear layers are accelerated using a per-block W8A8 GEMM implementation to limit the effect of outliers while maintaining numerical precision. Kernel fusion and CUDA Graphs are employed to reduce launch overhead and global-memory traffic. Multi-GPU context parallelism with quantized tensor exchange further scales execution efficiently. For Vidu S2-Editing, fine-grained inter-module scheduling coordinates the VAE encoder, backbone, refiner, and VAE decoder on a shared timeline to maximize GPU resource utilization.

The system incorporates an agentic workflow to manage user interactions and prompt generation. As illustrated in the system diagram, users provide text or audio instructions, an initial image, and optional reference images. A vision-language model agent processes these inputs to generate prompts for Vidu S2-Avatar and reviews the generated video frames to provide visual feedback, refining subsequent prompts.

The agent handles complex scenarios such as reference object generation, scene transitions, and accessory manipulation. The visual examples demonstrate the system's capability to seamlessly integrate reference objects, transition between different scenes, and manage the taking off and putting on of accessories while maintaining temporal consistency and character identity.

Experiment

The evaluation covers two complementary tasks: streaming digital character generation and streaming video editing. On public benchmarks, Vidu S2-Avatar achieves the best scores across all reported metrics for character generation, with particularly strong audio-visual alignment, synchronization, and near-ceiling subject and background consistency over long rollouts. For video editing, Vidu S2-Editing outperforms both offline and streaming baselines on instruction-guided editing, reference-conditioned editing, and virtual try-on benchmarks. Complementary human preference evaluations using randomized GSB pairwise comparisons show consistent advantages for Vidu S2 over commercial systems, with the largest gains in overall quality, temporal consistency, and long-horizon stability. Qualitative case studies confirm that Vidu S2 preserves identity, fine-grained geometry, and scene structure across extended streams, while baselines exhibit facial drift, body-proportion changes, and local distortions.

Vidu S2-Avatar outperforms all baselines on the StreamAV-Bench digital-character benchmark, achieving the best score in every reported metric. Its gains span visual quality, audio quality, cross-modal alignment, and long-horizon stability, indicating a balanced improvement rather than a single-dimension advantage. Vidu S2-Avatar leads all other systems in visual appeal and fidelity, with higher VA and VQ scores. The method also achieves the best audio quality metrics, with cleaner and more natural speech. Cross-modal coordination is stronger, as shown by tighter AVAlign and lower AVSync values. Identity and scene stability are near-perfect, with SC and BC scores close to the maximum.

Vidu S2-Editing achieves the top overall score and leads across all individual metrics on the Sparkle-Bench video editing benchmark, outperforming both offline and streaming baselines. Its gains are consistent across foreground and background criteria, indicating balanced editing quality. Vidu S2-Editing obtains the highest overall score among all evaluated models. It leads in global instruction, global visual quality, foreground instruction, foreground motion, background dynamics, and background visual quality. Its performance surpasses the strongest offline baseline by a clear margin on overall score.

The proposed streaming method, Vidu S2-Editing, achieves the highest scores across all reported metrics on the joint OpenVE and RefVIE evaluation, surpassing both offline and streaming baselines. Its advantage is most pronounced on OpenVE, particularly for global style and background change, while it also leads on RefVIE by a smaller margin. Vidu S2-Editing tops the joint overall score, outperforming the strongest offline baseline by a clear margin. The method leads on both OpenVE global style and background change, contributing to its high OpenVE overall score. On RefVIE, Vidu S2-Editing edges out the best streaming baseline, showing a smaller but still leading improvement.

On the ViViD unpaired virtual try-on benchmark, Vidu S2-Editing achieves a lower VFID score than all listed baselines, indicating closer alignment between generated and reference video distributions. The method outperforms CatV2TON and ViViD by a substantial margin, while other baselines such as StableVITO and OOTDiffusion show much higher VFID values. Vidu S2-Editing achieves the best VFID score among all evaluated methods. The improvement over the closest strong baseline (ViViD) is large, with VFID reduced by more than half. Diffusion-based try-on baselines (StableVITO, OOTDiffusion) produce notably higher VFID values, indicating weaker distribution alignment.

Vidu S2-Avatar and Vidu S2-Editing consistently outperform all offline and streaming baselines across four benchmarks. On StreamAV-Bench, Vidu S2-Avatar achieves the best scores in all visual, audio, alignment, and stability metrics, indicating balanced improvements. On Sparkle-Bench and the joint OpenVE/RefVIE evaluation, Vidu S2-Editing leads every metric, with the largest gains in global style and background changes. On the ViViD try-on benchmark, Vidu S2-Editing reduces VFID by more than half compared to the strongest baseline, showing markedly better distribution alignment.