One-sentence Summary

Tsinghua University and Shengshu Technology introduce Vidu S1, a real-time interactive video generation model that, built with TurboDiffusion and TurboServe, delivers infinite-length 540p video at up to 42 FPS on consumer GPUs, supporting voice-controlled digital characters and personalized custom images without blurring, drift, or visual distortion.

Key Contributions

• Vidu S1 is introduced as a real-time interactive video generation model that enables users to control digital characters through voice instructions at any moment, treating speech as a direct and explicit control signal.
• The model supports indefinitely long video generation without drift or visual collapse by mitigating error accumulation during streaming, allowing stable open-ended streams.
• An efficient inference and serving stack with TurboDiffusion and TurboServe delivers 540p real-time generation at up to 42 FPS on consumer GPUs, and experiments on Vidu-StreamBench and HDTF report leading metrics (CSIM 0.9192, Sync-D 7.847, DOVER 0.5660) while satisfying real-time constraints.

Introduction

Recent video generation models produce high-quality output but operate in an offline, one-shot paradigm where users passively wait for a complete video. This paradigm fails to serve interactive visual experiences such as live streaming, gaming, and face-to-face communication, where content must respond immediately. The authors argue that real-time interactive video generation represents a far larger demand than offline generation because offline videos can be replayed and shared, while interactive content scales directly with the number of users. Prior autoregressive video generation methods still lack real-time user interaction, do not use speech as a direct control signal, suffer from error accumulation that causes drift and collapse in long streams, and lack the efficient inference infrastructure needed for practical deployment. The authors introduce Vidu S1, a real-time interactive video generation model that enables continuous speech-guided control, supports indefinitely long stable streaming by mitigating error accumulation, and achieves 540p resolution at 42 FPS on commodity GPUs through an optimized inference and serving stack.

Dataset

The authors construct a training corpus of single-person, single-shot video clips drawn from two main sources, processed through a multi-stage pipeline that ensures visual clarity, temporal coherence, and cross-modal consistency.

Dataset composition and sources

• Livestream and talking-head videos: used to learn fine-grained facial expressions, body movements, and lip synchronization.
• High-quality footage from films and television dramas: used to improve generalization across diverse shot angles, scenes, and visual styles.
• All raw material is deduplicated and pre-filtered on frame rate, resolution, audio-visual integrity, and audio-visual synchronization.

Key processing details

• Segmentation: raw videos are split into single-shot clips of 3–60 seconds; cut points are constrained to avoid interrupting speech.
• Joint filtering: expert models (e.g., face detection, aesthetic scoring) are supplemented by an omni model that provides global semantic labels along dimensions like editing, subject, action, emotion, face, speech, scene, shot, and tone. Filtering rules enforce:
• Exactly one subject with a reasonable frame proportion.
• No watermarks, subtitles, or overlaid advertisements.
• High visual quality (clear, complete, no blur/jitter/flicker).
• Content safety (NSFW removal).
• Static or slow-motion shots for shot stability.
• Clear subject actions/behaviors for interactivity.
• Speech diarization: voice activity and active speaker detection assign each speech segment as onscreen (speaker matches visible subject), offscreen, or overlap. Overlapping segments are discarded. A heuristic further removes segments where the speaker is vocalizing but the speech energy proportion is too low, mitigating instability in singing or heavy background music.
• Captioning: two granularities are produced:
• Clip-level captions: a coherent global description of the entire clip.
• Speech-aware chunk-level captions: temporally aligned descriptions for fine-grained conditioning.
• A dual-path strategy decouples visual attributes (inferred from frames) and auditory attributes (inferred from audio) to reduce cross-modal hallucination. Descriptions cover subject appearance, actions, motion, emotion, scene, camera language, lighting, dialogue, sound effects, and background music.

How the data is used

• The processed clips and their structured captions serve as the training set for a controllable character video generation model. No explicit training split or mixture ratios are reported; the corpus is used to learn motion, lip sync, and stylistic diversity under the conditioning signals provided by both clip-level and chunk-level annotations.

Method

The authors leverage a progressive data processing pipeline to convert heterogeneous raw videos into high-quality, temporally coherent, and semantically annotated training clips. As shown in the figure below:

This staged design jointly improves visual clarity, temporal stability, audio-visual consistency, and cross-modal interpretability. The pipeline begins with a prefiltering stage to discard technically unreliable videos based on metrics like frame rate and audio-visual synchronization. The remaining data is segmented into single-shot clips, followed by subject-level filtering, quality and safety screening, speech diarization, and caption generation. To overcome the limitations of expert models, an omni model performs global semantic understanding, forming a joint filtering system with local detail sensitivity. Speech diarization extracts the speech component and assigns segments to onscreen, offscreen, or overlap categories, filtering out overlaps and low-energy vocalizations to preserve signal reliability. Finally, a dual-path captioning strategy decouples visual and auditory modalities to generate structured, fine-grained descriptions.

The training pipeline consists of three stages designed to build a streaming video-audio generation model. In the first stage, the authors train a bidirectional teacher on full video-audio sequences. The model is conditioned on the complete sequence and trained to denoise the joint latent states, establishing a high-quality generative prior. The objective minimizes the squared $L_2$L2​ distance between the predicted and clean joint states.

In the second stage, the authors initialize a causal teacher from the pretrained bidirectional model and adapt it to a streaming autoregressive setting by imposing a causal attention mask. Each target state is denoised conditioned on the available context and historical video-audio prefix. To bridge the gap between training and inference, a hybrid strategy combining Teacher Forcing and Diffusion Forcing is adopted. Teacher Forcing provides stable supervision using clean ground-truth prefixes, while Diffusion Forcing improves robustness to imperfect prefixes during autoregressive rollout.

In the third stage, the authors apply Distribution Matching Distillation with Phased Consistency Models regularization to compress the autoregressive generation process into a few sampling steps. The distillation objective minimizes the discrepancy between the generated and data distributions. To prevent mode collapse and temporal inconsistency, the consistency objective is introduced as a regularization mechanism using a perceptual feature distance. The generator is optimized via a weighted combination of the distillation and consistency objectives.

During inference, the authors employ sliding-window decoding to enable online autoregressive generation under limited resources. Attention is restricted to a fixed-length window comprising a persistent reference context, cached historical states, and the current state being denoised. To improve efficiency and temporal stability, Rotary Position Embedding Repositioning is utilized. Historical key-value features are cached before applying the positional embedding, and as the window advances, the embedding is applied according to updated relative positions, avoiding redundant recomputation.

Furthermore, the authors introduce TwinCache, a stage-aware caching strategy maintaining complementary noisy and clean caches for historical states. During intermediate denoising steps, the model attends to the noisy cache, which preserves coarse temporal dynamics and suppresses high-frequency artifact accumulation. At the final denoising step, the clean cache is attended to restore fine-grained visual details while maintaining temporal coherence.

To achieve real-time inference, the authors deploy a hardware-software co-designed acceleration stack. Attention is accelerated using specialized algorithms to reduce latency. For linear layers, custom per-block quantized GEMM operators are implemented in CUDA to mitigate the impact of outliers and reduce memory footprint. Operator sequences are fused into custom Triton or CUDA kernels to cut host launch overhead and memory bandwidth pressure. Additionally, CUDA Graph is adopted to capture fixed-structure subgraphs, merging independent kernel launches into a single graph launch to improve GPU utilization. Finally, context parallelism distributes computation and activation memory across multiple GPUs, with optimized collectives minimizing communication overhead.

Experiment

Vidu S1 is evaluated on a new interactive benchmark, Vidu-StreamBench, which tests action instruction following, motion naturalness, and stability, alongside standard HDTF metrics. In pairwise comparisons, it is consistently preferred over leading commercial systems, with a 100% preference rate for subject controllability against HeyGen and LemonSlice, and shows superior visual stability and instruction adherence. Additionally, Vidu S1 achieves 42 FPS on RTX 5090 GPUs, exceeding the real-time threshold while preserving strong identity consistency and audio-visual synchronization.

Vidu S1 is the only method that combines instruction following with real-time generation, while achieving leading scores in identity preservation (CSIM), audio-visual synchronization (Sync-D), and perceptual quality (DOVER) on the HDTF benchmark. It delivers 540p video at 42 FPS, exceeding the 30 FPS real-time threshold, and is strongly preferred over commercial systems in user studies, particularly for subject controllability. Vidu S1 uniquely supports both instruction following and real-time generation, and it achieves the best identity preservation, lowest audio-visual sync error, and highest perceptual quality among all compared methods on HDTF. With 42 FPS throughput at 540p resolution, Vidu S1 surpasses the 30 FPS real-time playback requirement and outperforms the 25 FPS of the commercial system HeyGen.

In pairwise human evaluations on Vidu-StreamBench, Vidu S1 was preferred over leading commercial avatar systems across all measured dimensions. The advantage was most pronounced in subject controllability, where raters unanimously favored Vidu S1, and in motion dynamics, where it received a large majority of preferences. Audio-video synchronization and identity consistency also favored Vidu S1, though identity consistency had a higher proportion of tied ratings. Overall, Vidu S1 was preferred over competitors by a clear margin, with 56% preference versus 28% for the opponent and 16% ties. Subject controllability showed a 100% preference rate for Vidu S1, indicating perfect agreement among raters when avatars needed to follow explicit action instructions. Motion dynamics strongly favored Vidu S1, with 68% preference compared to only 14% for the competing system. Audio-video synchronization also leaned toward Vidu S1 (52% vs. 20%), while identity consistency saw a smaller but still significant advantage (44% vs. 22%) with more tied ratings.

Vidu S1 uniquely integrates instruction following with real-time generation, achieving top performance on HDTF in identity preservation, audio-visual synchronization, and perceptual quality while delivering 42 FPS at 540p. In human evaluations, it was strongly preferred over commercial avatar systems across all dimensions, with unanimous preference for subject controllability and large margins in motion dynamics, confirming its superior ability to follow explicit action instructions and produce natural motion.