One-sentence Summary

Researchers from Tsinghua University, NVIDIA, CMU, et al. introduce StreamEval, a unified benchmark for interactive streaming video generation spanning five tasks, 300 scripted test cases, and ten task-specific metrics, showing that high visual quality and identity retention do not guarantee reliable request execution or sustained state consistency in omni-modal real-time interactive systems.

Key Contributions

• StreamEval is a unified benchmark for interactive streaming video generation, spanning avatar dialogue, avatar performance, world generation, instruction editing, and reference editing through 300 scripted test cases of 15–300 seconds with incremental multimodal requests and explicit state change and content preservation requirements.
• The benchmark evaluates performance across four dimensions, task execution, state consistency, visual fidelity, and audiovisual coordination, using ten task-specific metrics on a 0–100 scale.
• Evaluation of 37 models shows that high visual quality and identity retention do not guarantee reliable request execution or sustained state consistency, with the highest reported task-specific scores being 36.90 for avatar-performance Following, 25.93 for text-conditioned world generation Following, and 15.00 for reference-editing Edit Success.

Introduction

As streaming video generation advances, real-time interactive systems must execute new user requests while preserving prior content and state across sessions, with applications in conversational avatars, controllable virtual worlds, and live video editing. Existing benchmarks cover complementary pieces such as visual quality, temporal behavior, composition, multi-turn control, streaming audio-video updates, sustained editing, and conversational cues, but they do not unify avatar interaction, world generation, and real-time editing under explicit task-specific execution and preservation criteria. The authors introduce StreamEval, a benchmark spanning five interactive streaming tasks and 300 scripted test cases of 15 to 300 seconds. Each case specifies timed multimodal inputs, required changes, and content that must remain intact, while ten task-specific metrics evaluate task execution, state consistency, visual fidelity, and audiovisual coordination. They evaluate 37 models, revealing distinct strengths and remaining limitations in instruction following and full-edit completion.

Dataset

The authors construct StreamEval as an evaluation benchmark rather than a training corpus. It contains 300 test cases across five streaming tracks, with 60 cases per task and balanced durations.

• Composition and sources

  • Total: 300 test cases, 60 per task; 12 cases per duration tier at 15, 30, 60, 120, and 300 seconds.
  • Visual/avatar and world-generation inputs: GPT-Image-2 and Qwen-Image synthesize portraits and visual references. Portraits depict original synthetic adults or stylized fictional characters.
  • Dialogue and audio inputs: Qwen3.5-Omni generates expressive user speech and prescribed speech. These recordings are paired with authored personas and verified facts, including clean speech, background noise, and room reverberation.
  • Text-dialogue derivatives: 60 cases replace audio turns with authored text while retaining the same source-family identities.
  • Editing inputs: continuous source videos include licensed real-world footage, animation, and benchmark-generated scenes. Third-party sources include Wikimedia Commons, NASA, and Blender Studio, with per-asset licenses and attribution supplied in the release.

• Metadata and state construction

  • Qwen3.8-Max drafts session goals, timed controls, protected attributes, and expected outcomes.
  • An authored oracle specifies what to establish, retain, cancel, or restore at each checkpoint.
  • Target regions in editing sources are temporally grounded by Qwen3.8-Max at planned control timestamps.

• Processing and filtering

  • Media inputs are screened and revised against explicit state requirements.
  • Editing source videos are continuous: no looping, speed changes, or concatenation.
  • No explicit cropping strategy is described; editing uses temporally grounded target regions within continuous video sources.
  • Automated checks verify media decodability, duration bounds, file identity, and agreement among events, opportunities, and scoring requirements.
  • Synthesized speech is checked with ASR transcription and must have a normalized word-error rate of at most 0.25 against authored transcripts.
  • Qwen3.8-Max audits images for subject clarity and anatomical or rendering defects.
  • Editing sources are checked for scene cuts, frozen spans, and unsuitable targets. Revised candidates must pass the same checks.
  • Judge calibration targets include at most 5% false positives, 10% false negatives, and 10% unresolved decisions per class.

• Statistics

  • The 300 cases span 240 independent source families.
  • The 60 text-dialogue derivatives reuse existing families.
  • One pass through all 300 video-session cases represents 525 minutes of active streaming generation, excluding setup and response tails.
  • The benchmark contains 1,347 planned online events, excluding initialization and continuous source frames.
  • Avatar designs span photorealistic, stylized 3D, anime, and clay or stop-motion appearances.
  • Dialogue includes clean speech, background noise, and room reverberation.
  • Editing covers indoor and outdoor scenes with single or multiple targets.

• How the data is used

  • The dataset is used as a test/evaluation benchmark.
  • Evaluation uses all 300 cases, with three independent runs per case and per model.
  • The five duration strata receive equal aggregation weight within each task.
  • No training split or mixture ratios are reported in these sections; the paper evaluates 37 models on this benchmark.

Method

The authors construct the StreamEval benchmark through a structured pipeline that pairs online inputs with explicit state requirements. The process begins with scenario authoring, where the authors leverage Qwen3.8-Max to draft session goals, timed controls, protected attributes, and expected outcomes based on source frames, task goals, and target states. An authored oracle then specifies what to establish, retain, cancel, or restore at each checkpoint.

Following scenario authoring, the system generates multimodal inputs. Image synthesis utilizes GPT-Image-2 and Qwen-Image to create portraits and visual references. Speech synthesis employs Qwen3.5-Omni to generate expressive dialogue audio. Additionally, licensed videos are processed to extract source windows for live editing tasks.

Once the inputs are constructed, they undergo automated screening and release, which involves modality-specific checks. Visual screening is performed using the VLM Qwen3.8-Max to detect anatomical or rendering defects. Speech matching utilizes Qwen3-ASR to transcribe synthesized speech, requiring normalized word-error rates of at most 0.25 against authored transcripts. Media checks verify duration bounds, scene cuts, and frozen frames.

If an input is flagged during visual screening, such as a portrait with hand defects, the system initiates a recorded revision process. The flagged input is regenerated using Qwen-Image and then rechecked with the VLM. Only inputs that pass all screening checks are retained. Finally, the benchmark cases, including timelines, oracles, and media, are prepared for a versioned release that ensures source-family separation, input and target bindings, and version tracking via hashes.

Experiment

The paper evaluates 37 models on the StreamEval benchmark, which tests real-time interactive video across five tasks: digital-human dialogue, avatar performance, world generation, instruction editing, and reference editing. The setup uses 300 scheduled multi-duration sessions with repeated runs and scores task execution, state consistency, visual fidelity, and audiovisual synchronization. Results show that avatar customization and dialogue quality are relatively strong, but timing-constrained action execution, world continuity under online control, and full video-editing compliance remain difficult; models often preserve source content better than they adopt requested edits or maintain stable rendering.

The comparison indicates that existing benchmarks in the extracted rows are largely confined to world generation, with short or variable evaluation durations, while dialogue, avatar performance, and editing tasks are treated separately or omitted. StreamEval contrasts by spanning dialogue, avatar performance, world generation, instruction editing, and reference editing in one framework, with balanced multi-horizon durations up to 300 seconds. Its longer editing and world-generation horizons extend beyond prior benchmarks such as LongV2VBench and StreamAV-Bench. Listed existing video-generation benchmarks cover only world generation and report short or variable durations, typically up to 60 seconds where specified. StreamEval spans all five assessed streaming tasks and applies balanced duration tiers up to 300 seconds, providing longer evaluation horizons than the cited prior benchmarks.

In digital-human dialogue evaluation, custom-avatar models such as Vidu S2 Avatar lead on overall quality, visual fidelity, dialogue, and identity, with AVTR-1 following among the reported custom-avatar systems. Fixed-avatar systems receive no customization bonus; Tavus CVI Full outperforms LiveAvatar FULL on overall quality and identity and posts the highest audiovisual synchronization among shared models. High stability and visual quality do not necessarily translate into strong dialogue or synchronization. Vidu S2 Avatar achieves the top Overall, Quality, Dialogue, and Identity scores among custom-avatar digital-human dialogue models. Tavus CVI Full records the highest A-V Sync among shared systems and leads fixed-avatar models over LiveAvatar FULL in Q-Overall and Identity. MuseTalk 1.5 combines high visual quality, stability, and identity with much weaker A-V Sync and Dialogue, illustrating trade-offs across evaluation dimensions. LiveAvatar FULL reaches perfect Stability but no measurable A-V Sync, underscoring that temporal consistency can occur without audiovisual alignment.

Vidu S2 Avatar and LiveTalk 1.3B lead avatar performance overall, with Vidu S2 Avatar strongest in audio-visual synchronization, stability, identity preservation, and instruction following, while LiveTalk 1.3B leads in visual quality. Middle-ranking systems often show similar lip synchronization and identity preservation but diverge in temporal stability and visual quality. Executing scripted actions under timing constraints remains the primary challenge, with following performance varying widely across all seven systems. Vidu S2 Avatar and LiveTalk 1.3B lead overall but have distinct profiles: Vidu S2 Avatar excels in synchronization, stability, identity, and following, while LiveTalk 1.3B leads in quality. All seven systems support custom avatars and receive the custom-avatar bonus; following scripted actions under timing constraints remains the weakest area, varying widely even among systems with strong identity and stability.

In world generation, text- and image-conditioned configurations are ranked together, with text-conditioned results used for models evaluated under both inputs. MemFlow leads overall, followed closely by Rolling Forcing, Krea Realtime, HY-World, and HappyOyster. Performance varies sharply across quality, stability, following, and continuity, with leaders in individual metrics often differing from the overall leader. MemFlow has the highest Overall score, closely followed by Rolling Forcing, Krea Realtime, HY-World 1.5 WorldPlay, and HappyOyster 1.0 Adventure. HappyOyster 1.0 Adventure leads in Quality and Stability but scores much lower on Following and Continuity. Krea Realtime has the highest Following score, while Matrix-Game 3.5 has the highest Continuity score.

In instruction editing, Vidu S2 Editing and JoyAI-Video-Edit lead by a wide margin over the other reported models. Vidu S2 Editing achieves the strongest following, edit success, quality, and stability, while JoyAI-Video-Edit leads in preservation. Even the top systems show relatively low quality and stability, indicating that clean temporal rendering during live edits remains difficult. Vidu S2 Editing and JoyAI-Video-Edit lead instruction editing by a wide margin, while the remaining reported models score substantially lower overall. Vidu S2 Editing records the highest following, edit success, quality, and stability, and JoyAI-Video-Edit achieves the strongest preservation. Edit success is consistently lower than following across leading systems, reflecting the added difficulty of satisfying complete editing constraints such as scope, timing, and persistence. Quality and stability remain low even for the highest-scoring models, underscoring the challenge of sustaining clean temporal rendering during live edits.

StreamEval is introduced as a unified benchmark spanning dialogue, avatar performance, world generation, instruction editing, and reference editing, with balanced multi-horizon durations up to 300 seconds, extending beyond prior benchmarks that focus mainly on world generation with shorter or variable durations. In digital-human dialogue and avatar performance, custom-avatar models such as Vidu S2 Avatar lead on overall quality and identity, but high stability does not guarantee audiovisual synchronization, and executing scripted actions under timing constraints remains the weakest area. For world generation and instruction editing, overall leaders such as MemFlow and Vidu S2 Editing are not uniformly strong across all sub-metrics, and even top systems struggle with continuity, preservation, quality, and stability, underscoring task-specific trade-offs and the difficulty of sustained temporal rendering.