One-sentence Summary

Researchers from The Chinese University of Hong Kong, Shenzhen and LIGHTSPEED propose Ex-Omni-2D, an omni-modal dialogue model that generates text, personalized speech, and reference-conditioned video by predicting a Visual Thought Plan and shared acoustic-temporal speech units, enabling training from heterogeneous data without direct supervision, and distills a teacher video generator into a few-step streaming student with Prefix Streaming, achieving RTF 1.293.

Key Contributions

• Ex-Omni-2D is a dialogue-conditioned response formulation that uses a Visual Thought Plan (VTP) to explicitly describe scene, emotion, and motion, thereby guiding the coordinated generation of text, personalized speech, and reference-conditioned video.
• A native multi-codebook speech unit interface creates a shared acoustic-temporal representation that aligns speech decoding with frame-level video conditioning and enables training on heterogeneous speech, dialogue, and avatar-video data without paired video-response dialogues.
• A Prefix Streaming mechanism distills the full-sequence video generator into a block-causal streaming student, propagating a clean latent across chunks to reduce cumulative late-chunk degradation and achieving an end-to-end RTF of 1.293 at 400×720/720×400 with four-step inference.

Introduction

Omni-modal dialogue systems can understand speech, text, and visual inputs and produce spoken replies, but their responses remain visually disembodied. Existing talking-avatar and joint audio-video generation methods produce strong visual outputs, yet they are driven by completed waveforms, externally supplied prompts, or static content references rather than deriving visual intent directly from the dialogue state. The authors introduce Ex-Omni-2D, a framework that generates dialogue-native video responses by first inferring a structured Visual Thought Plan from the conversational context and then using a shared multi-codebook speech-unit stream to jointly control speech synthesis and frame-aligned avatar animation. This factorization sidesteps the need for large-scale dialogues paired with target video, training each modality-specific pathway on the heterogeneous data available to it, and a distilled streaming student model with prefix streaming enables efficient incremental inference for interactive deployment.

Method

Ex-Omni-2D receives a multimodal user query $\boldsymbol{x} = (\boldsymbol{x}_t, \boldsymbol{x}_s)$x=(xt​,xs​) containing text, speech, or both, along with a reference image $I^{\mathrm{ref}}$Iref and reference audio $a^{\mathrm{ref}}$aref. It generates an omni-modal response comprising text $\mathbf{y}$y, personalized speech, and a reference-conditioned video $\mathbf{V}$V. The framework is organized around two intermediate interfaces: a structured Visual Thought Plan (VTP) $\mathbf{p}$p that conveys high-level visual intent, and native multi-codebook speech units $\mathbf{U}$U that provide the acoustic content and timing shared by speech and video generation.

As shown in the figure below:

The authors leverage distinct pathways for character grounding. Text and speech queries are embedded and projected into a shared hidden space to form the multimodal query representation $\mathbf{E}_x$Ex​. The reference image is processed by a vision tower and projected into the dialogue-model space as $\mathbf{z}_I^{\mathrm{llm}}$zIllm​, while independently being encoded by a 3D VAE within the Video Generator to obtain an appearance-reference latent $\mathbf{R}^{\mathrm{ref}}$Rref. Reference audio is handled inside the Speech Generator, where a speaker encoder extracts the voice embedding $\mathbf{s}^{\mathrm{ref}}$sref.

For visual response planning, the dialogue backbone follows a structured protocol that separates the internal visual plan from the user-facing response. The model outputs a sequence containing a thinking block restricted to the structured VTP and a response block for the user-facing text. The VTP contains five fields describing the first-frame scene, overall scene, emotion, movement style, and detailed motion. This textual interface is encoded into a semantic condition $\mathbf{L}^{\mathrm{vtp}}$Lvtp for the Video Generator. The final-layer hidden states over the response span, denoted $\mathbf{H}_{\mathbf{y}}^{\ell}$Hyℓ​, are retained for speech generation.

The Speech Generator predicts a sequence of multi-codebook acoustic units $\mathbf{U} \in \mathbb{N}^{N \times C}$U∈NN×C conditioned on the response states, response tokens, and reference voice. These units serve as the shared representation of the generated speech. A codec decoder renders them as waveform audio, while an adapter maps them into frame-aligned video-conditioning features $\mathbf{A}$A. Since units are produced at 12.5 Hz and video at 25 FPS, each acoustic feature is repeated for two video frames, providing an explicit temporal alignment.

The Video Generator has two realizations. The full-sequence Teacher generates the primary video response using bidirectional temporal context. It receives the reference latent $\mathbf{R}^{\mathrm{ref}}$Rref, frame-aligned acoustic condition $\mathbf{A}$A, and VTP condition $\mathbf{L}^{\mathrm{vtp}}$Lvtp. The denoising network is conditioned on these inputs, and acoustic features are injected into specific DiT blocks. To enable incremental deployment, the authors distill the Teacher into a Prefix-Streaming Student. The Student uses bidirectional attention within four-latent denoising windows and causal attention across windows. It employs a Prefix Streaming mechanism where the last clean latent of the preceding chunk is reused as a one-latent prefix for the next window, anchoring the chunk boundaries and preventing temporal drift.

Training proceeds in four stages. Stage 1 aligns the speech interface using ASR and TTS data. Stage 2 adapts the omni-modal response with the VTP-response protocol. Stage 3 trains the full-sequence Video Generator on avatar clips to realize the semantic and acoustic-temporal interfaces, freezing it as the Teacher. Stage 4 distills the Teacher into the Streaming Student using flow-map learning and on-policy distribution matching.

Experiment

The evaluation uses CommonEval for audio-video generation, OmniCharacter for multi-turn dialogue, and VoiceBench for speech QA, comparing Ex-Omni-2D against cascaded baselines that share a common response controller. Multitask experiments show that the model can generate personalized speech and video with a controllable quality–efficiency trade-off, while maintaining competitive dialogue fluency and coherence. Ablations confirm that response-specific visual planning improves subject consistency and lip synchronization, and that prefix streaming reduces late-chunk subject degradation. Overall, the system achieves strong cross-modal generation but reveals capability trade-offs in dialogue and end-to-end latency, without uniform improvement across all metrics.

The comparison spans video-only, joint video-audio, and dialogue-based omni-modal generation paradigms. Standalone video renderers like echomimic and OmniAvatar-1.3B lead in subject consistency, motion, and lip-sync respectively, while the Ex-Omni-2D Teacher integrates dialogue, personalized speech, and visual planning at the cost of not dominating any single rendering metric. The Streaming Student variant exposes a controllable quality–efficiency trade-off through denoising step budget. Among video-only baselines, echomimic achieves the highest subject consistency and dynamic degree. OmniAvatar-1.3B obtains the strongest lip-speech synchronization score. The Ex-Omni-2D Teacher adds dialogue response and personalized speech generation, sacrificing standalone rendering scores for omni-modal capability. Increasing the Streaming Student's denoising steps from two to eight consistently improves video quality and sync scores. Replacing the visual plan with a fixed neutral plan reduces subject consistency and lip-sync, while motion incidence increases, revealing sensitivity to planning.

Ex-Omni-2D achieves the highest Fluency, Coherency, and Consistency among evaluated models on OmniCharacter's multi-turn dialogue benchmark, surpassing its Qwen3-8B backbone. Its overall twelve-dimension average is only marginally higher than Qwen3-8B, and some metrics decrease, indicating a capability trade-off rather than uniform improvement. Ex-Omni-2D reached the top Fluency, Coherency, and Consistency scores (average 3.938), compared to Qwen3-8B's 3.537. Ex-Omni-2D's twelve-dimension average (3.283) is close to Qwen3-8B (3.264), with both increases and decreases in individual metrics, revealing a trade-off.

The proposed model achieves the second-highest scores across all three speech QA benchmarks, trailing only Qwen2.5-Omni. It outperforms other open-source alternatives such as VITA-1.5, Moshi, Mini-Omni2, and SLAM-Omni by clear margins, with the largest advantage on the BBH benchmark. The model ranks second in every metric, with AlpacaEval 4.28, CommonEval 3.71, and BBH 58.70. It improves over VITA-1.5 by 0.07 on AlpacaEval, 0.05 on CommonEval, and 3.4 points on BBH.

The Streaming Student offers a tunable quality–efficiency trade-off. With only 2 denoising steps it achieves very high throughput and maintains a competitive SC score, but dynamic degree and sync quality are low. Increasing steps to 8 recovers much of the Teacher's quality, nearly matching SC and Sync-C while still delivering over 10× higher frame rate. At 2 denoising steps, the Student reaches 39.5 FPS and an end-to-end real-time factor of 1.2, whereas the Teacher runs at 1.4 FPS with a factor of 26.9. Raising Student steps from 2 to 8 improves DD from 9.50 to 48.00 and Sync-C from 3.51 to 4.00, closing the gap to the Teacher's 72.00 DD and 4.95 Sync-C.

Ablating response-specific visual planning reduces semantic consistency and synchronization while inflating a motion quantity metric, particularly when personalized speech is present. Removing personalized reference speech and using a fixed public utterance improves no-reference audio quality and clarity but nearly eliminates speaker similarity. The visual plan's first-frame field is much less grounded in the reference image than the remaining fields, which show strong cross-modal consistency. Removing response-specific visual planning lowers semantic consistency and sync, while increasing optical-flow motion incidence, especially with personalized speech. Replacing personalized reference speech with a fixed public utterance boosts no-reference audio quality and clarity but collapses speaker similarity. The first-frame field of the visual plan is far less grounded in the reference image (43%) than the other fields (average 85%). Co-generated emotion and movement style fields achieve high cross-modal consistency, with scores around 90–94%.

The evaluation compares video-only, joint video-audio, and omni-modal dialogue generation, with Ex-Omni-2D Teacher integrating dialogue and personalized speech but sacrificing standalone rendering metrics. The model achieves the highest dialogue fluency, coherency, and consistency and ranks second on speech QA benchmarks, outperforming other open-source alternatives. Ablations confirm that response-specific visual planning and personalized speech are crucial for semantic consistency and synchronization, while the Streaming Student offers a tunable quality-efficiency trade-off, recovering near-Teacher quality at over 10× higher frame rate.