Command Palette
Search for a command to run...
Duo: نموذج عالم تبادلي للتفاعل مزدوج الاتجاه
Duo: نموذج عالم تبادلي للتفاعل مزدوج الاتجاه
الملخص
نقدم في هذا التقرير نموذج العالم التبادلي (RWM) ونستعرض Duo v1، وهو نموذج عالم مزدوج الاتجاه يدرك المدخلات السمعية والبصرية باستمرار مع الاستجابة للمستخدمين في الزمن الفعلي. يعامل نموذج العالم التبادلي الذكاء والواقع المادي بوصفهما متساويين في الأهمية لما يشكّل العالم. وتركّز نماذج العالم القائمة إلى حد كبير على محاكاة الواقع المادي، تاركةً الاستدلال واتخاذ القرار بوصفهما ضوابط خارجية لا أجزاء جوهرية من العالم نفسه. وفي نموذج العالم التبادلي، نحدد قدرتين متساويتين في الأهمية: القصد، الذي يوفر الإدراك الكامن وراء التفاعلات مع العالم، والتحقيق، الذي يولّد سلوكًا سمعيًا بصريًا متماسكًا. ولتحقيق هذه الصياغة، يقدم Duo v1 ثلاثة تصاميم تقنية رئيسية: (1) الاستفادة من النمذجة الانحدارية الذاتية لاستدلال التفاعل المستمر وقرارات المشاركة، بدلًا من قصرها على فهم التعليمات لنماذج الانتشار. (2) بناء شروط خاصة بالمقاطع ومزامنة الرموز بين القصد والتحقيق على خط زمني موحد. وتفرض أقنعة الانتباه الخاصة بالوحدات تبعيات سببية داخل المقاطع وفيما بينها، مانعةً الوصول إلى المدخلات المستقبلية. (3) التخفيف من انزياح التوزيع في الوحدتين عبر تقطير مزدوج متدحرج، مع التدريب على رموز القصد المولدة انحدارياً ذاتياً وتواريخ سمعية بصرية مركّبة من التحقيق. وتدرك الوحدتان باستمرار الصوت والفيديو والنص في الزمن الفعلي، مما يتيح تفاعلًا مزدوج الاتجاه مع محادثات عفوية ومقاطعات. وإلى جانب المحادثة، يمكن للمستخدمين مشاركة أشياء حقيقية، وإعادة تشكيل مشاهد افتراضية، وتوجيه المساعدين، أو ممارسة ألعاب سمعية بصرية دون استخدام الفأرة أو لوحة المفاتيح. ويشكل إصدار Duo v1 خطوتنا الأولى، وستصدر النسخة التالية قريبًا. ونتطلع إلى استكشاف أوسع لهذا المجال الناشئ.
One-sentence Summary
The authors propose Duo v1, a Reciprocal World Model for full-duplex audiovisual interaction that unites Intent and Realization via autoregressive reasoning, chunk-synchronized tokens with causal attention, and dual rollout distillation, thereby supporting real-time conversations, interruptions, and embodied interactions beyond mouse-and-keyboard control.
Key Contributions
- Introduces the Reciprocal World Model (RWM) and Duo v1, a full-duplex world model that treats intelligence and physical reality as equally fundamental, defining Intent and Realization as coupled capabilities for interaction cognition and coherent audiovisual behavior.
- Duo v1 implements RWM through autoregressive modeling for continuous interaction reasoning and participation decisions, chunk-specific conditions and token synchronization across Intent and Realization on a unified timeline with causal attention masks, and dual rollout distillation producing a four-step streaming generator.
- Provides audio, video, and text data construction pipelines combining scripted synthetic video calls and recorded conversations, plus a streaming runtime that coordinates intent inference, audiovisual generation, and incremental decoding, enabling real-time full-duplex conversation, interruptions, object sharing, virtual scene reshaping, and audiovisual games without mouse or keyboard controls.
Introduction
World models have advanced rapidly at simulating physical scene dynamics from past observations and control signals, but most systems keep intelligence outside the generated world: users steer through limited inputs such as text or gamepad actions, while nothing inside the world perceives the user in real time or decides how to respond. Prior work addresses related pieces such as learned dynamics for control, full-duplex speech, and streaming audio-visual generation, yet it does not connect a generated audiovisual world to the real world through reciprocal, real-time perception and participation. The authors introduce the Reciprocal World Model and Duo v1, a modular implementation that combines a multimodal Intent Module with a causal audiovisual Realization Module. Duo v1 supports full-duplex interaction in chunks of 17 video frames, using autoregressive interaction decisions, causal chunk conditioning on a unified timeline, and dual rollout distillation to align training with live inference.
Dataset
Dataset overview
The authors construct Training Duo v1 from synthetic and recorded interactions, combined into a unified conversation timeline. The timeline preserves natural timing and controlled events so user inputs and assistant audiovisual behavior are temporally aligned. It is used for causal supervision for Intent, audiovisual targets for Realization, missing-view reconstruction, and recorded-video extensions.
Synthetic data
- Source: generated two-person video calls created by a text-to-video model.
- Layout: split-screen video with the left panel as the user and the right panel as the assistant, with shared synchronized audio.
- Control cards:
- Duration: 30 s, 45 s, or 60 s.
- Language: Chinese or English.
- Persona: relationship or occupation categories.
- Micro-actions: off or on, where on includes 24 small reactions such as nod, raise an eyebrow, or look away briefly.
- Interaction type: everyday exchange, interruption, follow-up/debate, or proactive inquiry.
- Emotion: fixed to follow the topic and conversation.
- Resolution: 1536 x 576 or 1152 x 768.
- Screenplay construction:
- An LLM writes a synopsis and a JSON screenplay.
- Each participant is described once, then the call is split into 5 to 15 s blocks at natural turn boundaries.
- Each event pairs descriptions of expressions, gestures, and listener reactions with at most one speech entry.
- A rule-based validator checks structure, block timing, language, and absence of capture-device words, with up to three retry attempts.
- Prompt and generation:
- A template converts each validated screenplay block into one text-to-video prompt.
- Reference roles include the previous final frame as starting state, the opening video as picture-quality reference, and the opening audio as voice and room-tone reference.
- Picture content rules enforce front-camera framing, exclude recording devices, UI elements, and subtitles, and prevent content from crossing the center divider.
- Causal order rules prevent a reaction from starting before its trigger finishes.
- Post-generation:
- Generation conditions are preserved as VAE latents for initial video and raw RGB for the final frame.
- The generated video is concatenated and split into user and assistant views, both sharing the same synchronized audio track.
- The opening video is used as a fixed reference to reduce visual error accumulation over long sequences.
Recorded data
The recorded data is organized into six source and interaction categories.
- Single-stream video: solo podcasts and single-person presentations; clips are 10 to 90 seconds and preserve natural pauses.
- Synchronous dual-stream video: both participants are visible at the same time; preserves listening behavior, overlapping speech, and reactions.
- Asynchronous dual-stream video: participants appear in separate clips that alternate with speaking turns.
- Live streams with audience comments: timestamped audience questions are aligned with presenter responses, and user videos are reconstructed into synchronized dual-stream conversations.
- Duplex audio recordings: full-duplex two-person dialogue recordings used to drive further video synthesis.
- Open datasets: single-stream talking videos filtered by common recording standards, with face detection, video understanding, VAD, ASR, and language identification used for further filtering and annotation.
Common recording standards:
- Real speakers with original voices.
- Speaker identities, transcripts, and temporal mappings retained.
- Unrelated audio excluded.
- Continuous, uncut half-body or full-body view.
- Accurate audio-visual synchronization.
- At least 720p effective resolution per view.
- No visual overlays.
- Dual-stream recordings contain at least one complete conversational exchange.
Processing and quality control
Synthetic data quality control:
- First-frame consistency: SSIM at least 0.90 and perceptual-hash Hamming distance at most 4 against the conditioning image.
- Unexpected audio: loudness normalization, waveform residuals, and VAD reject segments with unexplained audio activity lasting at least 0.40 s.
- Camera push-in: face scale sampled at 4 FPS; rejection thresholds include sustained enlargement of 1.30 times baseline for 0.75 s and a 1.22-fold jump within 0.50 s.
- Persistent large-area visual changes: LAB differences, changed-pixel proportions, and color/brightness histogram distances reject abrupt changes and cumulative drift.
Live-stream data processing:
- Collection and initial screening: archived live streams are collected, deduplicated, and checked through sampled frames for real participants and visual quality.
- Annotation and filtering:
- Qwen3-ASR-1.7B and Qwen3-ForcedAligner-0.6B produce timestamped transcripts.
- Silero VAD identifies speech and silence intervals.
- Qwen2.5-Omni-7B reviews audio types.
- FSMN-VAD, Paraformer, and CAM++ perform speaker diarization and speaker-count checks.
- Windows with singing, whispered or breathy speech, or multiple speakers are excluded.
- Dialogue construction:
- Audience comments are matched to presenter responses.
- When matching is not possible, counterpart utterances are generated and scheduled based on the presenter’s original speech and available silence.
- Assistant-side video is cropped around the presenter to remove on-screen comments and watermarks.
- User-side video is synthesized, and both views are paired on a shared timeline.
- Final delivery includes clean dual-stream video with transcripts, dialogue timing, and source metadata.
How the data is used
- Training Duo v1 uses temporally aligned user inputs and assistant audiovisual behavior.
- The unified timeline provides causal supervision for Intent and audiovisual targets for Realization.
- Additional routes support missing-view reconstruction and recorded-video extensions.
- The provided text does not specify explicit train/validation/test split sizes or mixture ratios for synthetic and recorded data.
Method
The authors present Duo v1, a reciprocal world model that couples a language-based Intent Module with a causal audiovisual Realization Module. Refer to the framework diagram.
The interaction is formalized as a chunk-wise, full-duplex process. At each chunk k, the Intent Module conditions on global instructions, user video embeddings, user text, and previous assistant text outputs to generate a text output yk and hidden states hk. These hidden states serve as the interface, providing semantic guidance to the Realization Module. The Realization Module then jointly generates audio and video latents conditioned on a reference image, the intent hidden states, user inputs, and previously generated assistant latents. This formulation ensures causality, as both modules only condition on interaction histories preceding the current chunk, and supports continuity across chunks.
The Intent Module adapts a pretrained multimodal language backbone to causal, chunk-wise interaction. As shown in the figure below:
Inputs enter the context according to when they become available. ASR recovers user text from user audio, and forced alignment assigns each text token a span on the source audio timeline. A token is admitted only after the end of its aligned span. Assistant text targets are assigned to chunks according to speech onset, while the initial chunk uses a fixed <wait> action. The text output contains either <speak> followed by a text body or <wait> without new text, separating the decision to begin speaking from the words of the response.
To support real-time response generation, audio and video are encoded into compact latent representations. For audio, each stereo channel is compressed into 40 latent vectors per second. For video, the encoder processes groups of 17 frames into 5 latent frames. The Realization Module uses a causal audiovisual diffusion transformer initialized with pretrained weights, jointly generating audio and video in these latent spaces through flow matching.
The training proceeds in three stages. As shown in the figure below:
Stage 1 trains the Intent Module for causal participation and text prediction through supervised fine-tuning on chunk-aligned conversations. The model minimizes a weighted autoregressive objective, with action-aware rebalancing to handle chunks that introduce no new assistant text. Stage 2 trains the Realization Module for causal joint audiovisual generation while keeping the Intent Module frozen. The training sequence layout and attention mask encode dependencies among modalities within each chunk and restrict access to preceding interaction history. As shown in the figure below:
The training context contains the reference, shared global instruction, clean media of all chunks, and for each chunk, the Intent Module hidden states followed by noisy media. A causal attention mask ensures noisy media of chunk k can read the reference, shared instruction, clean media of earlier chunks, and hidden states of chunk k, but not the clean media of chunk k or any later chunk. Flow matching supervises the joint media velocity from random noise to clean latent states.
Stage 3 distills the causal model into a four-step chunk-causal generator through Distribution Matching Distillation. As shown in the figure below:
This stage matches the distribution of the generator's own samples to the data distribution using a frozen real-score model and a trainable fake-score model. To align training with the language and media context encountered during inference, a dual rollout strategy is employed. The frozen Intent Module replays recorded assistant text and generates the final assistant turn, while the Realization Module generates the latent window chunk by chunk. The generator is updated by regressing toward a detached target derived from the difference between the fake and real score estimates, with an optional GAN loss adding direct feedback from training media.
Experiment
The evaluation combines a human study of 30 real-time duplex interactions, where model audio-visual replies are judged on six binary criteria, with a showcase of selected interaction examples. The human evaluation validates strong generation fidelity in language, speech intelligibility, and visual identity, and largely consistent cross-modal behavior, while content responsiveness remains the main difficulty because it requires real-time inference of user intent from speech and video together. The showcase complements this by illustrating capabilities such as plan revision, visual guidance, collaborative design, social support, negotiation, explanation, and coaching.