One-sentence Summary

KAIST introduces a Face-to-Face spoken dialogue model and MultiDialog, the first large-scale multimodal audio-visual dialogue corpus of 340 hours, which directly processes and generates audio-visual speech without intermediate text by adapting a pretrained LLM through speech-text joint pretraining, advancing end-to-end avatar chatbot systems.

Key Contributions

• MultiDialog is introduced as a large-scale multimodal spoken dialogue corpus with 340 hours and approximately 9,000 TopicalChat-based audio-visual dialogues, including emotion annotations. This dataset supports research on audio-visual spoken dialogue synthesis.
• A direct Face-to-Face spoken dialogue model processes audio-visual user speech and generates audio-visual speech responses without relying on intermediate text during generation.
• Experiments show that joint speech-text pretraining built on a textually pretrained large language model retains more original model knowledge than direct initialization and supports effective face-to-face conversation.

Introduction

Spoken dialogue systems are important for voice assistants and customer service, but natural human conversation also relies on visual cues such as facial expressions and gestures, which can improve understanding and engagement. Prior datasets and models mostly cover text or audio-only dialogue, with audio-visual spoken dialogue corpora limited in scale and existing multimodal LLMs typically using vision for auxiliary tasks rather than generating synchronized talking-face speech. The authors introduce MultiDialog, a 340-hour audio-visual corpus of about 9,000 real-conversation-derived dialogues, and leverage it to train a direct face-to-face spoken dialogue model that processes audio-visual speech input and generates audio-visual speech output. They use quantized audio-visual speech tokens from self-supervised features as pseudo-text units and jointly pretrain a large language model with speech and text, improving retention of the original LLM knowledge compared with direct initialization.

Dataset

• Dataset composition and sources:

  • The authors construct MultiDialog from dialogue scripts derived from TopicalChat, an open-domain, knowledge-grounded dialogue dataset collected from real human-human conversations.
  • The source data spans eight broad topics: fashion, politics, books, sports, general entertainment, music, science and technology, and movies.
  • The source dialogue data includes annotations for eight emotions: disgusted, angry, fearful, happy, sad, surprised, neutral, and curious to dive deeper.

• Recording setup and participant details:

  • The authors record 12 fluent English speakers, including 6 male and 6 female participants aged 20 to 30 from six countries.
  • Recordings are made in a professional studio with a green screen and minimal background noise.
  • Two conversation partners sit side by side and are captured with separate cameras and microphones.
  • Camera framing is adjusted to each participant’s height and records the upper body from the shoulders, providing a crop focused on facial and upper-body behavior.

• Audio-video schema and turn metadata:

  • Audio is recorded in mono WAV format at 48 kHz.
  • Video is recorded in full HD at 30 fps.
  • Participants press a button to move to the next utterance, which records the start and end time of each turn.
  • Turn-level timestamps are used in post-processing to segment recordings into conversations and utterances.

• Emotion and speaker construction details:

  • Actors follow scripts that specify the desired emotion for each utterance.
  • The authors provide visual and vocal instructions based on the Facial Action Coding System, plus sample images for each emotion.
  • Participants are also instructed to react naturally while listening, adding non-speaker behavior to the dialogue.
  • The original “Neutral” and “Curious to dive deeper” emotion labels are merged into a single “Neutral” label because the authors found little visual difference between them.

• Filtering, alignment, and subset selection:

  • An annotator reviews recordings to correct audio-video misalignments by adjusting turn start times.
  • Recordings missing either the audio or video stream are filtered out.
  • The authors use the recorded turn boundaries to segment the data into structured dialogues.
  • The final MultiDialog dataset contains about 340 hours of audio-video dialogue across roughly 9,000 dialogues involving 6 pairs of conversation partners.
  • The authors also release a gold emotion dialogue subset selected through rigorous annotation evaluation.

Method

The authors build an audio-visual spoken dialogue system that takes a user’s face video and audio stream as input and generates a response as a talking-face video with synchronized audio. The overall pipeline decomposes this task into three coupled modules: an audio-visual speech tokenizer, a multimodal spoken dialogue language model, and an audio-visual response generator. The tokenizer converts speech into compact discrete symbols, the language model treats those symbols as pseudo text for conversational modeling, and the generator projects the predicted symbols back into audible speech and visual facial motion. As shown in the framework diagram:

The first module is an audio-visual speech encoder that transforms the user’s raw face video and speech signal into discrete AV speech tokens. The authors leverage AV-HuBERT, a self-supervised audiovisual speech model that jointly observes lip motion and audio features. In practice, the encoder focuses on the mouth region while using the synchronized audio track, extracts audiovisual representations, and quantizes them into a discrete cluster sequence. These AV speech tokens are designed to retain both linguistic and phonetic content, while visual mouth movements provide complementary cues that improve robustness in noisy conditions. Because the resulting token sequence can be viewed as a symbolic representation of speech, it can be consumed by a language model in the same way as text tokens.

The core reasoning component is an audio-visual spoken dialogue language model. It is initialized from a pretrained large language model, OPT-1.3B, and its vocabulary is extended to include AV speech tokens alongside the original text vocabulary. This unified vocabulary allows the model to learn a single next-token distribution over both text tokens and AV speech tokens. If the full token sequence has length $N$N, the generic language modeling objective is:

where $t_i$ti​ denotes the $i$i-th token in either text or AV speech. The goal is to make the original text-only LLM learn to understand AV speech tokens and also generate relevant AV speech responses.

To adapt the text-based LLM to AV speech tokens without losing its language knowledge, the authors introduce a joint speech-text pretraining scheme. Training dialogues are organized as sequences of user and AI turns, with each utterance represented in two synchronized forms: a text-token sequence and an AV speech-token sequence. Modality markers indicate the beginning of an AV speech stream or a text stream. Later, speaker markers distinguish the User and the AI. These marked sequences define the training objectives for both token understanding and dialogue generation. As shown in the training data diagram:

The first pretraining stage teaches the LLM to interpret and generate AV speech tokens using paired turn examples. For each dialogue turn, the model sees a text stream and an AV speech stream, each introduced by a modality prefix. Two complementary next-token objectives are constructed. One AV speech-text objective is defined as:

and the paired text-AV objective is defined as:

where $T_{\mathrm{AV}}$TAV​ denotes the AV speech-token sequence and $T_{\mathrm{Text}}$TText​ denotes the text-token sequence. During this stage, the authors freeze most of the pretrained LLM and train only the embedding layer and projection layer. This design allows the new AV speech tokens to enter the model’s representation space while preserving the original LLM parameters that encode broad language and dialogue knowledge.

The second pretraining stage shifts from paired utterance modeling to full dialogue modeling. The training sequences include alternating User and AI turns, with additional speaker prefix tokens such as <User> and <AI>, followed by modality prefix tokens such as <Speech> and <Text>. This creates a dialogue context in which the model must predict the AI response given prior user and AI turns. If a dialogue contains $K$K rounds and the $k$k-th AI response contains $N_k$Nk​ tokens, the dialogue language modeling loss is:

where $T_k^{ai,n}$Tkai,n​ is the $n$n-th AI token in round $k$k, $T_k^{ai,\lt n}$Tkai,<n​ denotes earlier AI tokens in the same response, and $T_{\lt k}$T<k​ contains all earlier dialogue context. During this stage, the model is trained on a balanced mixture of text-based and AV speech-token-based dialogues. This gradual shift helps the model learn to operate over AV speech tokens while maintaining the dialogue quality inherited from the text LLM. After mixed pretraining, the system is further adapted using AV speech token dialogue sequences, enabling realtime face-to-face interaction using only discrete audiovisual speech representations.

The final module projects the generated AV speech tokens back into audio and visual modalities. Since the language model operates on a compact, duplicate-reduced AV speech token sequence, the generator first uses a length predictor to restore the temporal length needed for faithful speech synthesis. The restored token sequence is then passed to a token-based speech decoder, which converts AV speech tokens into an audible waveform. In parallel, a token-based face decoder generates facial motion and appearance from the same AV speech tokens. These decoders are adapted from pretrained audio and talking-face generators, but they are trained to consume AV speech tokens rather than raw audio.

To control speaker identity and visual consistency, the generator incorporates target identity information. A speaker embedding extracted from a target identity audio sample conditions the synthesized voice, while the target identity’s face and pose priors guide the face generator. In the face decoder path, an adapter or reprogramming bridge can map AV speech tokens into feature representations compatible with a pretrained talking-face generator, allowing the system to reuse strong visual generation capabilities without modifying the entire face generator. A final face enhancer can then upsample the generated facial frames for higher visual quality. The result is a complete closed-loop audio-visual spoken dialogue pipeline: user face video is tokenized, the dialogue LM produces a tokenized AI response, and the response is rendered as both audio and a talking-face video.

Experiment

The experiments evaluate generated audio-visual dialogue responses on both semantic content and generation quality, using transcribed text for dialogue coherence, visual quality, audio-visual synchronization, and speaker similarity, and compare against audio-only spoken dialogue systems and cascaded pipelines. The main model achieves contextually coherent, diverse responses without intermediate text generation, and the pretraining study shows that initializing from a text language model and gradually aligning it with audio-visual speech tokens improves semantic performance. The results also indicate that the proposed speech and face generation pipeline preserves speaker identity well and aligns audio with video effectively, while adding visual input makes the system more robust to acoustic noise.

MultiDialog is compared with prior spoken and multimodal dialogue datasets by scale and available signal types. It provides the largest collection of dialogues, turns, and total hours among the compared datasets while also including audio, text, video, and emotion annotations. This combination supports research on large-scale, tri-modal dialogue systems and expressive multimodal conversation modeling. MultiDialog is larger in dialogue count, turn count, and total duration than the other listed datasets. It supports audio, text, video, and emotion labels, making it broader than datasets that contain only one or two modalities or lack emotion annotations.

The MultiDialog dataset contains roughly 340 hours of audio-visual spoken dialogue, organized into a large training portion and smaller validation and test portions for frequent and rare evaluation. Across splits, conversations have broadly similar turn counts and durations, averaging about 21 utterances and a few minutes per dialogue. These statistics support evaluating multimodal dialogue models on both common and less frequent cases while using consistent dialogue-length distributions. The training split accounts for the majority of dialogues, utterances, and total video duration. Validation and test splits have comparable dialogue counts, enabling evaluation under frequent and rare conditions. Average dialogue and utterance durations remain similar across splits, suggesting consistent post-processed segmentation.

The proposed method compares most favorably on semantic coherence and diversity among the compared spoken dialogue systems, especially on BLEU and unigram and bigram diversity. It is also distinguished by accepting audio-visual input and producing audio-visual output without using intermediate text. Pretraining analysis suggests textual language model initialization and gradual adaptation of audio-visual speech tokens improve semantic quality, with mixed text and audio-visual token pretraining giving overall gains despite slightly worse perplexity. The proposed method ranks best on BLEU and diversity metrics relative to the other spoken dialogue systems It is the only compared method that supports both audio and visual inputs and outputs without intermediate text Gradual text and audio-visual token pretraining improves semantic quality, while slightly increasing perplexity

The proposed audio-visual spoken dialogue system shows the strongest speaker voice similarity among the compared methods, outperforming both the cascaded system and audio-only dialogue systems. Its visual quality is close to that of the cascaded system using the same talking-face generator, while its audio-visual synchronization is stronger, supported by better lip-sync metrics. These results suggest that using discrete audio-visual tokens helps preserve speaker information and align speech with facial movements more effectively than raw-audio cascaded generation. The proposed method achieves the highest speaker similarity score among the evaluated systems. Its face-generation quality is comparable to the cascaded baseline with the same visual generator. It produces better audio-visual synchronization than the cascaded system, consistent with its higher confidence and lower error lip-sync scores.

The comparison shows that adding visual information to audio input improves dialogue response generation measured by BLEU. The audio-visual setting outperforms audio-only input across noisy and clean conditions, supporting stronger robustness when speech is corrupted by acoustic noise. This suggests that visual cues can help recover information missing from degraded audio. Audio-visual input achieves higher BLEU scores than audio-only input at every tested SNR level. The advantage of visual input is clearer under noisy conditions than under clean input. Visual information helps compensate for speech content lost due to acoustic corruption.

Experiments compare the MultiDialog dataset with existing spoken and multimodal dialogue datasets and evaluate the proposed audio-visual dialogue system. The dataset analysis shows that MultiDialog offers broader coverage in scale, modalities, and annotations while maintaining consistent dialogue distributions across train, validation, and test splits. The system evaluation indicates that direct audio-visual modeling improves semantic coherence and response diversity relative to prior spoken dialogue systems, preserves speaker identity and lip-sync quality better than cascaded generation, and leverages visual input to enhance robustness under noisy conditions.