One-sentence Summary

Microsoft Research, the University of Chinese Academy of Sciences, and Shanghai Jiao Tong University present VIBEVOICE-ASR-STREAMING, an LLM-based end-to-end streaming speaker-attributed ASR that interleaves fixed-size audio chunks, a small amount of lookahead audio, and previous text to output "who said what" in real time, achieving the lowest average WER/CER across five evaluation sets and the best or tied-best speaker attribution on 12 of 13 settings, with $1.5\text{B}$1.5B and $7\text{B}$7B model weights and inference code released.

Key Contributions

• Introduces VIBEVOICE-ASR-STREAMING, an LLM-based end-to-end streaming speaker-attributed ASR system that interleaves fixed-size audio chunks, a small amount of lookahead audio, and previous text to generate “who said what” in a single pass without a separate diarization stage.
• The 7B model achieves the lowest average WER/CER across five evaluation sets for recognition-only ASR, and the best or tied-best cpWER/cpCER on 12 of 13 evaluation settings for speaker attribution, including improvements over Azure ConversationTranscriber by 2.39 to 12.45 points on meeting benchmarks and from 27.06 to 22.75 on the MLC-Challenge average.
• Releases the 1.5B and 7B model weights, high-performance inference code with vLLM support, and an online demo, enabling reproducible streaming speaker-attributed ASR.

Introduction

Streaming speaker-attributed ASR must output both words and speaker labels as a conversation unfolds, which is increasingly important for voice agents interacting with multiple people in real time. Prior work falls into three lines: LLM-based recognizers that transcribe long recordings in a single pass but require the full audio before emitting output, streaming LLM-based ASR systems that process audio incrementally with retained context but only support single-speaker transcription, and streaming multi-talker systems that achieve low latency but often rely on separate speaker-related components rather than producing speaker-attributed transcripts directly. The key gap is that speaker attribution places a stronger demand on retained history than ordinary ASR, since a speaker appearing in the current chunk may have first appeared minutes earlier and must keep the same label, yet the streaming Speech-LLM recipe has not been extended to multi-talker settings.

The authors present VIBEVOICE-ASR-STREAMING, which meets both requirements with a single model. They interleave incoming audio chunks with autoregressively generated speaker-attributed text, so future acoustic context is bounded by the chunk contract while accumulated speech, transcription, and speaker history stays in context, eliminating diarization as a separate stage. Each chunk uses a fixed 0.5-second lookahead, and they release 1.5B and 7B model weights for the 22-frame configuration with an expected speaker-attribution latency of 2.00 seconds. Across four meeting conditions and nine languages of MLC-Challenge, the 7B 22-frame configuration achieves the best or tied-best speaker-attributed error on 12 of 13 settings and the best recognition-only mean among compared streaming systems.

Dataset

The authors construct a training mixture from both real and synthetic recordings, all prepared through a unified pipeline. Word-level timing is obtained for every recording using Qwen3-ForcedAligner-0.6B, and the reference transcript is then split into per-chunk targets according to the rule defined in Appendix A.

• Synthetic data generation: To improve robustness to multi-speaker acoustic conditions and specialized vocabulary, the authors synthesize meeting-style conversations. These dialogues include domain-specific terminology and proper nouns, with spoken and written forms kept separate. The spoken form drives speech synthesis, while the written form is retained as the ASR target, ensuring numbers, abbreviations, and technical terms are spoken naturally but transcribed canonically.
• Augmentation: Synthesized speech undergoes waveform-level augmentation. Speakers are overlapped, and the mixture is convolved with room impulse responses, which apply room reverberation and microphone response in a single step. Speaker labels and alignment are updated alongside the waveform so that the supervision survives augmentation.
• Scale: This process yields 50,884 recordings totaling 4,519.6 hours of augmented multi-speaker training speech.
• Usage: The resulting dataset is used as the training mixture for the model, with the per-chunk targets derived from the aligned transcripts serving as supervision.

Method

The authors present VIBEVOICE-ASR-STREAMING, an architecture designed to extend long-form speaker-attributed transcription to streaming inference. The system is built upon a Qwen2.5 large language model backbone. Speech input is processed by the encoder halves of pre-trained dual tokenizers. The Acoustic tokenizer applies a hierarchical, cumulative downsampling to the waveform to capture spectral details, while the Semantic tokenizer operates at the same rate to yield deterministic features aligned with textual content. These representations are concatenated along the feature dimension and projected into the embedding space of the language model.

As shown in the figure below:

To enable streaming capabilities, the authors organize incoming speech and generated text as an interleaved sequence:

where $X_k$Xk​ denotes the $k$k-th speech chunk and $Y_k$Yk​ denotes the corresponding speaker-attributed transcription. Unlike independent chunk-wise decoding, previously observed speech and generated text remain in the context, allowing each chunk to be decoded against the accumulated conversation history. To provide limited future acoustic evidence near chunk boundaries, the model incorporates a fixed lookahead of $L = 4$L=4 latent frames, which corresponds to approximately 0.5 seconds of audio. The authors evaluate two chunk configurations: 15 latent frames (2.0 seconds) and 22 latent frames (2.9 seconds).

The transcription task is formulated as a single autoregressive generation process. For the text sequence $Y_k = (y_{k,1}, \dots, y_{k,N_k})$Yk​=(yk,1​,…,yk,Nk​​) associated with the current chunk and its lookahead $\tilde{X}_k$X~k​, the generation probability is defined as:

Each generated text chunk concludes with a special end-of-chunk token, which hands control back to the audio stream. The model also supports optional contextual prompting, allowing hotwords and technical terms to be provided before decoding. Speaker identities are assigned ordinal labels based on their order of first appearance.

The training process is structured into three distinct stages that vary in sample construction rather than model architecture or objectives. In the first stage, the model undergoes non-streaming training in an offline setting where the complete recording is visible, establishing foundational multi-speaker recognition and attribution abilities. The second stage transitions to streaming pre-training. Starting from the initial checkpoint, training samples are segmented into chunks and paired with their transcriptions using the interleaved formulation with lookahead. This stage utilizes a large corpus to adapt the model to bounded future context. The final stage involves streaming fine-tuning on a smaller, curated mixture of public and synthetic multi-speaker data. This stage refines the model behavior regarding transcription conventions, consistent speaker labeling, and reliable hotword following. During data preparation for the synthetic mixture, the authors generate meeting-style conversations with domain-specific terminology, applying waveform-level augmentations such as speaker overlapping and convolution with room impulse responses to ensure robust supervision.

Experiment

The evaluation compares VIBEVOICE-ASR-STREAMING against cloud baselines across meeting, conversational, and single-speaker benchmarks, showing it leads on most speaker-attributed settings with much lower labeling latency. Ablations reveal that larger backbones and longer chunks improve both transcription and especially speaker attribution, while head-placed speaker labels match tail placement in accuracy, and increasing lookahead depth consistently helps. The system maintains a low real-time factor, confirming its streaming feasibility.

VIBEVOICE-ASR-STREAMING matches or outperforms commercial streaming APIs on several meeting benchmarks, especially for English and Korean, while showing larger gaps on languages like Japanese and German. The 7B model with 22-frame chunks is the primary configuration, and both larger backbone scale and longer chunks improve accuracy more for speaker-attributed metrics than for plain recognition. VIBEVOICE-ASR-STREAMING achieves the best or near-best WER/CER on English and Korean, with values comparable to the top commercial API. On French, German, and Italian, VIBEVOICE-ASR-STREAMING outperforms all listed commercial APIs, while on Japanese it trails the best API by a notable margin. Increasing backbone scale from 1.5B to 7B reduces mean cpWER/cpCER by over 12 points, a larger gain than the roughly 4.5-point improvement in WER/CER. Reducing chunk size from 22 to 15 frames lowers expected latency from 2.00s to 1.53s, trading some accuracy for speed. Longer lookahead consistently improves both WER/CER and cpWER/cpCER, with speaker attribution benefiting more from additional right context.

The table compares streaming speaker-attributed ASR performance across multiple languages and systems, showing that the proposed 7B model with 22-frame chunks achieves the best or second-best results on most metrics while maintaining a much lower average latency than commercial alternatives. Google STT's retroactive speaker label revision substantially improves its cpWER, but the proposed model's end-to-end design achieves competitive accuracy without such post-processing. The proposed model reports the lowest average latency at 2.00 seconds, versus 8.21 seconds for Azure CT and 9.12 or 51.06 seconds for Google STT depending on label timing. On English MLC-Challenge, the proposed model achieves lower WER (8.44) and cpWER (11.99) than both Azure CT and Google STT. For Japanese, the proposed model's cpCER matches Google STT's final-label result (33.01) despite using only first-emission labels. Google STT's cpWER improves dramatically when using final revised labels instead of first-emission labels, indicating a large retroactive correction effect. The proposed model shows strong results across languages, often ranking first or second on both recognition and speaker-attribution metrics.

The 7B model with 22-frame chunks achieves the best average single-speaker recognition across the four benchmarks, outperforming the 15-frame variant and all compared systems. The 22-frame setting consistently lowers errors on most datasets, while the 15-frame variant trades a small accuracy loss for reduced latency. The 7B model with 22-frame chunks has the lowest average error (5.76) among all systems. Increasing chunk size from 15 to 22 frames improves average recognition by 0.61 points. The 22-frame variant outperforms the 15-frame one on every individual benchmark except GigaSpeech, where the difference is small. Compared to other real-time systems, the 22-frame model ranks best on AISHELL-1 and LibriSpeech test-clean, and second-best on GigaSpeech.

Streaming conversion from a non-streaming checkpoint consistently degrades both transcription and speaker attribution metrics across all benchmarks, with speaker attribution suffering larger degradations than word error rates. The degradation pattern suggests that the streaming setup primarily loses speaker evidence rather than lexical information. WER and cpWER both increase on every benchmark, with cpWER rising by 5.13 to 6.67 points versus 0.75 to 3.53 for WER. The larger cpWER degradation relative to WER indicates an additional accuracy loss tied to speaker assignment, not just transcription. The streaming model's performance gap is consistent across Chinese, English, and mixed-language datasets, with the smallest WER increase on AMI-IHM and the largest on AMI-SDM.

Under a fixed 4-frame lookahead, increasing the chunk size from 15 to 22 latent frames consistently improves both WER/CER and cpWER/cpCER across all benchmarks and model scales, with larger gains in speaker attribution metrics. Similarly, scaling from 1.5B to 7B parameters yields larger improvements in cpWER/cpCER than in WER/CER, indicating that both longer chunks and larger models primarily enhance speaker evidence rather than lexical accuracy. The 15-frame setting offers lower latency (1.53 s vs 2.00 s) at the cost of accuracy, presenting a trade-off between speed and quality. Enlarging the chunk from 15 to 22 frames improves both metrics on every benchmark and at both model scales, with the five-set mean gain being larger for cpWER/cpCER than for WER/CER. At a fixed chunk size, moving from 1.5B to 7B parameters reduces mean cpWER/cpCER by 12.76 points at 22 frames and 11.61 at 15, versus 4.69 and 4.54 for WER/CER, showing a stronger effect on speaker attribution. The 15-frame chunk cuts expected latency from 2.00 s to 1.53 s, but the accuracy loss is consistent across all benchmarks, making the choice between chunk sizes a latency-accuracy trade-off.

The evaluation compares VIBEVOICE-ASR-STREAMING against commercial APIs and ablations across multilingual meeting benchmarks. The 7B model with 22-frame chunks matches or outperforms commercial systems on English and Korean, leads on French, German, and Italian, but trails notably on Japanese. Larger backbone scale and longer chunks consistently improve accuracy, with gains twice as large for speaker-attributed metrics (cpWER/cpCER) than for plain recognition, indicating that both factors primarily enhance speaker evidence rather than lexical transcription. Reducing chunk size from 22 to 15 frames lowers latency from 2.00s to 1.53s at a modest accuracy cost, while streaming conversion from a non-streaming checkpoint degrades speaker attribution more than word error rates, confirming a specific loss in speaker assignment rather than lexical content.