One-sentence Summary

Researchers at Carnegie Mellon University, Shanghai Jiaotong University, and Toyota Technological Institute at Chicago propose XEUS, a cross-lingual speech encoder trained on over 1 million hours of speech across 4,057 languages, which augments SSL masked prediction with a novel dereverberation objective to improve robustness and achieves state-of-the-art results on the ML-SUPERB benchmark.

Key Contributions

• The paper introduces XEUS, a cross-lingual speech self-supervised encoder trained on more than 1 million hours across 4057 languages, and releases a new 7413-hour unlabeled corpus to extend model language coverage by about four times.
• This work augments standard self-supervised masked prediction with a novel joint dereverberation objective, giving the encoder stronger robustness to reverberant and noisy multilingual speech conditions.
• Experiments show that XEUS matches or surpasses strong self-supervised baselines across speech tasks and sets a new ML-SUPERB result, exceeding MMS 1B and w2v-BERT 2.0 v2 by 0.8% and 4.4%, respectively, while releasing code, data, checkpoints, logs, and intermediate checkpoints.

Introduction

Speech processing models still cover only a small fraction of the world’s languages because transcribed speech is scarce, and many low-resource recordings are noisy or acoustically diverse. Self-supervised learning can reduce dependence on labels, but existing multilingual models often cover only 50 to 150 languages, rely on closed crawled data, and remain vulnerable to reverberation and speaking-style variation even when they simulate noise. The authors leverage these gaps by introducing XEUS, a crosslingual speech encoder pretrained on more than 1 million hours of public audio and a new public corpus with 7,413 hours across 4,057 languages. They also add an acoustic dereverberation objective alongside masked prediction and denoising, yielding a more robust universal encoder and releasing code, checkpoints, and training logs for reproducible large-scale multilingual speech research.

Dataset

• The authors build a large unlabeled pretraining corpus for speech self-supervised learning by combining 37 publicly available speech datasets with three newly collected datasets.

• Existing datasets:

  • They use only the official training splits of each public dataset to avoid training on corrupted or leaked test data.
  • These public sources cover 150+ languages and contribute 1.074 million hours of audio.
  • Notable content includes accented speech, code-switching, and singing voices that were not covered in earlier self-supervised speech models.

• MMS-unlab v2:

  • The authors recreate an earlier unreleased MMS-unlab dataset by crawling religious audiobooks from the Global Recordings Network.
  • Unlike the original, they expand coverage to more than 200 additional languages.
  • Because the data is used for self-supervised learning rather than language identification, they do not filter out languages with very little audio.
  • They use an energy-based voice activity detector to segment long recordings, rather than a neural VAD model, for efficiency and robustness across many languages.
  • This subset contains 6,700 hours across 4,023 ISO3 languages, with wider language coverage than the original.
  • The authors obtained explicit written permission to use and redistribute this data because the source site did not clearly state licensing terms.

• WikiTongues:

  • They create a new corpus from WikiTongues, a grassroots recording project aimed at preserving low-resource, endangered, and extinct languages and dialects.
  • Recordings are 2-20 minutes long and typically contain 1-2 speakers casually speaking a target language or dialect.
  • The data is released under a CC-BY-NC license.
  • The collected subset includes around 700 languages or dialects, 821 recordings, and about 70 hours of speech.
  • The authors use the same energy-based VAD settings as MMS-unlab v2 to segment the recordings into shorter clips.

• Jesus Dramas:

  • They collect a second new corpus from Inspirational Films, which released the “Story of Jesus” audio drama in 430 languages under a non-commercial license.
  • Each drama is a 90-minute multi-speaker audio recording.
  • The full subset totals 645 hours of speech.
  • The same VAD settings are used to segment the dramas into utterance-level clips.

• Final corpus and usage:

  • The three newly collected subsets total 7,413 hours across 4,057 ISO3 languages.
  • When combined with the public datasets, the full pretraining corpus reaches 1.081 million hours of audio.
  • The authors filter out all utterances longer than 40 seconds due to memory constraints.
  • The final data spans 189 language families and follows a long-tailed language distribution: the top 50 languages account for 99.5% of the data, while about 2,000 languages have at least one hour each.
  • Data from YODAS is excluded from some language analyses because its language labels are too noisy.
  • Metadata and licensing information are tracked for the corpora, and language coverage is analyzed at the ISO3 level for the collected and public subsets.

Method

The authors design XEUS as a masked prediction pre-training pipeline that combines phonetic target learning with acoustic enhancement tasks. The framework contains a teacher branch that produces clean targets, a corruption branch that modifies the input waveform, and a student branch that predicts the clean targets from the corrupted input. As shown in the framework diagram:

The target generation process uses a pre-trained WavLabLM MS model as a teacher. The authors leverage this teacher to extract encoded representations from clean speech, then cluster those representations with k-means using 2048 clusters. The resulting discrete codes serve as phonetic pseudo-labels. These targets are derived only from uncorrupted speech, so the student must infer the same clean targets even when the input waveform has been distorted.

The first corruption mechanism is masked prediction. Random masking is applied to the waveform before it enters the student encoder, forcing the model to recover missing target information from the surrounding acoustic context. The second mechanism is acoustic denoising. With probability $p$p, an input utterance is augmented with either random noise or another utterance from the same batch used as interference. Because the pseudo-labels remain tied to the original clean speech, the student is encouraged to remove the added acoustic interference internally.

The authors also introduce a dereverberation objective. With probability $p_r$pr​, a selected utterance is augmented with a sampled room impulse response. Let $u$u denote the original utterance and $u_n$un​ the sampled room impulse response. The method first estimates a sample shift $d_t$dt​ from the highest peak in the impulse response. The reverberant signal is then simulated by convolution:

After convolution, $r$r is realigned with the original utterance using $d_t$dt​ and normalized to match the energy of $u$u. This realignment step is critical because, without it, the augmented audio would shift relative to the clean pseudo-labels and create label alignment errors. If both noise and reverberation augmentation are applied to the same utterance, noise augmentation is applied first.

For the student model, the authors build XEUS on a HuBERT-style architecture but replace the original Transformer with an E-Branchformer backbone. They choose this design because convolution-augmented models provide stronger self-supervised representation learning, while the E-Branchformer offers practical training advantages over the Conformer. The architecture consists of a convolutional feature extractor followed by 19 E-Branchformer layers. Each layer contains 8 attention heads, a hidden dimension of 1,024, a feed-forward size of 4,096, and a convolution kernel size of 31. The full model has 577M parameters. The training objective uses cross-entropy loss between the predicted pseudo-labels and the clean teacher-derived targets.

The pre-training process is carried out with the ESPnet toolkit on 64 NVIDIA A100 GPUs. Each GPU uses a maximum batch size of 100 seconds, yielding a total batch size of 106 minutes. The noise augmentation probability is set to $p = 0.2$p=0.2, with equal probability between random noise and another batch utterance as interference. The dereverberation augmentation probability is set to $p_r = 0.3$pr​=0.3. The model is trained for two passes through the data, corresponding to 670K total optimization steps. The authors also optimize the open-source ESPnet infrastructure to support this large-scale training workflow.

Experiment

The downstream experiments evaluate XEUS on multilingual speech processing, task universality, and acoustic representation quality, using ML-SUPERB, FLEURS, low resource speech translation, the English SUPERB benchmark, and speech resynthesis. The results show that XEUS is broadly competitive or superior to strong multilingual and English-only SSL baselines, with particular strength across many languages and low resource settings, suggesting broad coverage and efficient use of training data. It also yields strong universal speech representations and high quality generative acoustic features, supporting both recognition and synthesis applications.

The comparison shows that broad language coverage, large training-data scale, and open release practices rarely coincide across multilingual speech SSL models. Models with wider language coverage or larger data budgets often provide fewer public components, such as data, weights, training code, or artifacts, while more transparent models tend to operate at smaller scale. The described XEUS setup is presented as an effort to combine broad language coverage with public data, code, checkpoints, logs, and intermediate training artifacts. Transparency is often partial, with models releasing only some combination of data, weights, code, and artifacts. Models with broader language coverage or larger data budgets tend to be less openly released than smaller, more transparent models.

The pre-training data combine many publicly available speech datasets to create a broad multilingual mixture spanning diverse domains and audio types. The mixture includes extensive general speech, legal speech, audiobook speech, and several large language groups such as English, Mandarin, and Russian. It also incorporates content types such as accented speech, codeswitching, and singing voices. The dataset mix emphasizes wide language coverage rather than a single language or domain. YODAS and VoxPopuli appear among the largest contributors, showing a dependence on very large curated speech sources. The corpus includes both multilingual coverage and focused language datasets such as English, Mandarin, and Russian. Some listed datasets are new corpora intended for release.

XEUS is the best overall model on the multilingual speech processing benchmark, achieving the highest aggregate scores in both data-limited and more data-rich settings. It leads on monolingual speech recognition and normal-setting multilingual ASR with LID, and it outperforms the parameter-equivalent w2v-BERT 2.0 v2 across task categories while using much less accessible training data. MMS remains stronger on some multilingual ASR and few-shot ASR settings, but XEUS still surpasses most other self-supervised baselines. XEUS ranks first on the overall multilingual benchmark in both data settings. XEUS achieves the best monolingual ASR performance on both tracks. XEUS is the best model for normal multilingual ASR with LID in both tracks. MMS outperforms XEUS on some multilingual ASR and few-shot ASR results. XEUS outperforms w2v-BERT 2.0 v2 across task categories with much smaller accessible training data.

XEUS performs competitively on FLEURS multilingual ASR, outperforming XLS-R and MMS in recognition error and coming close to w2v-BERT. It also gives the strongest character-level quality on Jesus Film English speech translation among the compared models, consistent with its broader language coverage. XEUS is competitive with w2v-BERT on FLEURS while using less training data. XEUS outperforms XLS-R and MMS on FLEURS recognition error. XEUS leads all compared models on Jesus Film speech translation quality.

On the English-only SUPERB evaluation, XEUS is competitive with WavLM Large across recognition, detection, semantics, speaker, and paralinguistic tasks. It obtains the best scores on several tasks, including automatic speech recognition, keyword spotting, speaker diarization, and emotion recognition. The comparison suggests that XEUS transfers well across diverse speech tasks, although WavLM remains stronger on several remaining metrics. XEUS reaches the best result in four evaluated tasks: ASR, keyword spotting, speaker diarization, and emotion recognition. WavLM Large still outperforms XEUS on several remaining tasks, especially some semantics and speaker tasks.

Experiments evaluate XEUS’s transparency, pretraining data construction, and performance on multilingual and English-only speech benchmarks. The analyses show that XEUS addresses the common tradeoff between broad language coverage, large data budgets, and open release by combining extensive multilingual and diverse-domain speech with public components, whereas comparable models often lack one or more of these properties. Across benchmarks, XEUS is the strongest overall multilingual model in both data settings, particularly for monolingual and normal multilingual ASR with LID, and it transfers well to diverse English-only tasks, although some existing baselines remain stronger in specific areas.