One-sentence Summary

Researchers from Google and Tokyo University of Agriculture & Technology introduce LibriTTS-R, a restored $585\text{-hour}$585-hour, $24\,\text{kHz}$24kHz multi-speaker TTS corpus from $2456$2456 speakers, created by applying speech restoration to LibriTTS while keeping identical samples, which significantly improves sound quality over LibriTTS and enables neural end-to-end TTS with naturalness on par with ground truth, freely available at openslr.org/141.

Key Contributions

• Introduces LibriTTS-R, a quality-improved speech dataset created by applying the text-informed speech restoration model Miipher, which uses a w2v-BERT feature cleaner and Wave-Fit neural vocoder, to the 585-hour LibriTTS corpus while preserving identical samples and texts.
• Subjective experiments show that LibriTTS-R ground-truth samples have significantly improved sound quality over LibriTTS, and that a neural end-to-end TTS model trained on LibriTTS-R achieves speech naturalness comparable to that of the ground-truth samples.
• Releases the corpus publicly under a non-restrictive license at http://www.openslr.org/141/, making it freely downloadable to support reproducible TTS research.

Introduction

Text-to-speech (TTS) systems have made significant strides with deep learning, enabling near-natural reading-style synthesis when trained on studio-quality recordings. However, a major bottleneck persists: the lack of large, publicly available, high-quality datasets. Many state-of-the-art TTS models rely on over 100 hours of proprietary studio data, which is not openly accessible, hindering reproducibility and broader research progress. Concurrently, speech restoration (SR) techniques have advanced, using generative models to convert degraded audio, such as reverberated lectures or historical recordings, into studio-grade quality.

The authors address this data scarcity by applying SR to an existing public corpus. They introduce LibriTTS-R, a quality-improved version of the LibriTTS dataset, which contains 585 hours of speech from 2,456 speakers. To clean the original recordings, they leverage a text-informed SR model called Miipher, which combines a w2v-BERT feature cleaner with a Wave-Fit neural vocoder. Their subjective evaluations demonstrate that a TTS model trained on LibriTTS-R achieves significantly better speech naturalness than one trained on the original LibriTTS, reaching a quality comparable to ground-truth recordings. The corpus is released under a non-restrictive license, providing a valuable public resource to accelerate TTS research.

Dataset

Dataset Composition and Sources

The authors build their work on the LibriTTS corpus, a large-scale, multi-speaker speech dataset designed for text-to-speech (TTS) research. It contains 585 hours of speech at a 24 kHz sampling rate from 2,456 speakers, along with corresponding text. The audio and text are re-derived from the original LibriSpeech materials, specifically MP3 files from LibriVox and texts from Project Gutenberg, to fix issues like sampling rate and text normalization that were present in the original corpus.

Key Details for Each Subset

• LibriTTS (base dataset): 585 hours, 2,456 speakers, 24 kHz. It is used as the starting point for the authors' data processing pipeline. The authors note that LibriTTS has lower sound quality compared to smaller, high-quality datasets like LJspeech, which affects the quality of TTS model outputs.
• Proprietary training dataset for speech restoration: 2,680 hours of paired noisy and studio-quality speech. The target speech includes 670 hours of studio-recorded English from Australian, British, Indian, Nigerian, and North American speakers at 24 kHz.
• Noise sources: The TAU Urban Audio-Visual Scenes 2021 dataset, internally collected noise snippets (simulating cafe, kitchen, car conditions), and other noise sources.
• Augmentation patterns: Noisy utterances are generated by mixing speech and noise with signal-to-noise ratios (SNR) from 5 dB to 30 dB. Four patterns are created based on the presence or absence of reverberation and codec artifacts. Room impulse responses (RIR) are generated stochastically using the image method. Codec artifacts are simulated by randomly applying MP3, Vorbis, A-law, Adaptive Multi-Rate Wideband (AMR-WB), or OPUS with random bit-rates.

How the Data Is Used

• The authors apply a speech restoration (SR) model to LibriTTS to create a public, large-scale, high-quality TTS dataset.
• For training the SR model, they use the proprietary paired dataset described above. The feature-cleaner is pre-trained for 150k steps, and the WaveFit neural vocoder is pre-trained for 1M steps to reconstruct waveforms from clean w2v-BERT features. Then, the WaveFit vocoder is fine-tuned for 350k steps using cleaned features from the pre-trained feature-cleaner.

Processing and Metadata Construction

• The noisy dataset is generated by mixing speech and noise with specified SNR ranges.
• Reverberation is added using stochastic RIR generation via the image method.
• Codec artifacts are introduced by randomly selecting a codec and bit-rate.
• No additional metadata construction is described beyond the pairing of noisy and clean speech samples for training the restoration model.

Method

The authors leverage a text-informed parametric re-synthesis-based Speech Restoration (SR) model, Miipher, to clean noisy TTS datasets and ensure robustness against artifacts. This approach specifically addresses challenging degradation patterns such as phoneme masking and phoneme deletion. The overall data processing pipeline is shown in the figure below:

In this framework, the model first extracts w2v-BERT features from the noisy waveform. Instead of relying on conventional log-mel spectrograms, the use of w2v-BERT features, which are trained on large amounts of degraded speech, enhances the model robustness. To further improve restoration performance, the system incorporates linguistic features conditioned on the transcript. Specifically, PnG-BERT extracts these linguistic features from the corresponding text. A DF-Conformer-based feature-cleaner then predicts the w2v-BERT features of the clean waveform. Finally, a WaveFit-5 neural vocoder synthesizes the restored waveform.

For the training process, the authors utilize a proprietary dataset comprising 2,680 hours of noisy and studio-quality speech pairs. The target dataset includes 670 hours of studio-recorded English at a 24 kHz sampling rate. Noisy utterances are generated by mixing speech with various noise sources at signal-to-noise ratios ranging from 5 dB to 30 dB. The dataset is further augmented with reverberation, simulated via a stochastic room impulse response generator, and codec artifacts introduced by randomly applying formats like MP3, Vorbis, and OPUS at varying bit-rates.

The training procedure follows a multi-stage approach. First, the feature-cleaner and the WaveFit neural vocoder are pre-trained for 150,000 and 1,000,000 steps, respectively. During this phase, WaveFit is trained to reconstruct waveforms directly from clean w2v-BERT features. Subsequently, the WaveFit vocoder is fine-tuned for 350,000 steps using the cleaned w2v-BERT features generated by the pre-trained feature-cleaner.

During the inference pipeline, the system calculates PnG-BERT features from the transcript and a speaker embedding derived from the original 24 kHz waveform via a speaker encoder. For audio samples shorter than 2 seconds, the waveform is repeated to compute a stable speaker embedding. Since the w2v-BERT model operates on 16 kHz inputs, the original LibriTTS samples are down-sampled accordingly before feature extraction. The final restored 24 kHz waveform is then synthesized using the WaveFit vocoder.

Experiment

Subjective listening tests with native English speakers compared LibriTTS-R against the original LibriTTS corpus, first on ground-truth samples and then on TTS outputs from models trained on each corpus. While naturalness MOS scores were similar for ground-truth samples, side-by-side preference tests showed LibriTTS-R was significantly preferred, with spectrogram analysis confirming restoration of degraded speech. For TTS, models trained on LibriTTS-R consistently outperformed those trained on LibriTTS, particularly when using the larger Train-960 set, which improved naturalness and sound quality for most speakers. The findings indicate LibriTTS-R is a superior training corpus, enabling high-quality TTS synthesis on par with human speech, despite occasional distorted samples in the restored dataset.

The table reports subjective evaluations comparing LibriTTS and LibriTTS-R on ground-truth samples. LibriTTS-R achieves higher MOS scores in both test splits, though the differences are not statistically significant for naturalness. In contrast, side-by-side preference tests show significant improvements for LibriTTS-R in sound quality, with positive scores indicating a clear preference. LibriTTS-R improves MOS slightly on both test-clean and test-other, but the gains are not significant. SxS preference scores are strongly positive for LibriTTS-R, with a larger margin on test-other than test-clean. The significant SxS differences suggest that LibriTTS-R's restoration mainly enhances perceived sound quality rather than naturalness.

Models trained on LibriTTS-R consistently outperform those trained on LibriTTS across nearly all speaker IDs, with the largest gains when using the Train-960 split. The only exception is speaker ID 19, where LibriTTS-R Train-460 edges out Train-960, though the difference is not significant. Using LibriTTS Train-960 can even degrade quality compared to Train-460, likely due to degraded samples in the train-other-500 subset. LibriTTS-R training data yields substantially higher mean opinion scores than LibriTTS for almost every speaker. Adding the train-other-500 split to LibriTTS-R (Train-960) improves quality over Train-460 for most speakers, while the same addition to LibriTTS does not. The best LibriTTS-R models reach mean opinion scores above 4.0, whereas LibriTTS models stay below 3.5 for all speakers.

Side-by-side preference tests show that listeners consistently prefer TTS outputs trained on LibriTTS-R over those trained on LibriTTS, with positive scores for both speaker IDs and training set sizes. The preference is stronger when using the larger Train-960 set, indicating that restored speech in LibriTTS-R improves output quality beyond naturalness. Listeners preferred LibriTTS-R trained outputs for both tested speakers, with positive preference scores. Using the larger Train-960 set increased preference for LibriTTS-R compared to Train-460. The preference trend aligns with qualitative observations of fewer artifacts and background noise in LibriTTS-R outputs.

Subjective evaluations show that LibriTTS-R improves perceived sound quality over LibriTTS, with significant side-by-side preference gains, while naturalness differences remain statistically insignificant. Models trained on LibriTTS-R consistently outperform those trained on LibriTTS across nearly all speakers, with the largest gains from the Train-960 split, and the best LibriTTS-R models exceed MOS 4.0 versus below 3.5 for LibriTTS. Listeners also prefer TTS outputs trained on LibriTTS-R, especially with the larger training set, aligning with fewer artifacts and background noise. Overall, the restoration mainly enhances sound quality rather than naturalness.