One-sentence Summary

Google researchers present Tacotron, an end-to-end generative text-to-speech model that synthesizes speech directly from characters via a sequence-to-sequence framework trained from scratch with random initialization, achieving a $3.82$3.82 mean opinion score on US English, outperforming a production parametric system in naturalness, and running substantially faster than sample-level autoregressive methods.

Key Contributions

• Tacotron is an end-to-end generative TTS model that synthesizes speech directly from character sequences, enabling training from scratch without phoneme-level alignment or hand-engineered linguistic features.
• The model extends the sequence-to-sequence with attention paradigm through several key modifications that allow stable training on raw text–audio pairs despite large mismatches in sequence length and modality.
• On US English, Tacotron achieves a 3.82 mean opinion score, outperforming a production parametric system in naturalness, and its frame-level generation is substantially faster than sample-level autoregressive methods.

Introduction

The authors describe how traditional text-to-speech pipelines are complex, with independently trained components (linguistic frontend, duration model, acoustic predictor, vocoder) that require laborious feature engineering and where errors can compound across stages. Prior attempts at end-to-end synthesis still relied on external aligners or separately trained neural vocoders, and often needed phoneme-level inputs. The authors propose Tacotron, an end-to-end generative model that uses a sequence-to-sequence architecture with attention to directly predict raw spectrograms from character sequences, training from scratch without any phoneme alignment or hand-crafted linguistic features, and achieving higher naturalness than a production parametric system.

Dataset

The authors use a single, internal dataset to train the Tacotron model.

• Composition and source: An internal North American English dataset spoken by one professional female speaker.
• Scale and filtering: Contains approximately 24.6 hours of speech. The text is normalized (for example, “16” becomes “sixteen”).
• Usage in the model: The entire dataset is used as training data for Tacotron; no separate validation or test split is mentioned.

Method

The authors leverage a sequence-to-sequence (seq2seq) model with attention as the backbone of their architecture. At a high level, the model takes character sequences as input and produces spectrogram frames, which are subsequently converted into waveforms. The overall framework comprises an encoder, an attention-based decoder, and a post-processing network.

A fundamental building block utilized throughout this architecture is the CBHG module, which is designed for robust representation extraction from sequences.

The CBHG module consists of a bank of 1-D convolutional filters, followed by highway networks and a bidirectional gated recurrent unit (GRU). The input sequence is convolved with $K$K sets of 1-D filters, where the $k$k-th set contains $C_k$Ck​ filters of width $k$k (for $k = 1, 2, \ldots, K$k=1,2,…,K). These filters model local and contextual information, akin to unigrams through K-grams. The convolution outputs are stacked and max-pooled along the time dimension with a stride of 1 to preserve time resolution while increasing local invariances. The processed sequence is then passed through fixed-width 1-D convolutions, and the outputs are added to the original input via residual connections. Batch normalization is applied to all convolutional layers. Finally, the outputs are fed into a multi-layer highway network for high-level feature extraction, followed by a bidirectional GRU to capture sequential features from both forward and backward contexts.

The encoder aims to extract robust sequential representations of text. The input character sequence, represented as one-hot vectors, is embedded into continuous vectors. A pre-net, consisting of a bottleneck layer with dropout, is applied to each embedding to aid convergence and improve generalization. A CBHG module then transforms these pre-net outputs into the final encoder representation used by the attention module.

The decoder employs a content-based tanh attention mechanism. A stateful recurrent layer produces the attention query at each decoder time step. The context vector and the attention RNN cell output are concatenated to form the input for the decoder RNNs, which consist of a stack of GRUs with vertical residual connections to accelerate convergence. Instead of directly predicting raw spectrograms, the model targets an 80-band mel-scale spectrogram, a highly compressed representation that provides sufficient intelligibility and prosody information for the inversion process.

A key design choice in the decoder is predicting multiple, non-overlapping output frames (denoted as $r$r frames) at each decoder step. This reduces the total number of decoder steps, thereby decreasing model size, training time, and inference time. More importantly, it substantially increases convergence speed by allowing the attention mechanism to move forward more easily during training, as neighboring speech frames are correlated and each character typically corresponds to multiple frames. During inference, the last frame of the $r$r predictions is fed as input to the next decoder step, while during training, every $r$r-th ground truth frame is used. The input frame passes through a pre-net similar to the encoder, where dropout is critical for generalization since scheduled sampling is not used.

The post-processing network converts the seq2seq target (mel-scale spectrogram) into a linear-scale spectral magnitude suitable for waveform synthesis. Using a CBHG module, this network benefits from seeing the full decoded sequence, allowing it to utilize both forward and backward information to correct prediction errors for individual frames. The authors employ the Griffin-Lim algorithm to synthesize the waveform from the predicted spectrogram. To reduce artifacts, the predicted magnitudes are raised to a power of 1.2 before feeding them to Griffin-Lim, which enhances harmonics. The Griffin-Lim algorithm is implemented within the model framework, though no loss is imposed on it during training.

For training, the authors use the Adam optimizer with learning rate decay, starting at 0.001 and reducing to 0.0005, 0.0003, and 0.0001 after 500K, 1M, and 2M global steps, respectively. An L1 loss is applied to both the seq2seq decoder and the post-processing net with equal weights. To address the issue of models not knowing when to stop emitting outputs, which causes repeated sounds, the authors reconstruct the zero-padded frames rather than masking the loss on them. A batch size of 32 is used, with sequences padded to a maximum length.

Experiment

The model was trained on a 24.6-hour internal North American English dataset. Ablation experiments, relying on visual comparisons, show that the CBHG encoder yields clean alignments and reduces mispronunciations, while the post-processing net improves harmonic structure. Subjective MOS tests on 100 unseen phrases, with ratings from headphone users, indicate that Tacotron outperforms a production parametric system, demonstrating strong naturalness despite Griffin-Lim synthesis artifacts.

The Tacotron architecture uses a CBHG encoder for clean alignments and a post-processing CBHG net to refine spectral details, along with a 2-layer residual GRU decoder. Training employs 50 ms frames, pre-emphasis, and a stepped learning rate decay, with a trick of reconstructing padded frames to avoid repetitive outputs. Ablations show CBHG yields smoother attention than a GRU encoder, and the post-net improves harmonic structure and reduces artifacts. A CBHG encoder with a bank of 16 convolution filters and a bidirectional GRU learns clean attention alignments, reducing mispronunciations compared to a 2-layer residual GRU encoder. The post-processing CBHG net better resolves high-frequency harmonics and formant structure, lowering synthesis artifacts relative to the decoder’s direct prediction. Reconstructing zero-padded frames during training prevents the model from emitting repeated sounds at the end of utterances. Stepped learning rate decay from 0.001 down to 0.0001 and Adam optimizer are used with batch size 32.

In a 5-scale mean opinion score test of speech naturalness, Tacotron achieved a rating of 3.82, placing it between a production parametric system (3.69) and a production concatenative system (4.09). The result is notable given that Tacotron uses Griffin-Lim synthesis, which introduces artifacts, yet it still outperforms the parametric baseline. Tacotron outperforms the parametric system by a margin of 0.13 MOS points. The concatenative system yields the highest naturalness score among the compared systems. All ratings are based on crowdsourced evaluations from native speakers using headphones.

The Tacotron model's design choices were validated through ablation studies and a mean opinion score listening test. Qualitatively, the CBHG encoder yields cleaner attention alignments and reduces mispronunciations compared to a GRU encoder, while the post-processing net improves harmonic resolution and suppresses synthesis artifacts; a training trick of reconstructing padded frames prevents end-of-utterance repetition. In naturalness evaluations, Tacotron outperforms a production parametric system, though it remains below a concatenative baseline, demonstrating its effectiveness despite using the simple Griffin-Lim vocoder.