One-sentence Summary

Google introduces AudioPaLM, a unified multimodal language model that fuses PaLM-2’s linguistic knowledge with AudioLM’s paralinguistic capabilities to jointly process and generate text and speech, achieving state-of-the-art speech recognition and speech-to-speech translation, zero-shot speech-to-text for unseen language pairs, and cross-lingual voice transfer by leveraging large-scale text pretraining to improve speech processing.

Key Contributions

• AudioPaLM is a unified multimodal architecture that fuses text-based PaLM-2 and speech-based AudioLM to jointly process and generate text and speech, supporting tasks such as speech recognition and speech-to-speech translation.
• Initializing AudioPaLM with pre-trained text-only LLM weights improves speech task performance by leveraging large-scale text data, leading to state-of-the-art results on speech translation benchmarks and competitive speech recognition accuracy.
• AudioPaLM enables zero-shot speech-to-text translation for unseen language pairs and performs cross-lingual voice transfer from a short spoken prompt, with quality that surpasses existing systems according to both automatic metrics and human evaluation.

Introduction

Large language models (LLMs) have been extended to continuous audio by encoding waveforms as discrete tokens and modeling them autoregressively, but prior generative audio systems treat text and audio vocabularies as separate, preventing a single model from freely interleaving the two modalities. Hierarchical tokenization approaches such as AudioLM can produce high‑fidelity speech, yet text‑to‑speech models built on them still rely on disjoint text and audio representations, and speech‑to‑speech translation cascades compound errors while discarding paralinguistic cues. The authors introduce AudioPaLM, a decoder‑only Transformer that fuses text and speech into a unified discrete token vocabulary. This design allows a single model, initialized from a pretrained text LLM like PaLM‑2, to be jointly trained on speech recognition, text‑to‑speech synthesis, and speech‑to‑speech translation, achieving state‑of‑the‑art translation results and competitive recognition performance while enabling voice transfer and zero‑shot translation.

Dataset

The authors assemble a training mixture from multiple datasets, combining real recordings with synthetically generated speech and text. Each subset targets a different modality or task, and additional augmentation steps increase data volume and language coverage.

Dataset composition and sources

• CoVoST2: Speech-to-text corpus mapping 21 languages to English text.
• CVSS: Augments CoVoST2 with synthesized target speech; the canonical voice variant (CVSS-C) is the default for speech-to-speech translation.
• VoxPopuli: European Parliament recordings with transcripts (used for ASR) and spoken interpretations (used for speech translation).
• Common Voice: Volunteer-read text paired with recordings.
• Conversational dataset (Jia et al., 2019a): Crowd-sourced Spanish readings of a proprietary conversational Spanish-English MT subset.
• YouTube ASR: Unlabeled multilingual YouTube audio automatically transcribed by the USM-2B ASR model, used to improve captioning and translation.
• WMT/TED TTS: Based on text-to-text translation datasets (WMT, TED) where both source and target text are fed through a TTS engine to create synthetic paired audio.
• PaLM MT TTS: Transcripts from YouTube, Common Voice, and Babel are translated to English via PaLM-2, then a prior AudioPaLM model synthesizes the translated speech, generating extra S2ST training pairs.

Processing and synthetic generation

• Audio for text-based datasets is generated by running source and target text through a TTS system.
• The PaLM MT TTS set uses a combination of machine translation and speech synthesis to produce paired audio from existing transcripts.
• YouTube data is transcribed automatically and remains unlabeled, while CVSS synthesizes target speech from CoVoST2 references.

How the data is used

• The datasets are mixed for ASR, AST, and S2ST tasks (details in Section 3.5 of the paper), and the mixture ratios are explored in ablation studies.
• The models also inherit knowledge from the pre-trained PaLM text checkpoint and from the data used to train the audio tokenizers, so the effective training signal extends beyond the listed datasets.
• No explicit cropping strategy or metadata construction is described; the focus is on task-specific mixtures and synthetic data augmentation.

Method

The authors leverage a decoder-only Transformer to model sequences consisting of both text and audio tokens. Text and audio are treated as sequences of arbitrary integers, tokenized before input and detokenized after output. A multimodal vocabulary is constructed as the union of an audio vocabulary and a SentencePiece text vocabulary. The overall model architecture is illustrated below.

To convert raw waveforms into tokens, the authors extract embeddings from a speech representation model and discretize them into a limited set of audio tokens. They experiment with several approaches. The first uses a multilingual w2v-BERT model without normalizing the embeddings before k-means clustering. The second uses a 2B parameter multilingual Universal Speech Model encoder, extracting embeddings from the middle layer. The third uses a quantizer finetuned with an auxiliary automatic speech recognition loss to improve multilingual performance. These methods produce tokens at a rate of 25Hz with a vocabulary size of 1024.

To adapt a pretrained text-only decoder to model both modalities, the authors expand the token embeddings matrix $E$E from size $t \times m$t×m to $(t + a) \times m$(t+a)×m, where $t$t is the number of text tokens, $a$a is the number of audio tokens, and $m$m is the embedding size. The output embeddings matrix $E'$E′ is similarly expanded, maintaining the shared variable property $E' = E^T$E′=ET. The first $t$t tokens correspond to text, and the subsequent $a$a tokens represent audio. While the text embeddings are reused from the pretrained model, the audio embeddings are freshly initialized. All model parameters are then trained on mixed speech and text tasks.

To synthesize raw audio from the generated audio tokens, the authors employ either autoregressive decoding or non-autoregressive decoding. Both methods first generate SoundStream tokens, which are then converted to waveforms via a convolutional decoder. The autoregressive approach proceeds in two stages: the first generates low-bitrate SoundStream tokens using voice conditioning, and the second reconstructs higher levels of the residual vector quantizer to improve audio quality. The non-autoregressive approach applies an iterative method that processes all tokens in parallel, offering faster generation with high consistency. Voice conditioning is supplied as a 3-second voice sample to preserve the original speaker's voice during translation.

The model is finetuned on a mixture of tasks including automatic speech recognition, automatic speech translation, speech-to-speech translation, text-to-speech, and text-to-text machine translation. Tasks are signaled by prefixing inputs with tags specifying the task and language, such as [ASR French] or [S2ST English French]. The authors also utilize combined tasks, where the model is instructed to output intermediate steps for complex tasks in a single autoregressive decoding pass. For example, a speech-to-speech translation task might be expressed to first output the source text, then the translated text, and finally the translated audio tokens. This allows the model to attend to all prior decoded content at each stage.

Training mixtures are constructed to cover these tasks. The translation mixture includes recognition, translation, and machine translation tasks. The speech-to-speech mixture adds text-to-speech and direct speech-to-speech translation tasks. Finetuning is performed using the Adafactor optimizer with a constant learning rate of $5 \times 10^{-5}$5×10−5, a dropout rate of 0.1, and loss masking applied to the inputs.

Experiment

The evaluation setup spans speech-to-text translation (AST), speech-to-speech translation (S2ST), and automatic speech recognition (ASR) across CoVoST2, VoxPopuli, FLEURS, and CVSS benchmarks, using BLEU, WER/CER, and subjective quality metrics. The experiments demonstrate that finetuning a pretrained text model with multi-task training, combined intermediate tasks, richer data mixtures, and larger scale substantially improves AST and S2ST performance, while and allowing zero-shot transfer of translation capabilities to languages seen only in ASR data. Adding S2ST tasks yields direct speech translation with high voice similarity and audio quality that surpasses both cascaded baselines and ground-truth synthesis, though with a modest trade-off in text-output metrics.

AudioPaLM training draws on a diverse set of datasets, each offering a different combination of audio, transcripts, and translated audio that dictates their use in ASR, AST, or S2ST tasks. The training hours seen by each model variant differ due to dataset balancing, with YouTube ASR supplying the most audio for speech-to-text models (13k hours for AudioPaLM AST) and VoxPopuli S2ST contributing the largest translated audio volume for speech-to-speech training. Language coverage ranges from 2 languages in the Conversational EsEn dataset to 98 in CommonVoice 11. YouTube ASR provides the largest amount of training audio for the audio-to-text models, at 13k hours for AudioPaLM AST and 7.6k hours for AudioPaLM-2 AST, but is not used in the S2ST mixture. CoVoST2/CVSS and Conversational EsEn are the only datasets that include all four modalities — audio, transcript, translated audio, and translated transcript — making them directly usable for both speech-to-speech translation and combined ASR+AST tasks.

AudioPaLM 8B models substantially outperform existing baselines on speech translation tasks, with the AST variant achieving a large margin on CoVoST2 and the S2ST variant also surpassing Translatotron 2 on CVSS S2ST. On VoxPopuli ASR, the AST variant is competitive but not as strong as the best dedicated ASR models, while the S2ST variant loses some ASR accuracy. The AudioPaLM 8B AST model achieves a CoVoST2 AST BLEU score that considerably exceeds Whisper Large-v2, USM-M, and other baselines. The AudioPaLM 8B S2ST model outperforms Translatotron 2 on CVSS S2ST by a large margin (32.5 vs 25.6 ASR-BLEU). On VoxPopuli ASR, AudioPaLM 8B AST's WER is higher than the best dedicated models (MAESTRO 8.1%, mSLAM-CTC 9.1%), placing it as competitive but not top-tier.

AudioPaLM-2 achieves the highest zero-shot AST BLEU scores on both language groups, surpassing Whisper Large-v2 on AST-observed languages and nearly matching it on only ASR-observed languages, while using far less supervised training data. The large gain over AudioPaLM, especially on languages never seen with translation data, shows that improved text translation from PaLM-2 directly strengthens speech translation in a zero-shot setting. AudioPaLM-2 outperforms Whisper Large-v2 on AST-observed languages (28.6 vs 23.3 BLEU) despite training on roughly 15 times less AST data and an order of magnitude less total supervised audio. For languages with no translation training data, AudioPaLM-2 reaches 20.7 BLEU, nearly matching Whisper's 19.6 BLEU on the same set, even though Whisper had access to AST data for those languages. Switching the base model from PaLM to PaLM-2 more than doubles zero-shot translation BLEU (10.0 to 20.7) on languages only observed with ASR data, demonstrating the direct carryover of text translation improvements to speech tasks.

AudioPaLM achieves the highest scores across all evaluated dimensions, outperforming both the CVSS-T ground-truth baseline and Translatotron 2. Subjective evaluations give AudioPaLM notably higher audio quality and voice similarity ratings, while objective metrics confirm its superior voice preservation and acoustic consistency. AudioPaLM attains the highest subjective audio quality (4.44), surpassing CVSS-T and Translatotron 2. Objective voice similarity for AudioPaLM roughly doubles the score of Translatotron 2 (0.40 vs. 0.18). Acoustic consistency peaks with AudioPaLM (0.81), indicating better transfer of recording conditions.

Training a model on both automatic speech recognition (ASR) and speech-to-text translation (AST) tasks improves translation quality compared to training on AST alone. On the CoVoST2 AST benchmark, adding ASR data raised the BLEU score from 16.0 to 18.5. This indicates that jointly learning to transcribe and translate speech helps the model better connect audio inputs to text. Adding ASR tasks to the training mix boosted AST performance by 2.5 BLEU points. Joint training on ASR and AST outperformed AST-only training on the CoVoST2 speech translation benchmark.

The evaluation covers automatic speech translation (AST), speech-to-speech translation (S2ST), automatic speech recognition (ASR), zero-shot AST, and subjective assessments of translation quality. AudioPaLM models consistently outperform prior baselines on speech translation tasks, while leveraging a PaLM-2 backbone dramatically improves zero-shot AST by transferring stronger text translation capabilities to speech. Joint training on ASR and AST further boosts translation quality, and subjective evaluations confirm that AudioPaLM achieves high audio quality, voice similarity, and acoustic consistency in generated speech.