One-sentence Summary

NVIDIA researchers propose SALM, a speech-augmented language model that combines a frozen text LLM, an audio encoder, a modality adapter module, and LoRA layers to achieve performance on par with task-specific Conformer baselines for Automatic Speech Recognition and Speech Translation while enabling zero-shot in-context learning, with speech supervised in-context training further improving speech-to-text models, and the proposed model is open-sourced via the NeMo toolkit.

Key Contributions

• Proposes SALM, a unified speech augmented language model combining a frozen text LLM, an audio encoder, a modality adapter, and LoRA layers for multitask speech-language modeling, with ASR and AST performance on par with task-specific Conformer baselines and an open-source implementation via the NeMo toolkit.
• Equips speech-to-text models with zero-shot in-context learning ability, demonstrated through keyword boosting for ASR and AST without requiring external context-biasing graphs or learned keyword-specific weights.
• Proposes speech supervised in-context training to bridge the gap between LLM training and downstream speech tasks, further boosting the in-context learning ability of speech-to-text models.

Introduction

Large language models (LLMs) have become highly capable general-purpose learners through instruction tuning and in-context learning, which makes them attractive for speech tasks such as recognition, translation, and synthesis. Prior speech-LLM integration often relies on text as the interface, but text-only interfaces can lose speech-specific information such as speaker, emotion, and accent; deep audio-embedding approaches exist, yet in-context learning for speech-to-text models remains underexplored. Existing keyword boosting and contextual speech recognition methods also typically use external keyword language models, WFSTs, or explicitly learned contextual weights. The authors introduce SALM, which prompts a Megatron LLM with NeMo speech models to create a unified multitask speech-language model. Their approach matches bespoke Conformer baselines on ASR and AST, enables zero-shot in-context learning for speech-to-text keyword boosting, and introduces speech supervised in-context training to further improve this ability.

Dataset

The authors use the following datasets and data processing steps:

• LLM pretraining data: A 2B-parameter Megatron LLM is pretrained on 1.1T tokens. The mixture is 70% English text from web crawl, news, conversation, books, and scientific domains; 15% code from the Stack dataset; and 15% non-English text from CommonCrawl. The model is then fine-tuned on public instruction-following datasets.
• Speech recognition data: SALM is trained on the LibriSpeech training set. The best checkpoint is selected based on word error rate on LibriSpeech dev sets, and final evaluation uses test-clean and test-other.
• Speech translation data: All English audio available for the IWSLT 2023 Offline Track is paired with pseudo-generated translations into German and Japanese. The resulting training set contains 2.7M segments, about 4.8K hours of audio. Evaluation uses MuST-C v2 tst-COMMON. Target-language text is used to train 16384k BPE encodings.
• Keyword boosting data: An internal test set based on NVIDIA GTC talks is force aligned and segmented, totaling 8 hours. Keywords are selected by filtering for words and phrases with high occurrence and low recognition accuracy under greedy decoding. The default keyword list includes 64 keywords, and the authors also study different keyword counts. Keyword accuracy is measured with precision, recall, and F-score from alignment between recognition results and ground truth.

Method

The authors propose the Speech Augmented Language Model (SALM), which conducts supervised speech instruction tuning directly on a text pretrained and instruction fine-tuned Large Language Model (LLM). The resultant model learns to condition on speech prompts, text context, and instructions to predict textual outputs for various speech tasks. This architecture potentially equips speech-to-text models with in-context learning capabilities.

The authors select Fast Conformer and a GPT-style Megatron LLM as the speech and text backbones, respectively. Fast Conformer is a redesigned Conformer featuring a new downsampling schema for improved efficiency while maintaining state-of-the-art accuracy. Specifically, they utilize a 110M pretrained audio encoder from NeMo and a pretrained 2B Megatron LLM that has undergone text instruction fine-tuning.

To guide the LLM to condition on outputs from the audio encoder, the authors introduce modality adapter and Low-Rank Adaptation (LoRA) layers. Two Conformer layers with 4X subsampling serve as modality adapter layers to match the differing information rates and modeling spaces between text and speech. The resulting speech prompt achieves a frame-shift of 320ms. This prompt is projected to the LLM dimension and prepended to the text context and instruction as the input for the LLM. LoRA layers with 128 dimensions are added to the LLM during speech instruction tuning. The LLM weights are frozen, and gradients are back-propagated through the remaining components.

A central motivation for combining the speech model and LLM is to bring instruction tuning from Natural Language Processing (NLP) to speech multitask learning, thereby providing a unified speech model. The authors include diverse speech tasks, such as Automatic Speech Recognition (ASR) and Automatic Speech Translation (AST), along with varied instructions. This approach not only promotes instruction following but also improves the generalization of the modality adapter layers across different tasks. The work reuses paired speech and text data from public ASR and AST corpora, randomly prepending task instructions during training.

Another primary motivation for SALM is to leverage the in-context learning (ICL) ability of LLMs in speech tasks. ICL allows models to predict labels for unseen inputs without additional parameter updates. The authors assess this ability in speech understanding tasks, using keyword boosting as a primary example. Keyword boosting aims to bias the model toward recognizing specific words of interest. In this context, in-context learning is defined as learning boosted words from the prompting text context without back-propagation. The authors provide keywords to the model in the format of optional text context before the text instruction. This contrasts with previous keyword boosting methods that require learning explicit embeddings and weights for boosted words during training or utilizing external biasing graphs.

Given the differences in data formats and learning criteria between LLM pretraining and ICL stages, the authors propose Speech Supervised In-context Training (Speech ICT) to enhance the model's ability to leverage text context in speech understanding. During the speech instruction tuning stage, they augment the supervised data by randomly sampling words from the current utterance and other utterances in the dataset. These sampled words are included as optional text context for the utterance. This in-context training strategy is designed to generalize to unseen words and corpora in unseen domains.

Experiment

The work evaluates SALM, a speech and language model built from a Fast Conformer or SSL audio encoder, a modality adapter, and a 2B-parameter LLM, on LibriSpeech ASR, MuST-C speech translation, and GTC keyword boosting. A unified SALM trained for both ASR and AST improves over bespoke ASR and AST baselines and AST-only SALM, though it performs worse than ASR-only SALM on ASR, while showing strengths on rare words and proper nouns and benefiting from nucleus sampling to reduce hallucination and deletion issues. For keyword boosting, zero-shot in-context prompting achieves comparable gains without external biasing graphs, but only with speech instruction context tuning, and it tends to handle short, compound, and normalized words better than the baseline while struggling with longer phrases. Preliminary dictionary-guided translation results are only moderately better overall, yet successful cases show SALM can naturally incorporate prompted dictionary terms into translations.

SALM achieves competitive ASR and AST results, with ASR pretraining for the audio encoder improving over SSL pretraining and nucleus sampling yielding further gains. The best SALM configuration surpasses the from-scratch baseline but remains behind the stronger ASR-pretrained bespoke baseline overall. A unified SALM trained on ASR and AST data achieves the highest MuST-C BLEU among SALM models, while its ASR performance is slightly lower than the ASR-only SALM. Nucleus sampling improves both ASR and AST metrics relative to the same SALM model without it. The joint ASR and AST SALM achieves the best MuST-C translation quality among SALM configurations, but its LibriSpeech ASR performance is slightly worse than the ASR-only SALM.

SALM shows relative strengths on rare words, proper nouns, and compound segmentations compared with Fast Conformer-Transducer. Its main weaknesses are hallucination and longform deletion, where it can invent or omit content the baseline transcribes more faithfully. Nucleus sampling is reported to mitigate some of these hallucination problems. SALM tends to win on rare words and proper nouns, including compound forms that the baseline splits or misrecognizes. SALM tends to lose on hallucination and deletion cases, sometimes producing unrelated phrases or omitting long spans that the baseline retains.

Without keyword boosting, SALM performs on par with the Fast Conformer-Transducer baseline. Prompting SALM for keyword boosting improves both WER and F-score, and adding nucleus sampling yields further gains. The in-context prompting approach achieves similar relative boosting gains to conventional external biasing while avoiding external biasing graphs or learned biasing embeddings. SALM without boosting performs similarly to the strong Fast Conformer-Transducer baseline. Prompting improves SALM keyword boosting, and nucleus sampling provides additional gains in WER and F-score. The in-context method matches baseline relative keyword boosting gains without external biasing graphs or explicit biasing embeddings. Speech instruction context tuning is necessary for SALM to follow prompts effectively; without it only limited improvement occurs.

Speech in-context training is necessary for effective keyword boosting: without it, prompting yields only limited F-score improvement, with high precision but low recall. Adding augmented in-context training data substantially improves F-score, and a moderate positive ratio of ground-truth words in augmentation provides the best balance. Tuning this ratio shifts the precision-recall tradeoff, while scaling the number of boosted words tends to worsen precision with little change in recall. Without speech in-context training, prompting follows keyword boosting poorly, yielding high precision but low recall. Adding speech in-context training with augmented data substantially improves overall keyword boosting F-score. A moderate positive ratio achieves the best F-score; higher ratios reduce precision, while lower ratios improve precision but reduce recall. Scaling up the number of boosted words degrades precision while recall remains nearly unchanged, due to more false accepts.

The comparison highlights complementary error patterns between SALM and the fast conformer-transducer baseline. SALM is stronger on shorter words, compound words, and text normalization, while the baseline is stronger on longer words and phrases. SALM also shows hallucination and early stopping errors that are rare in the baseline. SALM wins include GPU, HPC, cuDNN, GeForce, NVLink, and healthcare, while baseline wins include NVIDIA, Omniverse, robotic, and cybersecurity. SALM hallucination examples add repetitive or unrelated content, such as 'okay ladies and gentlemen clap your hands cla cla', an error pattern seldom seen in the baseline.

The experiments evaluate SALM for ASR, AST, and keyword boosting against strong bespoke and transducer baselines, including ablations on ASR pretraining, nucleus sampling, and speech in-context training. SALM is competitive overall and particularly stronger on rare words, proper nouns, compound segmentations, and text normalization, while it is more prone to hallucination, longform deletion, and early stopping. Prompting with nucleus sampling improves keyword boosting and matches conventional biasing gains without external graphs or embeddings, but speech in-context training is required for effective prompt following.