One-sentence Summary

NVIDIA presents Nemotron 3.5 Transcribe, a speech-augmented language model that couples a Transformer speech encoder and a frozen Nemotron 3 Diarization model with the Nemotron 3.5 Lightning 30B-A3B hybrid Mamba-Transformer mixture-of-experts LLM to deliver instruction-steered transcription, translation, and spoken language identification across 40 languages, and it is the first SALM to outperform specialized cascaded systems on speaker-attributed ASR, achieving 4.71% English and 4.62% multilingual WER on the Open ASR Leaderboard.

Key Contributions

• The paper introduces Nemotron 3.5 Transcribe, a single speech-augmented language model built on the Nemotron 3.5 Lightning 30B-A3B hybrid Mamba-Transformer mixture-of-experts LLM with approximately 4.3B active parameters, performing verbatim, written-form, contextual, speaker-attributed, and rich transcription, speech translation, and spoken language identification in 40 languages, with speaker-attributed word-level timestamps on request. It achieves state-of-the-art speaker-attributed ASR accuracy on overlapping multi-speaker speech, is reported as the first SALM to outperform specialized cascaded systems on this task, reaches mean word error rates of 4.71% on the English and 4.62% on the multilingual Open ASR Leaderboard (best among open-weights models), and reduces named-entity errors on ContextASR-Bench six-fold when given expected entities.
• The architecture uses a Parallel Expert Encoder that injects speaker activity from a frozen diarization model into ASR representations, enabling speaker-attributed transcription with serialized output training without architectural changes to the LLM. The training recipe uses a single supervised fine-tuning stage followed by a short reinforcement learning stage, samples sources by answer-token mass, reuses ASR examples across task instructions, and includes joint multi-token prediction training for speculative decoding.
• The model checkpoint is released in BF16, FP8, NVFP4, and GGUF versions under the OpenMDW-1.1 license, along with the training recipe, a fine-tuning toolkit, and agent skills in NeMo Speech. Fine-tuning to new languages, domains, and tasks is demonstrated on 17 tasks on a single GB300 GPU, typically within 0.5 to 2.3 GPU-hours per task.

Introduction

Speech-augmented language models have recently begun to match dedicated ASR systems and offer promptable transcription, translation, and formatting, but in practice most models are trained for a single transcription task or for broad audio understanding at the expense of transcription quality. Real-world transcription also spans verbatim and written-form output, rare entity recognition, speaker attribution, long recordings, code-switching, and non-speech audio, and these capabilities compete for the same training budget. The authors introduce Nemotron 3.5 Transcribe, a speech LLM built on a hybrid Mamba-Transformer mixture-of-experts model. It uses a Parallel Expert Encoder that fuses a trainable ASR encoder with a frozen speaker diarization model, and it is trained with a large multi-task mixture followed by reinforcement learning. The model targets intelligent transcription across 40 languages, including verbatim, written-form, contextual, multi-speaker, and rich transcription, speech translation, and language identification with word-level speaker-attributed timestamps.

Dataset

Dataset composition and sources

• The authors use the Granary-v2 multilingual speech dataset and its reusable curation and labeling pipeline. The pipeline generates or improves transcripts and creates supervision for short and long-form ASR, text normalization, disfluency control, contextual ASR, code-switched transcription, speech question answering, speech captioning, spoken language identification, and speech translation.
• The public release contains the shareable portion of Granary-v2 data. The full training mixture also includes private multilingual speech data and additional task-specific sources.
• According to Figure 5, four sampling groups account for more than 95% of training examples: Granary-v2 pseudo-labeled data, Granary-v2 human-label-improvement data, Granary-v2 speech translation, and internal multilingual speech data.
• Granary-v2 also supplies long-form ASR, Indian English, and language identification examples as separate sampling groups. Other sources are sampled with probabilities between 0.01% and 3% and target capabilities such as multi-speaker transcription or audio understanding.
• Granary-v2 extends the original Granary corpus.

Scale and language coverage

• The speech data covers 77 languages. Of these, 41 have at least 370 hours of distinct audio each (Table 2), and 40 are supported for transcription.
• Norwegian is excluded from supported transcription because its training data did not provide usable quality.
• Regional variants are represented for English from the United States, India, and the United Kingdom; Spanish from the United States and Spain; French from France and Canada; and Portuguese from Portugal and Brazil.
• English accounts for 36% of audio seen during training. The next eight languages account for 2.5% to 7% each, and the remaining 32 supported languages account for less than 2.5% each.
• The other 36 languages appear for about 300 hours in total and are not officially supported.
• The mixture also includes 0.35 million hours of audio without a language label, mostly from audio and speech understanding sources. This unlabeled audio makes up 5.6% of audio seen during training.

Task distribution and data preparation

• Table 3 shows the distribution over task categories: transcription without and with context makes up more than 70% of examples, and speech translation makes up 17%.
• The data is prepared as a multi-task mixture, with a random task assigned per example, such as contextual ASR.
• Indexed data is used for O(1) random access.
• Cropping and metadata construction details are not separately described in this section.

Method

The authors present Nemotron 3.5 Transcribe, a speech-to-text model following the Speech-Augmented Language Model design. The architecture features a Parallel Expert Encoder that converts audio into embeddings, which are inserted into the prompt of a decoder-only LLM. The LLM then generates the response autoregressively.

The Parallel Expert Encoder processes 16 kHz audio split into independent 30-second chunks. It consists of two branches: a trainable Transformer ASR encoder with 32 layers and a frozen Sortformer diarization model. The diarization model estimates per-frame speaker activities, which are mapped through a fixed sinusoidal kernel and added to the ASR representations. This design provides the LLM with acoustic representations tagged with speaker information without modifying the LLM architecture. The encoder outputs are mapped to the LLM embedding space via a single linear projection from 1280 to 2688 dimensions, replacing the audio placeholder token in the prompt.

The language model backbone is Nemotron 3.5 Lightning 30B-A3B, a hybrid Mamba-Transformer Mixture of Experts model. It comprises 52 layers, including 23 Mamba-2 layers, 23 MoE feed-forward layers, and only 6 grouped-query attention layers.

This small number of attention layers minimizes the key-value cache size, which is crucial for processing long audio recordings. Each MoE layer utilizes 128 routed experts with top-6 routing and one shared expert, resulting in approximately 3.5 billion active parameters per token.

To enhance inference efficiency, the model is trained jointly with a Multi-Token Prediction objective. The MTP module consists of one attention block and one MoE block, reused across two prediction depths with shared parameters. During training, it predicts tokens at positions $t+2$t+2 and $t+3$t+3 by combining the backbone hidden state with ground-truth token embeddings. At inference time, this module serves as a draft model for speculative decoding.

For word-level timestamps, the authors include a lightweight Connectionist Temporal Classification adapter. This adapter reuses the ASR encoder representations, upsampling them via a transposed convolution and processing them through two Transformer layers. After text generation, CTC forced alignment aligns the transcript to the audio to produce word timestamps.

The training process begins with supervised fine-tuning on a large multi-task mixture, followed by a reinforcement learning stage to optimize transcription accuracy directly.

The training data is drawn from Granary-v2 and internal multilingual speech sources, covering 77 languages. The data mixture is balanced based on the answer-token mass of each source to ensure stable supervision across tasks.

During supervised fine-tuning, the model is trained for 28,500 steps using the cross-entropy loss on assistant turns, combined with the MTP loss and an MoE load-balancing loss. All parameters except the diarization model and feature extractor are trained jointly using AdamW. The training infrastructure utilizes 32 nodes with 8 NVIDIA H100 GPUs each, employing expert parallelism and fully sharded data parallelism.

Following supervised fine-tuning, the authors apply a short reinforcement learning stage using Group Sequence Policy Optimization for 400 steps. The reward is defined as $1 - \text{WER}$1−WER, where WER is the word error rate or character error rate for Mandarin. The reinforcement learning stage updates a subset of parameters, including the Mamba and attention blocks of the LLM, the last two ASR encoder layers, and the audio-to-LLM projection, while freezing the MoE blocks and routers.

Finally, the CTC timestamp adapter and a DFlash2 draft model for speculative decoding are trained separately on top of the frozen backbone. The CTC adapter is optimized using the CTC objective, while the draft model is trained to predict token blocks in parallel using local dynamic convolutions and a pairwise selector.

Experiment

The evaluation uses the released post-reinforcement-learning checkpoint under a deployment-oriented vLLM setup with speculative decoding, comparing against open and commercial baselines on public benchmarks, with greedy decoding for transcription, translation, and language identification and reasoning mode for audio understanding and instruction following. Across these experiments, the model shows competitive open-weights performance on English, long-form, multilingual, contextual, entity-heavy, and speaker-attributed ASR, performs speech translation and generally accurate language identification, and benefits clearly from the loop guardrail and supplied context, while code-switched speech and Mandarin far-field meetings remain relative weaknesses. The quantization study finds that FP8 preserves accuracy well and NVFP4 offers stronger efficiency at the cost of larger regressions in some multilingual and multi-speaker settings, and additional evaluations highlight accurate multi-speaker word-level timestamping and useful audio understanding and voice instruction following, while positioning the system primarily as a transcription model rather than a general-purpose omni or text-only model.

Nemotron 3.5 Transcribe pairs a Transformer-based ASR encoder with a frozen speaker diarization model and a linear adapter. The ASR encoder is much larger than the diarization model and processes log-mel inputs at a low frame rate. The adapter maps the encoder output into the LLM's wider hidden size. The ASR encoder uses RoPE and QK-norm with 32 layers, width 1280, and 10 attention heads. The frozen diarization model supports up to 8 speakers and has about 0.1B parameters, compared with about 0.6B for the ASR encoder. A linear adapter projects the 1280-dimensional encoder output to 2688 dimensions.

Among the languages covered, 41 have at least 370 hours of distinct audio, and 40 are supported for transcription. The training audio distribution is heavily skewed toward English, which dominates both distinct hours and total audio share. Spanish, French, Portuguese, German, Japanese, and Italian form a second tier, while the remaining listed languages contribute much smaller amounts. English accounts for more than a third of all audio hours, over five times the share of any other listed language. Japanese has a total audio share comparable to Spanish and French despite having fewer distinct hours.

The training audio mixture is dominated by ASR and text normalization, which accounts for a clear majority of examples and the largest seen audio volume. Speech translation and contextual ASR are the next largest categories by example share. Smaller categories such as sound and music understanding and speaker-attributed ASR contribute substantial distinct audio hours relative to their example counts, indicating lower repetition or longer recordings. ASR and text normalization accounts for over half of all training examples and the most seen audio hours. Speech translation and contextual ASR together account for roughly one-third of training examples. Sound and music understanding has a near one-to-one ratio of distinct to seen audio hours, indicating low repetition. Speaker-attributed ASR has the smallest example share but contributes substantial distinct audio hours.

Supervised fine-tuning uses AdamW with weight decay and gradient clipping, applying a short warmup followed by a long cosine learning-rate decay over tens of thousands of steps. Training uses packed sequences totaling tens of thousands of tokens per GPU. The checkpoint used for subsequent reinforcement learning is taken near the end of the scheduled decay. AdamW optimizer settings include standard momentum coefficients and a small epsilon, with a low peak learning rate and a much lower final learning rate. Cosine decay follows a short warmup phase, and the checkpoint near the end of the schedule is used for reinforcement learning. Each GPU processes 65,536 packed tokens, with weight decay at 0.05 and gradient clipping at 1.0.

Reinforcement learning improves the model over the SFT checkpoint on every reported benchmark. Word error rates decline across English, multilingual, and contextual ASR evaluations, with the largest relative gains on CoVoST 2 ASR and Numb3rs. VoiceCodeBench CTEM also increases, indicating stronger performance on that metric. The RL-trained model outperforms the SFT checkpoint on all reported metrics, lowering WER on ASR evaluations and improving VoiceCodeBench CTEM. CoVoST 2 ASR and Numb3rs show the largest WER reductions among the reported benchmarks, while Contextual Earnings-22 improves more modestly.

The Nemotron 3.5 Transcribe setup combines a large Transformer ASR encoder with a frozen speaker diarization model through a linear adapter, and its training data is heavily English-skewed with multilingual support dominated by ASR and text normalization tasks. Supervised fine-tuning uses AdamW with packed sequences and a long cosine decay schedule, after which reinforcement learning is applied from a late SFT checkpoint. The RL stage consistently improves over the SFT model across English, multilingual, and contextual ASR benchmarks, with the largest word error rate reductions on CoVoST 2 ASR and Numb3rs and gains in VoiceCodeBench CTEM.