One-sentence Summary

Fish Audio Team's Fish Audio Drama 3 is an open-source 35B-parameter mixture-of-experts speech generator for directed text-to-performance synthesis that unifies voice cloning, natural-language direction, revision, multi-character dubbing, infilling edits, and streaming through a hybrid linear-attention backbone, a codebook multi-token-prediction audio decoder, and preference-reinforcement training for efficient, real-time, controllable generation.

Key Contributions

• Fish Audio Drama 3 is an open-source speech generation model built on a 35B-parameter mixture-of-experts backbone with 3B active parameters per token. It pairs a hybrid linear-attention backbone with a codebook multi-token-prediction audio decoder whose layers attend over the entire sequence, and uses three-stage pre-training, instruction fine-tuning, and reinforcement learning with a single learned human-preference reward.

• The model handles multi-reference voice cloning, natural-language direction, multi-turn revision, multi-character scenes, infilling edits, long-form synthesis, and streaming synthesis within one shared conversation format. Its serving engine runs the codebook multi-token-prediction decoder once per frame, uses block-scaled FP8 experts, removes kernel launches left by CUDA graphs, optimizes kernels for live serving shapes, separates prefill from decode, and reaches 271 frames per second for one stream on one H200 versus 177 for Fish Audio S2.

• Evaluation shows that revision rounds improve emotion and style accuracy on the Step-Audio-Edit benchmark, achieve the lowest voice cloning error rate among compared systems across Chinese, Dutch, English, German, and Turkish on the MiniMax Multilingual Testset, and give a lower error rate than the Cartesia Sonic-3 infill API on the Ming-Freeform-Audio-Edit benchmark in English and Chinese. The model weights are released on GitHub and Hugging Face with SGLang-Omni inference support, and Drama 3 is described as the largest open-weight dedicated speech generation model by total parameters.

Introduction

Large-scale text-to-speech models have progressed to near-human intelligibility and speaker similarity, but real applications such as audiobooks, dubbing, and interactive characters require directable performances rather than simple readings. The authors build on Fish Audio S2, their earlier model with inline natural-language direction and multi-speaker generation, but note that its full-attention backbone struggled with long sessions, its Fast AR module gave limited access to reference audio, its control was single-shot, local edits required regenerating whole takes, and reinforcement learning relied on word error and speaker similarity rewards that could penalize laughs, pauses, and whispers. The authors leverage a hybrid linear-attention mixture-of-experts backbone and a codebook multi-token-prediction decoder to redesign the model, task formulation, and post-training objective around directed performance. Their main contribution is Fish Audio Drama 3, a single open-weight speech generation model that supports multi-turn revision, scene dubbing, infilling-based editing, and streaming text synthesis while using a learned human-preference reward instead of rule-based rewards.

Dataset

1. The pre-training data is built from two main sources: quality-controlled transcribed speech and text documents.

2. The speech data is transcribed and passed through quality control before use, while the text documents help preserve the model's underlying language ability.

3. The full pre-training corpus contains several hundred billion tokens and covers roughly 80 languages and dialects, following the language coverage of S2.

4. The early stages use text documents and multi-turn TTS conversations as the main data types. These sequences are kept to at most 8K tokens and are used to teach the model to map text into audio codes.

5. Later stages expand the data to full recordings, streaming input, and speech editing tasks. The sequence-length limit increases to 16K tokens, and the data is organized to support whole-recording synthesis from transcripts, line-by-line synthesis, streaming interleave, and speech infilling.

6. The authors pack samples with document masking into rows of 128K tokens, which lets text and speech samples be mixed within the same training batches.

7. The supervised structure targets text tokens, line separators, closing segment tokens, and semantic audio codes. Lower codebook targets are also included for decoder-side modeling, so the data schema reflects both text and multi-codebook speech representation.

8. For multi-reference zero-shot voice cloning evaluation, the authors extend the LibriSpeech-PC test-clean list from F5-TTS. The base set contains 1,127 sentences from 39 speakers.

9. Each target speaker is supplemented with two additional reference clips, and those clips are filtered so they never include the target sentence.

10. The reference clips are formatted into separate reference blocks, and the resulting outputs are scored using the same evaluation protocol as OmniVoice.

Method

The Drama 3 architecture is designed for high-fidelity, expressive speech synthesis, leveraging an autoregressive language model backbone coupled with a specialized decoder for acoustic detail. The system processes interleaved text and audio tokens to generate speech frame by frame.

Architecture and Module Design

Audio Tokenizer and Backbone The system utilizes a causal audio tokenizer based on the Descript Audio Codec, which encodes 44.1 kHz audio into frames of ten codebooks at approximately 21.5 frames per second. The first codebook represents semantic content, while the remaining nine are residual vector quantizers providing progressively finer acoustic detail.

The backbone is initialized from the Qwen3.5-35B-A3B language model. It employs a hybrid linear-attention mechanism suitable for long-form speech, stacking blocks of three Gated DeltaNet layers (which maintain history in a constant-size state) with one gated full-attention layer (which preserves exact retrieval over the entire context). To manage computational cost, every layer uses a mixture-of-experts (MoE) feed-forward network that routes each token to 8 of 256 experts, activating only about 3 billion parameters per token.

Partial Codebook Fusion Unlike previous designs that fed all ten codebooks back into the backbone, Drama 3 employs Partial Codebook Fusion to limit exposure bias. Only the semantic code ($q_t^{(0)}$qt(0)​) and the first acoustic code ($q_t^{(1)}$qt(1)​) are fed back. At audio frame $t$t, the backbone input is computed as:

where $\mathbf{E}^{\mathrm{LM}}$ELM is the backbone token embedding and $\mathbf{E}^{(1)}$E(1) is the embedding table for the first acoustic codebook. The remaining codes ($q_t^{(2)}$qt(2)​ through $q_t^{(9)}$qt(9)​) bypass the backbone and enter the decoder directly.

Codebook-MTP Decoder The Codebook-MTP (Multi-Token Prediction) decoder replaces the per-frame Fast AR design of previous iterations. At position $t$t, while the backbone predicts the next frame's semantic code $q_{t+1}^{(0)}$qt+1(0)​, the decoder predicts the remaining nine acoustic codes $q_{t+1}^{(1)}$qt+1(1)​ through $q_{t+1}^{(9)}$qt+1(9)​ at the same position.

The decoder consists of 13 layers: four shared layers followed by nine per-codebook layers.

1. Shared Layers: These layers merge the backbone's hidden state with the predicted semantic code and the input frame's codes 2–9. This fuses the backbone state, current semantic information, and carried-over acoustic detail into a single representation.
2. Per-Codebook Layers: Each of the nine subsequent layers predicts one specific acoustic codebook. Layer $k$k takes the hidden state from the previous layer and injects the code chosen by layer $k-1$k−1.

Crucially, every layer in the decoder attends causally over the entire sequence (including reference audio), allowing fine codes to be predicted from full context rather than a summary vector.

Refer to the framework diagram:

Training Process

The training pipeline progresses from pre-training a base language model to aligning it with speech tasks, followed by instruction tuning and reinforcement learning.

Pre-training The model undergoes three pre-training stages using transcribed speech and text documents.

• Stages 1 and 2: Focus on speech-text alignment with sequences up to 8K tokens.
• Stage 3: Extends the model to whole recordings, streaming input, and speech editing, raising the maximum sequence length to 16K tokens.

The objective function combines text, semantic, and codebook losses:

where $\mathcal{L}_{\text{text}}$Ltext​ is the standard language modeling loss, $\mathcal{L}_{\text{sem}}$Lsem​ is the loss for the semantic code $q_t^{(0)}$qt(0)​, and $\mathcal{L}_{\text{cb}}$Lcb​ is the average loss over the nine acoustic codebooks.

Instruction SFT and Reinforcement Learning Following pre-training, the model undergoes Supervised Fine-Tuning (SFT) on instructions formatted in ChatML style. This stage teaches the model to follow directions, perform dialogue, and handle multi-speaker scenarios using an indexed reference layout.

Finally, Reinforcement Learning (RL) aligns the model with human preferences using Group Relative Policy Optimization (GRPO). A single reward model (RM), trained on human preference judgments via Bradley-Terry loss, scores generated takes. The RM reads the policy's prompt followed by the audio tokens, evaluating voice consistency and content fidelity without relying on external ASR or speaker similarity metrics which might penalize expressiveness. The policy is optimized using mean-centered advantages:

where $R_i$Ri​ is the expected win rate of take $i$i against other takes in the group.

As shown in the figure below:

Inference Engine Optimization

To support real-time streaming, the authors implemented a shape-driven kernel optimization loop. General-purpose kernel libraries are often inefficient for the specific shapes found in TTS serving (e.g., small batch sizes, thin matrices). The optimization process iterates through five steps: shape capture from serving logs, profiling against hardware rooflines, writing candidate kernels, interface testing with cold caches, and in-engine validation.

This loop is executed by coding agents under strict contracts defined by engineers. For example, the Gated DeltaNet decode kernel was optimized to reduce memory traffic, and MoE kernels were developed to run in block-scaled FP8, significantly reducing the weight bytes read per frame and increasing the per-stream decode rate.

Refer to the framework diagram:

Experiment

Experiments evaluate Drama 3 across inference serving and public speech benchmarks, using benchmark-specific references, standard intelligibility and speaker-similarity protocols, and regenerated invalid outputs. Serving results show that the lighter generation structure, FP8 experts, and separating prefill and decode improve streaming throughput, capacity, and stability without harming speech quality. The qualitative findings also validate the model’s single-model design: the learned preference reward better respects style directions and reduces generation failures, while multi-reference cloning, multilingual synthesis, streaming, infilling edits, and multi-turn revision all preserve strong speech quality and controllability.

Drama 3 has a lower per-frame decoding cost than S2 because it uses fewer sequential layer passes and reads each decoder weight once per frame. The combination of single-pass codebook-MTP decoding and FP8 routed experts reduces bytes moved, which supports higher frame-rate decoding than S2 on the evaluated GPUs. Drama 3 requires fewer sequential layer passes after the backbone step than S2. Drama 3 reads substantially less weight data per frame, helped by one-pass decoding and compressed expert storage. The lower memory traffic gives Drama 3 a frame-rate advantage over S2 in single-stream decoding and helps throughput at higher concurrency.

Kernel-launch reductions below CUDA graphs and PDL improve per-stream decode rates by removing auxiliary kernels and fusing dependent operations. The largest reported gain comes from capturing the host-launched first-frame sampling chain into graphs, while auxiliary decode cleanup, prefill launch merges, and host-side execution planning also increase throughput. Removing auxiliary decode kernels modestly raises decode rate. Fusing Gated DeltaNet decode and folding sampler state improves throughput. Merging Gated DeltaNet prefill launches reduces per-layer overhead. Capturing first-frame sampling into graphs sharply cuts host time. Host-side execution planning greatly reduces GPU idle time.

Styling reduces speaker similarity relative to each speaker's default reading, with the strongest effects for whispering and laughing. Its effect on transcript error varies, increasing error for laughing but reducing it for clearly enunciated reading. The learned preference reward favors styled readings more often when the style is provided as a direction, unlike similarity scoring, which usually favors the default reading. Whispering and laughing are strongly disfavored by speaker similarity compared with the default reading. Adding a style direction raises preference for the styled reading in every style.

Drama 3 achieves lower first-take and final error rates than the SFT checkpoint on both Chinese and English Seed-TTS-Eval evaluations. Reinforcement learning also reduces flagged first takes, indicating fewer outputs failing signal checks. After regeneration replaces flagged takes, Drama 3 still retains a slight error-rate advantage. Drama 3 improves both first-take and final recognition error rates over the SFT checkpoint Flagged first takes fall after reinforcement learning, even without a transcription reward Final outputs remain better after flagged takes are regenerated

In zero-shot voice cloning, providing multiple reference clips improves speaker similarity for Drama 3 while leaving word error rate and naturalness unchanged. Compared with quoted one-reference systems, Drama 3 shows slightly higher UTMOS than two baselines, but lower word accuracy and lower similarity to the benchmark reference. Using several references raises speaker similarity without changing word error rate or UTMOS. With one reference, Drama 3 exceeds OmniVoice and CosyVoice 3 on UTMOS but trails the quoted systems on word error rate and speaker similarity.

Experiments evaluate Drama 3 across efficiency, stylization, reinforcement learning, and zero-shot voice cloning, validating its faster decoding from reduced layer passes, single-pass codebook decoding, FP8 experts, and kernel-launch optimizations, along with improved throughput. Stylization reduces speaker similarity, especially for whispering and laughing, with variable effects on transcript error, while style directions increase preference for styled readings. Reinforcement learning lowers both first-take and final recognition errors and reduces failed signal checks, and multiple reference clips improve speaker similarity without affecting word accuracy or naturalness.