One-sentence Summary

ByteDance and Zhejiang University researchers present SwanTale, a unified multi-speaker speech and audio generation model for instruct and zero-shot tasks that combines SwanVAE, reward-conditioned quality control, Engram conditioning, Unified MoE, curriculum learning, and GRPO post-training to lead on multiple key zero-shot and instruct metrics, achieve the best expressiveness scores, and support complex multi-speaker audio generation.

Key Contributions

• SwanData-Caption provides a data pipeline that cleans raw speech and audio, adds targeted synthetic coverage, and annotates multi-level captions covering environment, speaker styles, and fine-grained content, together with a style-persona library and quality filtering to produce high-quality training data.
• SwanTale is a unified multi-speaker speech and audio generation model supporting zero-shot and instruct tasks through SwanVAE for multi-modal generation, reward-conditioned quality control, Engram conditioning, a Unified Mixture-of-Experts architecture with task and audio routers, curriculum learning, and GRPO post-training.
• Experimental results show that SwanTale leads on multiple key zero-shot and instruction-following benchmarks, achieves the best expressiveness scores in both tasks, and enables complex instruct generation involving multi-speaker speech and audio within a single model.

Introduction

Recent zero-shot text-to-speech systems can clone a speaker from a short reference clip, but media production often requires designing a voice from scratch and controlling the entire acoustic scene, including environment and audio effects, through natural language instructions. Existing instruct TTS models generate only speech and rely on decomposed attribute labels, while downstream audio pipelines can cause temporal and acoustic drift. Jointly supporting expressive multi-speaker speech, scene audio, singing, and music generation under both instruction and zero-shot conditioning remains an open challenge due to data scarcity, task compatibility, and multi-modality complexity. The authors address this by constructing SwanData-Caption, a pipeline that produces richly annotated captions from speech-centered media, and proposing SwanTale, a unified model that combines a variational autoencoder, a flow-based Transformer with quality control and mixture-of-experts routing, curriculum learning, and GRPO post-training to generate diverse audio from captions or reference audio.

Dataset

The authors construct SwanData-Caption, a large-scale dataset of approximately 70 million fine-grained captioned audio records, to train the SwanTale model. The dataset combines real-world media-style audio with targeted synthetic expansions and is processed through a multi-stage pipeline that includes separation, diarization, transcription, alignment, automatic caption annotation, and rigorous refinement.

Dataset composition and sources

• Real-world media-style data: Internal, speech-centered corpora covering short dramas, advertisements, animations, and similar content. These recordings contain diverse speaker densities, recording conditions, character styles, scene locations, sound fields, persistent background effects, and local audio effects.
• Targeted synthetic subsets: Three subsets generated by a phoneme-aware TTS teacher, each containing 100k utterances, designed to fill rare-scenario gaps:
  • Elderly speech: average duration ~10 seconds; expands demographic age coverage.
  • Short utterances in Chinese and English: single words, names, and short sentences; average duration ~1.5 seconds; improves stability for sub-second spoken content.
  • Challenging pronunciation targets: Chinese texts with polyphonic characters, proper nouns, brand names, English insertions, and mixed-language phrases; average duration ~10 seconds; pronunciation hints are fed to the teacher model while target text remains unchanged.

Key details for each subset

• Real-world data is segmented into speech spans: 1–60 seconds for single-speaker segments, up to 120 seconds for multi-speaker segments (with at least two speaker turns when possible). After filtering, the retained speech set averages about 10 seconds per segment.
• Synthetic elderly and pronunciation subsets average ~10 seconds; short utterances average ~1.5 seconds.
• All subsets are automatically captioned using the same annotation pipeline, with style-persona matrices applied to animation, short drama/film/TV drama, and advertisement/digital-human content to guide richer speaker descriptions.

How the data is used

• The entire SwanData-Caption mixture (~70M caption records) is used to train SwanTale, a model that generates expressive speech and audio from structured captions.
• Each caption record contains three fields: Environment (scene location, sound-field impression, persistent background sounds), Speakers (stable speaker traits for every speaker who actually speaks), and Content (fine-grained local style, emotion, and local audio effects, with speaker-tagged text). This schema supports both zero-shot and instruction-based text-to-speech.
• No explicit train/validation split is mentioned; the dataset serves as the training corpus for the model.

Processing and metadata construction

• Preprocessing (SwanData-Speech backbone): For mixed-media audio, vocal and background streams are separated with Ultimate Vocal Remover. Speaker diarization uses the 3D-Speaker toolkit (VAD, CAM++ embeddings, clustering) to produce coarse segments. ASR is performed with Seed-ASR 2.0 on the vocal stream; SenseVoice ASR provides a pronunciation check. SwanAligner aligns the transcript with the vocal stream and stores pause evidence, while punctuation is later supplied by the caption annotator.
• Caption annotation: Seed2.0 Lite acts as the annotator, receiving the audio, a de-punctuated transcript, and a captioning prompt. The prompt enforces strict output format (contiguous speaker IDs, language preservation, etc.) and includes dataset-tailored examples. A style-persona library provides soft priors for three media families, guiding the description of stable speaker traits versus transient delivery without introducing non-audible labels.
• Data refinement:
  • Waveform filtering: Speech samples are removed if PESQ < 2.0, STOI < 0.85, SI-SDR < 0, or MOS < 2.5 (using DNSMOS and torchaudio-SQUIM). Duration filter keeps segments between 1 and 120 seconds.
  • Caption normalization: SwanVerifier checks gender/age consistency against the vocal stream. Punctuation is regularized using alignment data, preserving expressive marks (exclamation, question) when they match pause evidence. Caption validity checks enforce correct speaker indices, matched spans, and transcript agreement.
  • Human verification: Auditors review transcription accuracy, caption quality, audio quality, and expressiveness. They correct ASR errors, speaker attribution, omitted effects, and over-interpretation. Expressiveness is audited via group-wise best–worst comparison within matched groups, selecting the most and least expressive candidates based on naturalness, emotion strength, prosodic variation, and contextual appropriateness.

Method

The authors leverage a comprehensive data processing pipeline to generate fine-grained captions for complex multi-speaker speech and audio data. As shown in the figure below:

The pipeline consists of four main blocks: coverage design, SwanData-Speech preprocessing, caption annotation, and data refinement. The coverage design incorporates both media-style real-world data and targeted synthetic subsets to ensure diverse speaker densities, recording conditions, and rare scenarios like elderly speech and challenging pronunciations. In the preprocessing stage, the authors separate vocal and residual background streams, perform speaker diarization, and apply ASR followed by alignment to prepare clean speech spans and reliable text anchors. The caption annotation stage utilizes a large multimodal model to convert raw audio and transcripts into unified, multi-level captions containing environment, speaker, and content fields, guided by a style-persona library. Finally, data refinement applies waveform filtering, caption normalization, and human verification to ensure high acoustic quality and expressiveness.

SwanTale operates on continuous acoustic latents produced by SwanVAE, a 48 kHz waveform-latent autoencoder. The architecture of SwanVAE is detailed in the following figure:

The encoding path uses a local anti-aliased convolutional encoder and a Gaussian VAE bottleneck, while the decoder employs a Transformer Resampling Block to synthesize waveform patches. To balance reconstruction fidelity and learnability, the authors apply reconstruction and adversarial training objectives, alongside latent alignment objectives such as generative alignment and acoustic readouts to shape the posterior mean for the downstream flow model.

The core generation backbone is a non-causal flow-based Transformer with flow matching, which avoids autoregressive instabilities. As illustrated in the figure below:

The architecture maps inputs for both zero-shot and instruct tasks to a latent audio trajectory. The conditioning stack separates caption-level control from text alignment, utilizing a Qwen-family text encoder for captions and a lightweight Transformer for spoken content. To handle heterogeneous acoustic patterns across speech, environmental audio, and music, the authors introduce Unified MoE, a caption-conditioned dynamic-capacity sparse feed-forward module. Unified MoE employs a task router for sample-level shared experts and an audio router for frame-level dynamic Top-P routing, allocating model capacity based on the generation task, acoustic state, and diffusion time.

The training process follows a curriculum learning strategy. The authors first train a zero-shot base model, then adapt it to dense caption conditioning on clean speech using dense feed-forward layers. Subsequently, they expand to the full caption mixture and introduce Unified MoE, followed by supervised fine-tuning on high-expressiveness and high-quality subsets. To address recurring errors in pronunciation and speaker-attribute control, the authors apply reward-guided GRPO post-training. This stage converts the deterministic flow ODE into a marginal-preserving SDE for stochastic trajectory exploration, optimizing task-specific rewards like phoneme accuracy and speaker similarity while preserving multi-speaker and audio capabilities through supervised anchor replay. During inference, a two-stage decomposed classifier-free guidance rule is applied to separately control content consistency and acoustic attributes.

Experiment

The evaluation spans SwanVAE reconstruction across speech, singing, general audio, and music, zero-shot TTS on monologue and dialogue, and instruct generation via InstructTTSEval, SwanBench-Scene, and SwanBench-Caption. SwanVAE delivers competitive vocal reconstruction with a compact 25 Hz latent representation, while SwanTale achieves top timbre consistency and expressiveness in zero-shot settings, with gains driven by expressive data filtering and GRPO post-training. In instruct tasks, SwanTale leads on explicit acoustic control and scene appropriateness, though role-play remains a challenge, and ablations confirm that the Unified MoE and a larger caption encoder substantially improve instruction accuracy, acoustic quality, and expressiveness.

The SwanData-Caption schema structures audio captions into three fields: Environment describes the scene-level acoustic backdrop and recording context, Speakers catalogs each speaker's perceived persona and stable vocal habits, and Content presents the spoken dialogue with inline annotations of local delivery style and audio effects. Environment includes persistent background sounds like traffic or wind and recording-space cues such as reverberation. Content wraps speech in speaker tags and local audio effects in effect tags, with notes on changes in emotion, volume, or pace.

The table compares latent configurations of audio autoencoders and codecs, spanning continuous and discrete representations with varying frame rates, latent sizes, and nominal bitrates. SwanVAE uses a 25 Hz continuous latent with 96 dimensions, yielding 38.40 kbps, which is higher than most continuous baselines but lower than SAME-L's 44.10 kbps. Discrete codecs achieve much lower nominal rates (7.75–24.00 kbps) by operating at high frame rates with residual vector quantization. SwanVAE produces 96-dimensional continuous latents at 25 Hz, corresponding to a nominal rate of 38.40 kbps. Discrete codecs DAC and EnCodec reach nominal rates as low as 7.75 kbps through high frame rates (86–150 Hz) and multi‑codebook quantization. Continuous representations exhibit frame rates from 10.77 Hz to 25 Hz, with nominal rates ranging from 12.80 to 44.10 kbps.

On speech, SwanVAE achieves the best PESQ and MCD while staying close to top baselines on STOI and ViSQOL. On singing voice, it ranks first in PESQ, STOI, and MCD, and second in ViSQOL. The consistently low MCD across both domains indicates that the 25 Hz latent representation preserves vocal spectral structure effectively. SwanVAE leads speech reconstruction with the highest PESQ (4.17) and lowest MCD (0.96), while its STOI and ViSQOL are near the strongest competitors. For singing voice, SwanVAE obtains the best PESQ, STOI, and MCD, and the second-best ViSQOL. Low MCD values on both speech and singing voice suggest the 25 Hz latent rate retains fine spectral detail despite the low frame rate.

On general audio, SwanVAE achieves the highest ViSQOL and the second-lowest LSD, behind only Stable Audio Open 1.0 on LSD. For music, EnCodec leads both metrics, while SwanVAE ranks second on ViSQOL and third on LSD. The same SwanVAE checkpoint is used across all domains without domain-specific selection. SwanVAE attains the best ViSQOL and second-best LSD on general audio, outperforming all other systems except Stable Audio Open 1.0 on LSD. On music, SwanVAE places second on ViSQOL and third on LSD, with EnCodec achieving the top scores on both metrics.

SwanTale achieves the best timbre consistency, expressive richness, and expressive hierarchy in both monologue and dialogue zero-shot TTS, and leads SpeechJudge in dialogue. However, other systems obtain better content accuracy and sound fidelity: FishSpeech in monologue and SoulX-Podcast in dialogue. The improvements over SwanVoice come from a training recipe that adds filtered high-expressiveness data, ASR-based pronunciation checks, reward-conditioned quality control, and GRPO. SwanTale ranks first in Timbre Consistency, Expressive Richness, and Expressive Hierarchy in both monologue and dialogue settings. In monologue, FishSpeech achieves the lowest Content Error and highest Sound Fidelity among compared systems. In dialogue, SoulX-Podcast performs best on Sound Fidelity and Content Error, while SwanTale leads on SpeechJudge. SwanTale consistently improves over SwanVoice on all five reported metrics in both zero-shot settings. The gains are attributed to high-expressiveness data, ASR-based filtering, reward-conditioned quality control, and GRPO.

SwanVAE, a continuous audio autoencoder, is compared against discrete codecs and other models on reconstruction across speech, singing, general audio, and music; it consistently preserves vocal spectral structure, achieving top perceptual quality on speech and singing voice while remaining competitive on general audio and music. SwanTale is evaluated on zero-shot TTS expressiveness in monologue and dialogue, where it leads in timbre consistency and expressive hierarchy but trails in content accuracy and sound fidelity, with gains attributed to high-expressiveness data, ASR-based filtering, and reward-conditioned quality control.