One-sentence Summary

AuK, an open-source foundational model introduced by the authors, unifies speech generation and editing through natural-language instructions and audio context, leveraging $3.03 \times 10^9$3.03×109 instruction–audio instances and $1.95 \times 10^6$1.95×106 hours of supervision across five task families, combining a multimodal large language model, a jointly trained VAE, and a hybrid rectified-flow Transformer with dual-stream MMDiT and single-stream DiT blocks, followed by post-training with human-feedback preference optimization and reward-based reinforcement learning, and a distilled AuK-Flash variant that achieves $4\text{-step}$4-step inference with a $4.5\times$4.5× wall-clock speedup, leading on zero-shot and instruction-controlled speech generation and editing while remaining competitive on signal-level restoration.

Key Contributions

• Introduces AuK, an open-source foundational model that unifies speech generation and editing through a shared natural-language instruction and audio context interface, trained on approximately 3.03 billion instruction–audio instances and 1.95 million hours of supervision across five task families.
• Combines a multimodal large language model for semantic conditioning, a VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer with dual-stream MMDiT blocks followed by single-stream DiT blocks, enabling text-only generation and reference-conditioned editing within one backbone.
• Presents AuK-Flash, a distilled variant using consistency initialization and task-routed Decoupled DMD, which performs 4-step inference without classifier-free guidance and achieves a 4.5× wall-clock speedup over the full model; experiments show leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, with competitive results on signal-level restoration tasks.

Introduction

Recent speech systems have moved beyond conventional text-to-speech toward zero-shot voice cloning, instruction-controlled synthesis, and flexible editing, yet real-world requests often combine these capabilities, such as altering style, replacing segments, or restoring audio. Prior work relies on separate task-specific models, which fragments user experience and duplicates effort, while unifying them is hard due to differing output constraints (generation vs. editing), varied conditioning interfaces (text-only vs. audio-plus-text), and heterogeneous supervision needs. The authors introduce AuK, a unified foundational model that maps natural-language instructions and optional audio context to target waveforms, combining an MLLM semantic encoder, a jointly trained audio VAE, and a hybrid flow Transformer. They address training challenges with generation-only warm-up, joint pre-training, human-feedback preference optimization for open-ended editing, and reward-based RL for generation, then distill into AuK-Flash for 4-step inference and a 4.5x speedup. Their experiments show leading performance on speech generation and instruction-guided editing benchmarks, with competitive results on restoration tasks, and they release source code and weights.

Dataset

The authors construct a large-scale, multi-task pre-training corpus for a unified audio model. The dataset is organized into five task families: speech generation, acoustic editing, paralinguistic editing, content editing, and enhancement and separation. All tasks share a common interface of a natural-language instruction, optional input audio, and a target waveform. In total, the corpus comprises approximately 3.03 billion instruction-audio instances, providing 1.95 million hours of effective audio supervision.

Dataset Composition and Sources

• The corpus is built from a mix of public datasets, in-house recordings, and synthetic data generated by auxiliary models.
• The authors use source separation, speech enhancement, and quality filtering to clean raw audio before constructing training pairs.
• For tasks requiring paired supervision, the authors generate targets using deterministic signal processing, neural synthesis models, or forced alignment tools.

Key Details for Each Subset

• Speech Generation: Includes two forms of supervision.
  • Zero-shot TTS is transcript-free. For each speaker with n utterances, the authors enumerate all unordered pairs and create n x (n-1) bidirectional instances, where one utterance serves as the acoustic prompt and the other as the synthesis target. The prompt transcript is never provided.
  • Instruct TTS uses a bilingual speech pool. Each utterance is annotated by Qwen3-Omni with a free-form caption and structured attributes (gender, age, speaking rate, clarity, fluency, vocal state, intonation, loudness, timbre, pitch, accent, emotion, personality). The description and target text form the input instruction.
• Acoustic Editing: Uses deterministic transformations on source utterances. Targets are created at five speaking-rate multipliers (0.5x, 0.75x, 1.25x, 1.5x, 2.0x), six loudness offsets (+/-5, +/-10, +/-15 dB), and six pitch shifts (+/-1, +/-2, +/-3 semitones). Each transformed waveform is paired with its source and a natural-language instruction specifying the attribute and magnitude.
• Paralinguistic Editing: Covers emotion, timbre, accent, nonverbal vocalization, and whisper-style editing.
  • Emotion editing uses eight target emotions (angry, happy, sad, fearful, surprised, disgusted, calm, excited). Qwen3-TTS-CustomVoice synthesizes an expressive reference, and IndexTTS2 conditions on the original utterance for speaker identity and the reference for emotion.
  • Timbre editing uses the X-VC training corpus, built with SeedVC-Small, containing four aligned source-target pairs per group. All waveforms are enhanced and standardized to 24 kHz.
  • De-accenting uses an in-house corpus of 13 Chinese dialect and regional-accent categories. CosyVoice2 synthesizes a standard Mandarin reference, and OmniVoice reconstructs the source with the reference as a condition.
  • Nonverbal editing normalizes annotations into 39 event types. Qwen3-ForcedAligner locates event spans, and F5-TTS reconstructs event-free waveforms.
  • Whisper-style conversion uses a public Mandarin corpus with parallel normal and whispered speech, retaining only pairs with WER = 0 and resampling to 24 kHz.
• Content Editing: Includes speech and lyric editing.
  • Speech content editing uses an LLM to generate annotations for insertion, deletion, and substitution. Qwen3-ForcedAligner provides word-level alignments, and F5-TTS performs masked infilling. Samples are retained only if the synthesized waveform has low word error rate against the target transcript.
  • Lyric editing uses high-quality dry-vocal recordings. An LLM generates source-target lyric pairs. Chinese replacements preserve character count and are checked at pinyin level; English replacements preserve word boundaries and count. YingMusic-Singer-Plus synthesizes the target vocal, and low-WER samples are retained.
• Enhancement and Separation: Covers speech enhancement, multi-speaker separation, and music processing.
  • Speech enhancement applies randomly sampled degradations: background noise, localized events, reverberation (measured and simulated), bandwidth limitation, clipping, dropout, coloration, underwater filtering, and DC offset. Training targets are not always clean speech; selective targets remove only the named corruption.
  • Multi-speaker separation constructs conversational mixtures with turn-taking, pauses, interruptions, and overlap. Instructions identify speakers via content, order, loudness, or timestamp.
  • Music enhancement and separation uses source-separation models to obtain vocal and accompaniment stems. Native-song examples use the original song as input with time-aligned stem crops as targets. Scene-based examples mix speech, singing, and background music with controlled timing and gain.

How the Data Is Used

• The authors use the entire corpus to train a single model across all five task families, with the unified instruction-audio-target interface.
• The training mixture is not explicitly detailed in the provided text, but the authors state that all tasks share the same interface, implying a joint training setup across all subsets.
• The authors apply quality control at multiple stages, including MOS-based filtering, ASR cross-validation, speaker identity verification, and word error rate checks on synthesized targets.
• For acoustic editing, all transformations are validated and peak protection is applied to prevent clipping.
• For whisper-style conversion, normal-speech inputs are normalized to a target RMS of -24 dBFS to match the broader training corpus.
• The authors do not describe a separate validation or test split in the provided text; the focus is on pre-training data construction.

Method

The authors propose AuK, a unified audio generation and editing framework that processes textual instructions and optional reference audio to synthesize or modify waveforms. The overall architecture integrates three complementary components: a multimodal large language model (MLLM) for semantic conditioning, a variational autoencoder (VAE) for acoustic conditioning, and a hybrid transformer backbone for latent prediction.

As illustrated in the framework diagram, the system routes tasks based on the availability of reference audio. For tasks with reference audio, such as zero-shot text-to-speech and content editing, the input audio is processed by both the audio encoder of the MLLM and the VAE encoder. The MLLM jointly encodes the textual instruction and audio context, while the VAE maps the reference audio into a latent space. For text-only tasks like instruct TTS, the semantic condition is derived solely from the user instruction, and the acoustic stream contains only the noisy target latents.

To construct the semantic condition, the authors leverage Qwen2.5-Omni as the semantic encoder and aggregate its layer-wise hidden states to capture complementary linguistic and cross-modal cues. The hidden state at layer $\ell$ℓ is computed as:

These representations are aggregated into the final semantic condition $\mathbf {c} _ { \mathrm{sem} }$csem​ using a learnable weighted sum:

where $L$L is the number of MLLM layers and $w _ { \ell }$wℓ​ is an unconstrained learnable scalar.

For the acoustic condition, a flow-augmented audio VAE encodes a 24 kHz reference waveform $\mathbf {x} _ { \mathrm{ref} }$xref​ into a 64-dimensional latent sample $\mathbf {z} _ { \mathrm{ref} }$zref​ at 50 Hz. The acoustic condition $\mathbf {c} _ { \mathrm{ac} }$cac​ is set to $\mathbf {z} _ { \mathrm{ref} }$zref​ when reference audio is available, and empty otherwise.

The transformer backbone follows a FLUX-style hybrid design, comprising $M$M dual-stream multimodal diffusion transformer (MMDiT) blocks followed by $N$N single-stream diffusion transformer (DiT) blocks. The semantic condition, optional acoustic condition, and noisy target latent $\mathbf {z} _ { t }$zt​ at flow time $t$t are mapped to two input streams:

The dual-stream MMDiT blocks update the semantic and acoustic streams jointly. Each block utilizes stream-specific query, key, value, and residual projections, applying rotary position embeddings before concatenating the attention tensors for joint attention. This enables bidirectional interaction while preserving separate residual pathways. The updated streams are then concatenated and processed by the single-stream DiT blocks to predict the flow velocity $\widehat { \mathbf {v} } _ { t }$vt​:

where $\mathbf {e} _ { t }$et​ is the projected flow-time embedding.

Unified pre-training follows a two-stage curriculum using a flow-matching objective. The model interpolates between Gaussian noise $\mathbf {z} _ { 0 }$z0​ and the clean target latent $\mathbf {z} _ { 1 }$z1​ to construct $\mathbf {z} _ { t } = (1 - t) \mathbf {z} _ { 0 } + t \mathbf {z} _ { 1 }$zt​=(1−t)z0​+tz1​, minimizing the masked mean-squared error between the predicted and target velocities. To support classifier-free guidance, the authors apply hierarchical condition dropout, randomly dropping acoustic and semantic conditions during training.

Post-training aligns the model with human preferences and automatic rewards. Editing preference optimization employs a flow-based diffusion policy optimization score and an ordinal listwise objective to align diverse editing behaviors with subjective judgments. Generation reinforcement learning utilizes flow-group relative policy optimization to improve zero-shot and instruct TTS. For zero-shot TTS, rewards combine content correctness and speaker similarity, while instruct TTS relies on a dedicated style-consistency model.

To accelerate inference, the authors distill the full model into a four-step, classifier-free guidance-free student. The distillation process begins with consistency initialization, training the student to map any noisy state directly to the trajectory endpoint under teacher guidance. This is followed by task-routed decoupled distribution matching distillation. To prevent degradation in separation tasks, separation examples are routed to a supervised clean-prediction objective, excluding them from the distribution matching update to preserve the model's ability to isolate specific audio sources.

Experiment

The experiments evaluate AuK and AuK-Flash across VAE reconstruction, speech generation, and speech editing, with AuK-VAE achieving top scores on all fidelity metrics in speech, general audio, and music. In zero-shot TTS and instruction-controlled synthesis, the full model generally yields lower recognition error and better edit fidelity, while the Flash variant often improves perceptual quality under faster inference. For editing, AuK excels in linguistic preservation and instruction following, whereas AuK-Flash tends to better retain speaker identity and perceptual metrics across enhancement, separation, and super-resolution tasks. Additional observations highlight cross-utterance training benefits, emergent cross-task transfer such as whisper synthesis and English de-accenting, and the persistent need for prompt enhancement to handle free-form instructions reliably.

The second pre-training stage uses a fixed per-batch sampling mixture across five task families, with speech generation receiving the highest probability and acoustic editing the lowest. The distribution is heavily weighted toward generation, content editing, enhancement/separation, and paralinguistic editing, while acoustic editing is a minor component. Speech generation dominates the sampling mixture at 28.10%. Content editing, enhancement/separation, and paralinguistic editing each receive roughly 21-23% of samples. Acoustic editing is assigned a markedly lower sampling probability of 3.96%.

AuK and AuK-Flash share the same audio preparation, duration estimation, latent representation, and VAE decoder, but differ in checkpoint type, numerical precision, solver, function evaluations, and classifier-free guidance. AuK uses an EMA checkpoint with 32 function evaluations and CFG scale 2.0, while AuK-Flash uses a task-routed distilled checkpoint with 4 function evaluations and no CFG, yielding a 4.5x speedup. Both output at 24 kHz and use the same sway coefficient. AuK-Flash uses a distilled checkpoint with 4 function evaluations and no guidance, while AuK uses an EMA checkpoint with 32 evaluations and CFG scale 2.0. The 4.5x speedup is measured under identical hardware, output duration, and batch size, comparing the 4-step CFG-free Flash sampler against the 32-NFE CFG-2.0 AuK sampler. Both variants share the same prompt enhancement, audio preparation, duration estimation, latent representation, and VAE decoder, and output at 24 kHz.

AuK and AuK-Flash consistently outperform prior systems across speech generation, editing, enhancement, and separation benchmarks. The full model excels in linguistic accuracy and edit fidelity, while the Flash variant often achieves better perceptual quality and competitive instruction following. AuK reduces average recognition error to 2.65% and raises speaker similarity to 0.795, surpassing the strongest prior baselines in zero-shot TTS. AuK improves content editing joint success from 76.46% to 91.83% and prosody editing from 26.50% to 71.33% relative to Ming-UniAudio. AuK-Flash achieves the highest EMR on MMAE-Speech at 13.85%, outperforming the full model and prior baselines in satisfying all editing rubrics simultaneously. In speech enhancement, AuK-Flash attains the highest UTMOS scores on both DNS and CHiME-4, while both variants match the best speaker similarity at 0.99. On Libri2Mix, AuK preserves linguistic content better with the lowest WER and PER, whereas AuK-Flash leads in predicted perceptual quality with the highest OVRL and UTMOS.

AuK-VAE consistently outperforms all compared VAE models across speech, general audio, and music, achieving the best scores on every reconstruction metric in each domain. The gains are evident in perceptual quality, intelligibility, and spectral fidelity, with the largest improvements seen in speech reconstruction. AuK-VAE ranks first on all four metrics in every domain, surpassing the second-best model by a clear margin in speech PESQ and STOI. For general audio, AuK-VAE shows notably higher intelligibility (STOI) and lower spectral distances than the alternatives, while other models vary more across metrics. In music, AuK-VAE maintains the best balance of perceptual and spectral fidelity, with the smallest mel and STFT distances among all methods.

The experiments evaluate AuK and AuK-Flash across speech generation, editing, enhancement, and separation, using a pre-training mixture where speech generation dominates sampling and acoustic editing is minimal. AuK-Flash, a distilled 4-step CFG-free variant, achieves a 4.5x speedup over the full AuK model while often delivering better perceptual quality, though the full model excels in linguistic accuracy and edit fidelity. Both variants consistently outperform prior baselines, with AuK reducing recognition error and improving editing success rates, and AuK-Flash leading on joint editing satisfaction and enhancement quality. Additionally, the AuK-VAE reconstruction model outperforms all compared VAEs across speech, general audio, and music on every metric, with the largest gains in speech.