One-sentence Summary

StepFun introduces Step-Audio-EditX, the first open-source LLM-based reinforcement learning audio model that achieves expressive, iterative audio editing and zero-shot TTS by leveraging only large-margin synthetic data in post-training, thereby circumventing embedding-based priors and outperforming MiniMax-2.6-hd and Doubao-Seed-TTS-2.0 in emotion editing.

Key Contributions

• Step-Audio-EditX is the first open-source LLM-based reinforcement learning audio model for expressive and iterative editing of emotion, speaking style, and paralinguistics, surpassing MiniMax-2.6-hd and Doubao-Seed-TTS-2.0 in emotion editing and fine-grained control tasks.
• Post-training on large-margin synthetic data alone controls emotion and speaking style, circumventing the need for embedding-based priors or auxiliary modules.
• The large-margin learning approach enables iterative control and high expressivity across voices, representing a fundamental pivot from conventional representation-level disentanglement.

Introduction

Recent zero-shot text-to-speech systems can generate natural speech conditioned on a reference prompt, but they lock attributes such as emotion, style, accent, and timbre to that prompt, preventing independent control. Prepending style or emotion instructions to the input text offers partial controllability, yet it often fails to disentangle attributes when the cloned voice cannot reliably follow the instruction. The authors propose a data-driven approach that bypasses the need for adversarial training or complex representation modeling. They design a pipeline that creates high-quality training pairs where linguistic content is held constant while a target attribute varies distinctly, then post-train a large language model on these pairs with large-margin data. The resulting model, Step-Audio-EditX, is the first open-source LLM-based audio model to support expressive and iterative editing of emotion, speaking style, and paralinguistics, alongside robust zero-shot TTS, without additional adapter modules. Iterative edits can progressively strengthen or weaken the target attribute, and the method extends to speed adjustment, denoising, and silence trimming.

Dataset

The authors construct the post‑training dataset for Step‑Audio‑EditX from two main branches: supervised fine‑tuning (SFT) data and reinforcement learning (RL) preference data. All data work targets zero‑shot text‑to‑speech (TTS) and audio editing capabilities.

SFT data

• Zero‑shot TTS – A high‑quality, professionally annotated in‑house corpus covering Chinese and English, with a small amount of Cantonese and Sichuanese for dialect support. The dataset spans approximately 60,000 unique speakers, capturing intra‑speaker and inter‑speaker variation to ensure diverse expressiveness and robust zero‑shot performance.
• Emotion and speaking style editing – A large‑margin synthetic approach: voice actors record one ~10‑second clip per emotion/style combination. These clips are used in a zero‑shot cloning pipeline (StepTTS) to build triplets of (text prompt, neutral audio, emotional/style audio). A small human‑annotated scoring model rates each triplet on a 1‑10 scale; only samples with a margin score above 6 (adjusted per emotion/style) are retained. This strategy converts complex emotion descriptions into comparative pairs while keeping data collection cost low.
• Paralinguistic editing – A semi‑synthetic construction using the NVSpeech corpus, which contains rich paralinguistic annotations (e.g., breathing, laughter, filled pauses). Quadruplets are formed by pairing the original NVSpeech audio/text with a version where paralinguistic tags are removed from the text and a neutral audio is generated via voice cloning. Since the task is time‑domain editing with large intrinsic margins, no margin scoring is applied and a small set of quadruplets suffices to elicit the capability.

RL preference data

• Human annotation – Real‑world user prompts and audio are collected. The SFT model generates 20 candidate responses per prompt. Human annotators rate each response on a 5‑point scale for correctness, prosody, and naturalness. Only chosen/rejected pairs with a score margin greater than 3 are selected.
• LLM‑as‑a‑Judge – A comprehension model scores emotion and style editing responses on a 1‑10 scale. Preference pairs are formed from these scores, keeping only pairs where the margin exceeds 8 points.

Usage The SFT subsets are used directly to instil zero‑shot TTS and diverse editing skills. The RL preference pairs (both human and LLM‑judge) are employed to train the reward model and subsequently for PPO alignment. No explicit mixture ratios are reported; the focus is on large‑margin filtering and synthetic data construction to avoid costly manual annotation.

Method

The system architecture integrates three core components: a dual-codebook audio tokenizer, an audio large language model (LLM), and an audio decoder. The audio tokenizer converts reference or input audio into discrete tokens using two parallel codebooks. A linguistic codebook operates at 16.7 Hz with a vocabulary size of 1024, while a semantic codebook runs at 25 Hz with 4096 codes, interleaved in a 2:3 ratio. This dual-codebook design intentionally retains a substantial amount of emotional, prosodic, and paralinguistic information, which would be a limitation for pure content disentanglement but becomes a useful property for validating the large-margin post-training strategy.

The audio LLM is a 3B-parameter model initialized from a text-based LLM to leverage its strong language capabilities. It is trained on a blended dataset that mixes text and audio dual-codebook tokens in a 1:1 ratio. The model processes text tokens and their corresponding audio tokens in a chat format, generating only dual-codebook token sequences as output. This design allows the model to simultaneously handle zero-shot text-to-speech and a variety of editing tasks under a unified framework, directly benefiting from post-training techniques developed for text LLMs.

The audio decoder reconstructs waveforms from the predicted dual-codebook tokens. It consists of a flow matching module built on a diffusion transformer backbone and a BigVGANv2 vocoder. The flow matching module generates Mel spectrograms conditioned on the output audio tokens, reference audio, and speaker embedding, after which the vocoder converts them to audio. The flow matching module is trained on 200,000 hours of high-quality speech, yielding substantial improvements in pronunciation accuracy and timbre similarity.

To enable the model to perform emotion and speaking style editing, the authors devise a large-margin synthetic data pipeline. Voice actors record a single approximately 10-second clip for each emotion and style. For each combination, a triplet $\langle text_{prompt}, audio_{neutral}, audio_{emotion,style} \rangle$⟨textprompt​,audioneutral​,audioemotion,style​⟩ is created by using the corresponding emotional and neutral clips from the same speaker as prompt audio and synthesizing the output through a zero-shot voice cloning interface with a text instruction describing the target attribute. A margin scoring model, trained on a small human-annotated dataset, evaluates each pair on a 1–10 scale, and samples are selected using a margin score threshold (with 6 as the universal lower bound, adjusted per emotion and style). Because every triplet is generated from the same emotional or stylistic text prompt, the model is forced to focus on the variation in emotion and style during supervised fine-tuning.

For paralinguistic editing, a “semi-synthetic” strategy constructs quadruplets $\langle text_{without\_tags}, audio_{without\_tags}, text_{nv\_source}, audio_{nv\_source} \rangle$⟨textwithout_tags​,audiowithout_tags​,textnv_source​,audionv_source​⟩. The original NVSpeech audio and transcript serve as the target output, while the input is generated by cloning the original transcript after removing paralinguistic tags, which produces an audio version lacking those cues. Since paralinguistic editing operates in the time domain and exhibits large intrinsic margin differences, no margin scoring model is required; a small set of quadruplets suffices to elicit the desired editing capabilities.

The post-training alignment proceeds in two stages: supervised fine-tuning (SFT) followed by reinforcement learning. In SFT, the model is trained with distinct system prompts in a chat format. For zero-shot TTS, the prompt waveform is encoded into dual-codebook tokens, detokenized into a string, and embedded in the speaker information of the system prompt; the user prompt contains the text to synthesize, and the model responds with the generated audio tokens. For editing tasks, a unified system prompt defines all operations, and the user prompt provides the original audio and a descriptive editing command. The model is fine-tuned for one epoch with a learning rate decaying from $1 \times 10^{-5}$1×10−5 to $1 \times 10^{-6}$1×10−6.

Reinforcement learning further improves stability in zero-shot TTS and editing expressiveness, especially when the source prompt and target output differ strongly in emotion or style (e.g., happy to sad, loud to whisper). A reward model is initialized from the 3B SFT model and trained with a combination of human-annotated and LLM-as-a-judge-generated large-margin data using the Bradley-Terry loss. It operates as a token-level reward model on dual-codebook token pairs, avoiding the need to decode tokens back to audio during reward computation. The reward model is fine-tuned for one epoch with a cosine-decayed learning rate starting at $2 \times 10^{-5}$2×10−5 and a lower bound of $1 \times 10^{-5}$1×10−5. Proximal policy optimization (PPO) is then conducted using the most challenging prompts for the SFT model. The critic model is warmed up for 80 steps before the actor begins training. The actor uses an initial learning rate of $1 \times 10^{-6}$1×10−6, cosine decay to $2 \times 10^{-7}$2×10−7, a PPO clip threshold of $\epsilon = 0.2$ϵ=0.2, and a KL divergence penalty with coefficient $\beta = 0.05$β=0.05.

The large-margin methodology extends naturally to downstream tasks. For speed editing, triplets $\langle text, audio_{source}, audio_{faster,slower} \rangle$⟨text,audiosource​,audiofaster,slower​⟩ are created by speed perturbation using the SoX toolkit, and the large token length differences enable effective editing with SFT alone. For denoising, a triplet $\langle text, audio_{augment}, audio_{source} \rangle$⟨text,audioaugment​,audiosource​⟩ is constructed with additive noise and reverberation applied to the source. For silence trimming, the triplet $\langle text, audio_{source}, audio_{trimming} \rangle$⟨text,audiosource​,audiotrimming​⟩ uses Silero-VAD to extract and concatenate speech segments, removing silent portions. These extensions demonstrate how a sufficient margin between paired samples allows the model to rapidly acquire new editing capabilities through SFT, with reinforcement learning optionally applied to challenging cases.

Experiment

The evaluation is conducted on a benchmark that uses an LLM judge to assess emotion, speaking style, and paralinguistic editing of zero-shot cloned speech from multiple speakers. Iterative editing significantly improves emotion and style accuracy, and even a single paralinguistic edit achieves strong performance, with consistent generalization to closed-source TTS systems; an ablation study confirms that the large-margin method drives the gains. The model often outperforms the native emotion controls of closed-source models, and the overall approach delivers high accuracy and robust generalization across all editing tasks.

Applying the editing model to zero-shot cloned audio yields a large leap in emotion and speaking style accuracy. The first editing iteration provides the most substantial gain, and successive rounds bring further, smaller improvements. Accuracy continues to rise even when the prompt audio is kept fixed, confirming that the editing mechanism itself drives the enhancement. The first editing iteration produces the largest accuracy jump for both emotion and speaking style. With a fixed prompt audio, accuracy still improves with each editing round, showing the editing process is robust. Speaking style accuracy follows a similar trend, rising sharply after the initial edit and continuing to increase through subsequent iterations.

Closed-source TTS models exhibit moderate in-context emotional and stylistic expression, but a single editing pass with the proposed model yields large accuracy improvements for both emotion and speaking style. The gains are most pronounced in the first iteration, and subsequent edits further refine the output, with accuracy rising steadily through the third iteration. This pattern holds across all tested Chinese and English voice models, demonstrating strong generalization. The first editing iteration produces a dramatic boost in emotion and speaking style accuracy across every closed-source system. Speaking style accuracy sees especially large absolute gains from the initial zero-shot clone to the first edited version. Iterative editing continues to improve both metrics, though the incremental benefit diminishes after the first pass.

Step-Audio-EditX's zero-shot cloning achieves higher initial emotion accuracy than the closed-source MiniMax and Doubao clones. A single editing iteration substantially boosts emotion accuracy across all models, and the edited zero-shot clone outperforms the native emotion control outputs of the closed-source systems. Successive editing rounds continue to improve emotion accuracy, demonstrating the effectiveness of iterative refinement. Zero-shot cloning by Step-Audio-EditX yields higher baseline emotion accuracy than MiniMax and Doubao clones. One editing iteration provides large emotion accuracy gains for all models in both Chinese and English. Editing a zero-shot clone with one iteration surpasses the emotional accuracy of the closed-source models' native emotion control. Iterative editing further improves emotion accuracy, with each additional round contributing incremental gains.

A single editing iteration with Step-Audio-EditX substantially improves paralinguistic reproduction, raising the average LLM-Judge score from 1.91 to 2.89 on a 1–3 scale. This gain holds for both Chinese and English, and the post-edit quality rivals that of closed-source models’ native paralinguistic synthesis. One editing round raises paralinguistic scores from a low baseline to near the ceiling, with Chinese improving from 1.80 to 2.89 and English from 2.02 to 2.89. After editing, the paralinguistic performance matches that of closed-source models when they synthesize paralinguistic speech directly, demonstrating strong generalization.

A single paralinguistic editing iteration with Step-Audio-EditX dramatically improved paralinguistic reproduction across closed-source TTS models in both Chinese and English. Scores rose from around 1.7 for unedited audio to roughly 2.9 on a 1–3 scale after editing, reaching quality comparable to the models' native paralinguistic synthesis. Replacing paralinguistic tags with onomatopoeic words before editing also gave strong results, though slightly lower than editing the label-removed audio directly. One editing iteration boosted paralinguistic accuracy from ~1.7 to ~2.9 on a 1–3 scale for all tested models and languages. Editing the initial audio without paralinguistic labels slightly outperformed the onomatopoeia substitution approach (e.g., 2.90 vs. 2.80 for Chinese MiniMax). Post-editing paralinguistic quality was comparable to the closed-source models' own native paralinguistic synthesis.

The editing model delivers a substantial leap in emotion and speaking style accuracy for zero-shot cloned audio, with the first editing iteration providing the most dramatic gain and subsequent rounds yielding further, smaller improvements. This pattern holds across both its own clones and closed-source TTS systems, where a single edit consistently raises emotion and paralinguistic reproduction to levels that rival or exceed the models’ native control capabilities. The enhancement is robust even when the prompt audio remains fixed, confirming that the iterative editing mechanism itself drives the improvement, and the approach generalizes strongly across Chinese and English.