One-sentence Summary

StepFun, ACE, the Chinese University of Hong Kong, and University of California San Diego introduce StepAudio 3 Music, a large-scale long-form music generation model that combines a $50\,\text{Hz}$50Hz, $65536\text{-entry}$65536-entry single-codebook tokenizer with semantically informed self-supervised and multi-task training, a flow-matching diffusion Transformer predicting continuous VAE latents decoded to $48\,\text{kHz}$48kHz audio, and a Mixture-of-Experts autoregressive model using ABC notation for explicit planning (ABC-CoT), achieving via DPO the highest AudioBox Content Enjoyment, Content Usefulness, and Production Quality scores along with the highest MuQ-MuLan similarity among evaluated systems, plus a $1105$1105 Quality Elo on the preliminary Music Arena Vocals leaderboard.

Key Contributions

• Introduces StepAudio 3 Music, a large-scale music generation model that combines a 50-Hz, 65536-entry single-codebook tokenizer, a flow-matching diffusion Transformer for continuous latent prediction, and a VAE decoder for 48-kHz audio rendering, with tokenizer design informed by controlled comparisons of single-codebook VQ and Semantic and Acoustic RVQ.
• Presents explicit musical planning via ABC-CoT, where a Mixture-of-Experts autoregressive model generates a readable intermediate arrangement in ABC notation before predicting music tokens, using music-to-ABC and ABC-to-music training to connect symbolic plans to acoustic output and support text, notation, and reference-conditioned control.
• Achieves the highest AudioBox Content Enjoyment, Content Usefulness, and Production Quality scores, the highest MuQ-MuLan similarity among evaluated systems, competitive SongBench outcomes, and a Quality Elo of 1105 on the Artificial Analysis Music Arena Vocals leaderboard, behind only Suno V5.5 and Mureka, following progressive curriculum training, supervised fine-tuning, and direct preference optimization.

Introduction

Recent advances in music generation have enabled models to produce high-quality audio over long durations, with diffusion and flow-matching methods handling complex acoustic distributions and diffusion Transformers modeling long temporal contexts. However, as generation shifts from excerpts to full songs, models must coordinate decisions across multiple scales, from melodic development and section transitions to detailed timbre and vocal texture. Prior hierarchical systems, such as Seed-Music and ACE-Step 1.5, combine language models with diffusion renderers, while chain-of-thought approaches like MusiCoT and Qwen-Music explicitly plan aspects like melody or instrumentation. Yet these plans often remain opaque or limited in temporal structure, and direct control conditions for chords or melodies in prior work like JASCO and InstructME do not expose a model's internal arrangement decisions for user inspection or revision. Symbolic music representations, such as ABC notation used in ChatMusician, offer a readable interface but have not been tightly integrated with acoustic generation in long-form settings.

The authors introduce StepAudio 3 Music, a large-scale music generation model that supports explicit musical planning via ABC-CoT and open-domain text control. They compare single-codebook VQ, Semantic RVQ, and Acoustic RVQ at 25 Hz, finding that multi-codebook RVQ improves reconstruction but single-codebook VQ enables more stable autoregressive generation. Based on this, they design a 50-Hz, 65536-entry single-codebook tokenizer with semantically informed training, paired with a flow-matching DiT renderer that converts continuous VAE latents into 48-kHz audio. This separation lets the autoregressive model focus on musical sequence organization while the renderer handles acoustic detail, and the ABC-CoT planning mode provides a readable arrangement of chords, tempo, meter, and melody before synthesis, allowing creators to inspect and revise decisions prior to audio generation.

Dataset

The authors build a structured dataset pipeline to prepare large-scale music training data. The pipeline filters low-quality material and converts each retained song into a structured training record through several stages: frequency-band analysis, audio-event detection, source separation, language identification, multilingual lyrics transcription, and song-structure analysis.

• Acoustic filtering: Each audio file undergoes spectrum analysis to estimate effective bandwidth. This removes low-bandwidth recordings and those with suspected artificial upsampling. Next, multi-label audio event detection (AED) identifies speech, singing, and music, excluding non-musical recordings and samples with insufficient musical content before further processing.

• Source separation and transcription: Retained samples are processed with Demucs to separate vocals from accompaniment. The isolated vocal track is used for language identification (via MMS-LID) and lyrics transcription. Samples with confident language predictions are routed to language-specific ASR systems, with Chinese and English handled by FireRed ASR and other supported languages by WhisperX. Uncertain predictions lead to exclusion from lyrics annotation.

• Structured annotations: For each recognized segment, the authors keep the transcribed text, start and end timestamps, and ASR confidence scores. These scores serve as quality signals for filtering and task-specific data selection, allowing the pipeline to remove or label low-confidence samples without prematurely discarding the corpus.

• Song structure analysis: Timestamped lyrics are fed into SongFormer, which partitions each song into sections such as intros, verses, choruses, bridges, and outros. Each lyrics segment is aligned to its corresponding section based on temporal overlap with predicted boundaries.

• Final training record: Each record includes detected language, timestamped lyrics, section labels and boundaries, frequency-band measurements, and ASR confidence scores. These structured annotations support tasks like lyrics-to-song generation, song-to-lyrics recognition, and long-form song-structure modeling.

• Supervised fine-tuning (SFT) usage: The SFT corpus contains approximately 20B tokens, including direct music-token generation and generation preceded by explicit ABC-CoT planning. The ABC-CoT subset retains the notation plan as context for the music-token sequence. For general music-generation examples, samples are grouped by genre and ranked within each group using the SongBench Musicality score to retain high-quality examples without letting high-resource genres dominate selection.

Method

The authors leverage a shared autoregressive backbone to support open-domain text-controlled generation and explicit musical planning. As shown in the framework diagram, a Mixture-of-Experts decoder conditions on lyrics, a text prompt, and optional task-specific references to predict music tokens. The StepAudio Music Codec provides the interface between this discrete sequence and high-fidelity audio, separating long-range musical organization from acoustic rendering. When explicit planning is enabled, the autoregressive model first produces an ABC-CoT arrangement plan and incorporates it into the context for music-token generation. This two-pass factorization allows the model to produce an arrangement plan containing chords, tempo, time signature, and an ABC sequence before predicting the 50-Hz music-token sequence.

The StepAudio Music Codec comprises a music tokenizer and a flow-matching DiT detokenizer. The tokenizer converts 24-kHz audio into a 50-Hz stream of discrete tokens from a 65536-entry single codebook. Conditioned on these tokens, the detokenizer generates continuous VAE latents, which are decoded into 48-kHz waveform audio. To design an effective discrete bottleneck, the authors compare three candidate architectures at a common 25-Hz frame rate. The single-codebook VQ produces one semantic token per frame, offering the simplest prediction target. Semantic RVQ quantizes residuals within the same semantically supervised latent space, while Acoustic RVQ freezes the semantic tokenizer and quantizes the residual between an acoustic feature and the aligned semantic feature. The authors select the single-codebook VQ design because it maintains good musicality and avoids the error accumulation observed in multi-stream residual quantization, ultimately adopting a 50-Hz configuration to provide denser token-level supervision.

To support large-scale music training, the authors construct an automated data pipeline that progressively filters low-quality material and converts retained songs into structured training records. Raw audio first undergoes bandwidth analysis and audio-event detection to reject low-bandwidth or non-musical samples. Retained samples are separated into vocal and accompaniment stems using Demucs. The vocal stem is routed to language identification and multilingual transcription systems, such as FireRed ASR for Chinese and English, and WhisperX for other languages. Finally, SongFormer analyzes the global song structure, partitioning the audio into sections and aligning timestamped lyrics to produce structured training records containing language, section labels, and confidence scores.

The training process proceeds from the audio representation to the language model. The authors first train the StepAudio Music Tokenizer and the DiT renderer, then hold the codec fixed while optimizing the MoE language model. The tokenizer is trained through three successive stages. Stage 1 adapts the BEST-RQ self-supervised objective, where contiguous waveform spans are replaced with Gaussian noise before Mel extraction, and the model predicts clean targets at masked 50-Hz positions. Stage 2 introduces multi-task supervision over full-song inputs, utilizing CTC, Mel, and Chroma reconstruction objectives to adapt the representation into a structured space suitable for music tokenization. Stage 3 inserts a discrete bottleneck between the lower and upper 12-layer Conformer stacks, quantizing the continuous representation into a single codebook with 65536 entries. The joint training objective combines lyric recognition, Mel reconstruction, Chroma reconstruction, and vector-quantization losses.

Following tokenizer training, the flow-matching DiT renderer is trained on ground-truth music tokens and corresponding continuous latents from the StepAudio VAE. The discrete condition is aligned frame by frame with the 50-Hz VAE latents. Long-form generation is performed in 30-second chunks, where each subsequent chunk is conditioned on the final two seconds of the latent sequence generated for the preceding chunk to maintain local acoustic continuity.

With the codec fixed, the LLM is initialized from a Mixture-of-Experts text language model and trained with a three-stage curriculum. Stage 1 focuses on large-scale pretraining for lyrics-to-music generation and music-to-lyrics recognition. Stage 2 extends the context length and introduces ABC-CoT generation to teach the two-pass generation procedure, alongside music understanding and reference-conditioned tasks. Stage 3 concentrates on high-quality annealing for core creation tasks such as song generation, cover-song generation, and vocal-to-mix generation. Post-training further specializes the model through supervised fine-tuning and Direct Preference Optimization, utilizing expert pairwise preferences to align outputs with high-quality listening preferences.

Experiment

The evaluation covers the full StepAudio 3 Music pipeline, starting with ablations on the flow-matching detokenizer: the 50-Hz VAE is preferred over 25-Hz for direct reconstruction, but scaling the DiT from 0.9B to 8B yields no consistent gain, suggesting a tokenizer-imposed information ceiling, so the 0.9B model is adopted. Preference optimization via DPO improves all metrics over SFT, including musicality, perceived quality, and caption alignment. Adding ABC-CoT planning yields modest gains, while LLM-based editing of the symbolic plan provides larger improvements, especially in mixing, arrangement, and melody, supporting interpretable plans as an effective interface for musical knowledge. In the main comparison, StepAudio 3 Music leads in AudioBox-Aesthetics and MuQ-MuLan similarity, though SongBench ranks Mureka V9 higher, and blind arena evaluation places the model fourth; limitations include non-independent SongBench filtering, single-sample evaluation, and coverage restricted to vocal tracks.

StepAudio 3 Music's LLM is trained in three stages with decreasing token budgets and increasing context lengths, shifting from a constant to a cosine learning rate schedule. The training corpus is extensive, covering diverse music tasks and including a dedicated ABC notation corpus to enhance explicit musical control. Training progresses from 600B tokens of large-scale pretraining to 500B tokens of multi-task mid-training and finally 80B tokens of high-quality annealing. Context length doubles from 16,384 to 32,768 after the first stage, while batch size decreases from 768 to 512. The learning rate schedule shifts from constant in Stage 1 to cosine decay in later stages, with Stage 3 including a warm-up and a lower final learning rate. The training corpus includes over 100 million songs and about one trillion tokens, spanning lyrics-to-music generation, music-to-lyrics recognition, and ABC-based understanding and generation tasks.

The supervised fine-tuning corpus comprises about 20B tokens across six task subsets, each with distinct conditioning inputs, including direct generation and generation with explicit ABC-CoT planning. The inclusion of the ABC-CoT subset with notation plans into the SFT data supports a link between musical conditions and their audio realization. Data selection for general music examples uses genre grouping and SongBench musicality ranking to retain high-quality samples without genre dominance. The corpus spans six task subsets, with lyrics-to-music generation being the largest at 6B tokens, followed by lyrics-to-music with ABC-CoT and cover-song generation at 4B each. Conditioning formats vary from simple text prompts to combinations of lyrics, text, arrangement plans, reference songs, or dry vocal tracks. ABC-CoT subsets carry an explicit arrangement plan as context, reinforcing the relationship between musical conditions and acoustic output. General music examples are grouped by genre and ranked by SongBench musicality, balancing musical quality and genre representation.

StepAudio 3 Music leads in caption adherence and the AudioBox-Aesthetics metrics of content enjoyment, usefulness, and production quality, while Mureka V9 tops individual SongBench dimensions. The results indicate that StepAudio 3 Music excels in perceived quality and text alignment, but different evaluator families yield different rankings. StepAudio 3 Music scores highest on music-text similarity and on Content Enjoyment, Content Usefulness, and Production Quality. Mureka V9 leads the individual SongBench dimensions, showing the ranking depends on which evaluator is used. Production Complexity values are similar across systems, with Mureka V9 having the highest and MiniMax Music 3 the lowest.

In the Artificial Analysis Music Arena vocals leaderboard, StepAudio 3 Music achieves a Quality Elo of 1105 from blind preference votes, placing it in the upper-mid range among listed models. The top-ranked models are Suno V5.5 and Mureka V9, with StepAudio 3 Music trailing them by a modest margin. StepAudio 3 Music ranks 4th with an Elo of 1105, based on about 2,119 samples. Suno V5.5 leads the board with an Elo of 1170, and Mureka V9 follows closely at 1159. StepAudio 3 Music's Elo is within 65 points of the top model, indicating competitive vocal quality.

The ablation compares direct VAE reconstruction at 25 Hz and 50 Hz with token-conditioned detokenization using a 50 Hz VAE and DiTs of 0.9B, 4B, and 8B parameters. The 50 Hz VAE outperforms the 25 Hz VAE in direct reconstruction across all metrics. In token-conditioned generation, the smallest 0.9B DiT yields the best scores on most metrics, indicating an information ceiling from the tokenizer rather than renderer capacity. The 50 Hz VAE achieves better direct reconstruction than the 25 Hz VAE, with lower MCD and higher SI-SNR and SDR. In token-conditioned detokenization, the 0.9B DiT gives the best MCD, MS-Mel-L1, MS-STFT-L1, SI-SNR, and UTMOS, while the 4B model leads only in SDR. Scaling the DiT from 0.9B to 4B or 8B does not consistently improve quality, suggesting the tokenizer limits achievable performance.

StepAudio 3 Music trains its LLM across three stages with decreasing token budgets, increasing context lengths, and a shift from constant to cosine learning rate decay, using a corpus of about one trillion tokens including ABC notation for enhanced control. Its supervised fine-tuning uses roughly 20B tokens across six task subsets, where the inclusion of ABC-CoT planning links musical conditions to audio output and genre-balanced selection maintains quality. In evaluations, the model leads on caption adherence and AudioBox-Aesthetics metrics, while Mureka V9 outperforms on individual SongBench dimensions, and the arena leaderboard places StepAudio 3 Music fourth with competitive vocal quality. Ablations show that a 50 Hz VAE beats a 25 Hz VAE in reconstruction, but token-conditioned detokenization saturates at a small 0.9B DiT, indicating the tokenizer limits performance rather than renderer capacity.