One-sentence Summary

StepFun-Audio Team presents StepAudio 3 Realtime, an audio-language foundation model built around a continuous listen-converse-think-act loop that integrates Deep Perception, Seamless Duplex, and Think-While-Speaking to parallelize private reasoning with spoken delivery, achieving $73.0$73.0 on StepAudioChat, $90.6$90.6 on MMSU, $98.9$98.9 on the Artificial Analysis Full-Duplex Bench, and a $56.0\%$56.0% success rate on $\tau$τ-Voice.

Key Contributions

• Introduces StepAudio 3 Realtime, an audio-language foundation model built around a continuous listen-converse-think-act loop that coordinates perception, conversational timing, reasoning, and tool use, enabling concurrent handling of user speech, model speech, and tool results as interactions progress.
• Proposes Think-While-Speaking, a mechanism that runs private reasoning in parallel with spoken delivery, supported by Adaptive Thinking and multi-token prediction, to balance deep deliberation with real-time response; this achieves dialogue and reasoning performance comparable to dedicated reasoning models while speaking in real time.
• Demonstrates top-tier evaluation results: a 73.0 macro average on StepAudioChat in reasoning mode, 90.6 on MMSU, 98.9 Overall on the Artificial Analysis Full-Duplex Bench, and a 56.0% macro task-success rate on τ-Voice, while identifying remaining gaps in multi-turn constraint following and retail tool-use tasks.

Introduction

Natural spoken interaction requires a system to follow the user while managing its own response, where pauses and overlapping speech complicate the flow. Prior audio-language models have advanced speech recognition and generation, but they struggle with real-time coordination of perception, reasoning, and action during a conversation, especially when handling complex requests or external tool use that outlasts the spoken exchange.

The authors present StepAudio 3 Realtime, which builds on the Step-Audio series’ shared audio-language foundation to coordinate listening, speaking, reasoning, and acting through a continuous loop. They introduce functions for deep perception of acoustic and linguistic cues, seamless full-duplex management of conversational turns, and think-while-speaking capabilities that allow deliberation alongside spoken delivery. The system also integrates a streaming Voice Agent to carry conversational intent into tool execution, with new user input dynamically shaping ongoing interaction. Evaluations show strong performance on audio understanding benchmarks and conversational-floor management, though gaps remain in multi-turn constraint following and retail tool-use tasks.

Dataset

• Shared pretraining and midtraining: StepAudio 3 ASR Max and StepAudio 3 Realtime use the same pretraining and midtraining stages; they only diverge during supervised fine-tuning, where the ASR branch is specialized for transcription and the realtime branch is tuned for spoken interaction.

• Supervised fine-tuning for ASR: The ASR model is fine-tuned with examples packed into sequences of up to 32K tokens. Time-frequency masking is applied following SpecAugment, while the audio encoder is frozen and the audiolanguage adapter and language decoder are updated to produce normalized transcripts. Context-aware recognition examples may include dialogue history, a preceding model response, a scenario description, or task-specific terminology as optional evidence. The target transcript remains grounded in the input waveform.

• Short- and long-form ASR data: The ASR mixture combines short labeled utterances with long pseudo-labeled recordings. Multiple recognition systems transcribe segmented audio, and their hypotheses are aligned and fused with Recognizer Output Voting Error Reduction (ROVER). Agreement-based filtering selects reliable segments, which are then recomposed into longer sessions. An LLM restores punctuation and improves consistency across each session.

• Long-tail terminology augmentation: To handle rare names and technical terms that are often confused with common homophones, the authors build targeted synthetic training examples. An LLM expands a knowledge taxonomy to identify categories rich in homophones, uncommon characters, abbreviations, and product identifiers. Candidate terms are enumerated, deduplicated, and placed in natural carrier sentences. These sentences are converted to speech and retained only when their pronunciation matches the target text. For acoustically confusable terms, examples may include dialogue history or entity hints.

• Audio understanding data construction: A hierarchical taxonomy covers lexical content, paralinguistics, acoustic events, speaker and temporal structure, music, and audio-grounded reasoning. Sampling controls balance duration and language coverage while removing duplicates and unsuitable recordings. Each recording is described and mapped to capabilities supported by its content, guiding the construction of clip-specific questions. Multiple models independently label each audio–question pair, and outputs are consolidated through agreement and quality checks.

• Data selection and quality control for audio understanding: Deterministic checks remove empty, truncated, malformed, or severely repetitive outputs. Text-only LLM judges assess query and response quality and assign a case-value score, which considers the query, the response, the useful information available in the audio annotations, and the training value of the question. Grounding reliability is estimated from cross-model consistency of responses. Only high-quality, high-case-value, and cross-model consistent candidates are retained as SFT candidates; broadly useful examples may enter midtraining, while disagreements and correctable cases are routed to relabeling or further review.

• Less is more ablation: An ablation compares roughly two million randomly sampled SFT examples with about 100K high-quality examples retained after quality control. The quality-controlled set improves MMSU from 78.78 to 89.70 and MMAR from 74.70 to 84.50, despite using roughly one twentieth as many examples, highlighting the importance of data quality over raw SFT volume.

• StepAudioChat benchmark: This is a closed, text-based benchmark for foundational conversational intelligence. It evaluates dialogue behavior and reasoning expressed through dialogue, isolating text-level response quality from prosody, turn timing, and interruption handling. It uses a common capability taxonomy and newly constructed items to reduce reliance on public test questions.

• Capability taxonomy for StepAudioChat: Conversational abilities are organized into a hierarchy with leaves targeting observable behaviors and defined evaluation boundaries. Tasks and examples from public benchmarks serve as a coverage check without reusing test questions. Overlapping mappings refine capability definitions, and unmapped examples identify coverage gaps. Each item targets a primary capability, and only capabilities supported by validated items enter the evaluation suite.

• Naturalistic item construction for StepAudioChat: Each item contains a dialogue prompt, independently checkable criteria, a valid reference response, and a deliberately flawed response. De-identified utterances from real interactions inform prompt phrasing and local context, preserving brevity, fragmentation, colloquial wording, and transcription noise. Personal entities are replaced with typed placeholders. Scenarios are recast across everyday settings where appropriate.

• Quality control and difficulty calibration for StepAudioChat: Structural validation checks that prompts are complete and self-contained, with valid responses satisfying all judging criteria and flawed responses violating at least one. A semantic audit checks premise, reference response, and judging logic. Difficulty is calibrated using responses from two reference systems with different capability levels, guiding the mixture of baseline, discriminative, and difficult examples. The judge is calibrated separately using valid and flawed controls.

• Multi-turn dialogue data: Training for StepAudio 3 Realtime focuses on following user intent and constraints across turns, clarifying underspecified requests, and adapting to conversational context. Joint training on dialogue and reasoning examples supports deliberation on difficult requests and concise responses to routine turns. Training examples span diverse topics, personas, and interaction lengths.

• Factorized construction for dialogue data: Generation follows four axes: topics and concrete discussion points, participant personas with roles and interaction styles, turn depth, and target capabilities such as contextual recall and logical reasoning. A stratified rotation schedule coordinates these choices. Turn depth is an explicit variable, with longer dialogues progressing through deeper engagement with discussion points, allowing later turns to refine constraints or change direction while remaining consistent with history.

• Quality assessment and routing for dialogue data: Quality control separates three concerns. A context–query review evaluates the depth of the dialogue history, whether the opening is self-contained, and whether the final query is substantive. A capability-specific review checks whether the query instantiates each requested capability and whether the response satisfies its requirement. A response review evaluates answer quality, persona and style consistency, and naturalness as spoken dialogue. Deterministic checks handle defects such as empty output or role confusion. Only examples passing all checks and receiving high scores are retained for supervised fine-tuning.

• Training for conversational tool use: Training combines targeted voice-agent dialogues with real multi-step agent trajectories. Targeted dialogues cover request routing, clarification, private-context retrieval, confirmation before consequential actions, execution-time updates, progress queries, and result reporting. Negative examples discourage unnecessary tool invocation and unsupported claims of successful execution. Multi-step trajectories are filtered and normalized, focusing on tool-call structure, argument consistency, evidence grounding, and suitability for spoken interaction.

Method

StepAudio 3 Realtime coordinates listening, speaking, reasoning, and action through an evolving conversational context. User speech, model speech, and tool results can arrive while other parts of the interaction remain in progress, creating a tightly coupled realtime loop. Refer to the framework diagram for an overview of these coupled functions.

The underlying system employs a mixture-of-experts architecture with approximately 196 billion total parameters and 11 billion active parameters per token, built upon the Step 3.7 Flash language backbone. The audio frontend utilizes an Audio Transformer encoder, with an adapter mapping the encoder outputs into the representation space of the language model. As shown in the figure below, the full-duplex input path incorporates both user and model audio streams. Audio representations pass through the encoder and adapter to the LLM decoder, while text tokens enter through a separate input path, allowing the model to jointly condition on acoustic information and textual context. A speech generator produces incremental streaming audio, which returns to the model audio stream for subsequent interaction.

To manage conversational turns seamlessly, the system implements a dual-stream architecture for full-duplex interaction. It tracks the interaction state incrementally through a time-interleaved representation where audio is organized into 320 ms blocks, each followed by a state or text token. Acoustic evidence and semantic context guide decisions to continue listening, initiate a response, continue speaking, or yield the conversational floor. The model also uses its own ongoing speech to interpret overlapping user utterances in the context of what the user is currently hearing. Refer to the figure below for the dual-stream recurrent architecture and temporal organization.

Conversational intelligence is driven by a Think-While-Speaking mechanism that coordinates realtime reasoning. Two concurrent calls to the audio model act as a Formulation Brain, generating a private reasoning trace, and an Articulation Brain, producing short response segments conditioned on the reasoning available so far. Adaptive Thinking controls whether a turn uses explicit reasoning to reduce unnecessary compute on routine turns, while Multi-Token Prediction acceleration speeds up the private thinking stream. To support these capabilities, multi-turn dialogue training examples are constructed through a factorized pipeline involving discussion point selection, case planning, self-play, final-turn query construction, and rigorous quality labeling. As shown in the figure below, this pipeline ensures diverse topics and persona consistency.

The training process is organized into three foundational pretraining stages: modality alignment to establish the interface between acoustic representations and the language model, multimodal mixed training for joint audio-text modeling at scale, and a cooldown stage that places greater weight on high-quality data. Midtraining extends the context length to 128K tokens to accommodate longer dialogue histories and intermediate tool results, substantially increasing the share of audio-understanding and agent-interaction data. Finally, specialized teacher checkpoints are trained on complementary capabilities and integrated via weighted parameter merging, allowing the model to retain diverse strengths in dialogue, audio understanding, and general text reasoning without introducing additional inference-time routing.

Experiment

The StepAudio 3 family is evaluated across speech recognition, audio understanding, dialogue, full-duplex interaction, agentic task completion, and general text. StepAudio 3 ASR Max leads on standard ASR benchmarks and all ContextASR-Bench subsets. StepAudio 3 Realtime outperforms baselines on four of eight audio-understanding benchmarks, with the largest gains on MMSU and MMAR, and a data-quality ablation shows that roughly 100K high-quality SFT examples beat two million random ones. The model achieves top scores on Full Duplex Bench and competitive agentic task completion on tau-Voice, while Adaptive Thinking and MTP3 reveal that selective reasoning improves dialogue quality but that thinking allocation is not yet fully optimal.

StepAudio 3 ASR Max demonstrates competitive ASR performance across standard English and Mandarin benchmarks, leading on most subsets while trailing slightly on WenetSpeech. On ContextASR-Bench, it achieves the best error rates on all four subsets, with macro-average improvements over the closest competitor, indicating strong robustness on long-form, entity-rich speech without external context. StepAudio 3 ASR Max outperforms all other models on LibriSpeech test-clean, test-other, and AISHELL-1, with the largest gaps on AISHELL-1. On WenetSpeech, StepAudio 3 ASR Max ranks second on both subsets, trailing HY3.0 ASR Preview by a small margin but ahead of Doubao 2.0 ASR and Seed 2.0 Lite. StepAudio 3 ASR Max is the best on all four ContextASR-Bench subsets, and its macro-average error rates are notably lower than the next best model for both English and Mandarin.

StepAudio 3 Realtime demonstrates broad strength across eight audio-understanding benchmarks, leading on four of them and achieving a macro average close to Gemini 3.1 Pro. Its largest margins occur on MMSU and MMAR, while it trails notably on AudioMultiChallenge, indicating room for improvement in complex multi-turn audio reasoning. StepAudio 3 Realtime achieves the best score on four of the eight benchmarks, including MMSU and MMAR with gains of 7.0 and 4.8 points over the next best system. The model is competitive with Gemini 3.1 Pro on MMAU and WildSpeech, but lags by 17.7 points on AudioMultiChallenge. Big Bench Audio is nearly saturated for all systems, with scores above 98 for every model.

StepAudio 3 Realtime achieves the top overall score in a full-duplex interaction benchmark, outperforming strong baselines across pause handling, turn taking, interruption handling, and backchannel handling. Its balanced performance across these categories indicates strong conversational floor management, notably excelling in turn taking and interruption handling while maintaining robust backchannel behavior. StepAudio 3 Realtime leads the overall ranking with a score near 99, slightly ahead of the closest baseline. It achieves a perfect score on turn taking and a near-perfect score on user interruption handling. Its backchannel handling score is notably higher than one major baseline, though slightly below another, reflecting a balanced interaction profile.

StepAudio 3 Realtime in reasoning mode achieves a macro average of 73.0 on StepAudioChat, outperforming Doubao 2.0 Lite and DeepSeek-V4-Flash but trailing Kimi K3. The model shows competitive performance in reasoning, memory, knowledge, conversational pragmatics, and persona consistency, with its strongest scores in safety and reliability and knowledge, while instruction following and conversational pragmatics are relatively weaker. StepAudio 3 Realtime ranks second overall among the four models, with a macro average of 73.0, above Doubao 2.0 Lite and DeepSeek-V4-Flash but below Kimi K3. The model achieves its highest score in safety and reliability (79.0), outperforming Doubao 2.0 Lite and DeepSeek-V4-Flash but trailing Kimi K3. In reasoning, StepAudio 3 Realtime (73.0) is on par with Doubao 2.0 Lite and clearly ahead of DeepSeek-V4-Flash, but significantly below Kimi K3. Conversational pragmatics is a relative weakness for StepAudio 3 Realtime, scoring below Kimi K3 and showing only a modest lead over Doubao 2.0 Lite and DeepSeek-V4-Flash.

Adaptive Thinking selectively applies reasoning to dialogue turns, keeping most performance while reducing reasoning overhead compared to always reasoning. It consistently outperforms a no-think baseline, and in some domains it matches or improves on the original reasoning-heavy approach. Adaptive Thinking retains most of the benefit of full reasoning, with scores only slightly below full reasoning in most domains. Compared to a forced no-think baseline, Adaptive Thinking improves scores substantially, especially on Reasoning, Memory, and Knowledge. The smallest reasoning budget is applied to Memory and Safety, while Knowledge and Faithfulness keep higher think rates.

StepAudio 3 ASR Max performs strongly across standard English and Mandarin benchmarks, leading most subsets and showing robust gains on long-form and entity-rich speech, while trailing slightly on WenetSpeech. StepAudio 3 Realtime excels on four of eight audio-understanding benchmarks with a macro average near Gemini 3.1 Pro, though it lags on AudioMultiChallenge, and it also leads in full-duplex interaction with near-perfect turn taking and interruption handling. In reasoning mode, it ranks second overall on StepAudioChat, trailing Kimi K3, with strengths in safety and knowledge but weaker conversational pragmatics. Adaptive Thinking cuts reasoning overhead by applying it selectively, improving on a no-think baseline while preserving most of the performance of always reasoning.