One-sentence Summary

Researchers from National Taiwan University and NVIDIA introduce Full-Duplex-Bench-v3 (FDBv3), a benchmark using real human audio annotated for five disfluency categories and chained API calls across four task domains to evaluate six spoken-model configurations (GPT-Realtime, Gemini Live 2.5, Gemini Live 3.1, Grok, Ultravox v0.7, and a Cascaded Whisper→GPT-4o→TTS pipeline), finding that GPT-Realtime leads on Pass@1 (0.600) and interruption avoidance (13.5%), while the Cascaded baseline incurs the highest latency (10.12 s).

Key Contributions

• Full-Duplex-Bench-v3 (FDBv3) is introduced as an open, reproducible benchmark built from real human audio annotated for five disfluency categories and paired with scenarios requiring chained API calls across four task domains.
• An evaluation across accuracy, latency, and turn-taking is provided for six configurations: GPT-Realtime, Gemini Live 2.5, Gemini Live 3.1, Grok, Ultravox v0.7, and a cascaded Whisper→GPT-4o→TTS pipeline; GPT-Realtime leads on Pass@1 (0.600) and interruption avoidance (13.5%), Gemini Live 3.1 achieves the fastest latency (4.25 s) but lowest turn-take rate (78.0%), and the cascaded baseline has the highest latency (10.12 s) despite perfect turn-taking.
• Across all systems, self-correction handling and multistep reasoning under hard scenarios are the most consistent failure modes, with GPT-Realtime succeeding on fewer than 59% of self-correction scenarios. These results reveal a trade-off between fast tool execution and flexible handling of changing conversational intent.

Introduction

Voice agents are increasingly expected to perform real-world actions such as checking flight prices or updating account settings, but spoken interaction adds a strict latency requirement that text-based tool use does not face. Prior work has largely relied on cascaded offline pipelines, synthetic speech, synthetic training data, or closed proprietary systems, and existing benchmarks often miss natural disfluencies, mid-utterance corrections, and reproducible multi-step API execution. The authors introduce Full-Duplex-Bench-v3, the first benchmark to evaluate real-time voice agents on multi-step tool use with authentic human speech, systematic disfluency annotations, deterministic API constraints, and self-correction/state rollback scenarios across four task domains. They evaluate six leading configurations and show that handling mid-sentence corrections remains a major open challenge, revealing a trade-off between latency, turn-taking, and reliable reasoning.

Dataset

The authors describe a spoken tool-use benchmark built from human audio recordings, not a training dataset in the provided text.

Sources and composition

• 100 recordings from 12 speakers, including native and non-native English speakers with Korean and Russian backgrounds and varying accent strengths.
• Each speaker contributed 10 scenarios across four task domains, with disfluencies represented proportionally.
• Audio was collected in uncontrolled environments, mostly with built-in microphones in 11 of 12 setups, ranging from quiet rooms to mild background noise.
• All recordings were reviewed for quality.

Task domains and mock APIs

• The benchmark spans four task domains, each with a small set of callable tools.
• Mock APIs run locally with deterministic zero-latency responses, removing network variability and enabling automatic scoring.

Difficulty tiers and annotations

• Scenarios are divided into three tiers: Easy single-step, Medium two-step with moderate ambiguity, and Hard multi-step with conflicting constraints.
• Each recording is annotated for five disfluency categories: false starts, self-corrections, fillers, pauses, and hesitations.
• Twenty-one scenarios specifically include self-correction events to test real-time state rollback.

Processing details

• For trailing silence, the authors capture 30 seconds of each speaker’s actual ambient environment rather than appending digital silence, keeping the acoustic background coherent.
• The provided text does not specify a cropping strategy, training split, or mixture ratios.

How the data is used

• The dataset is used as an evaluation benchmark for spoken tool-use under disfluent conditions.
• Difficulty tiers and disfluency annotations support testing of failure modes such as obsolete context, dynamic parameter updates, and end-of-turn detection robustness.

Method

The authors structure the benchmark as a controlled evaluation framework rather than a live service testbed. Instead of routing spoken queries to real web APIs, the system uses locally executed mock APIs with deterministic, zero-latency responses. This design isolates model reasoning and parameter-passing from network variability, service downtime, and external latency. As a result, measured processing time reflects only model overhead, and all expected outputs can be scored automatically.

The benchmark spans four task domains with a small set of callable tools per domain. Scenarios are organized into three difficulty tiers according to the number of required tool calls and the complexity of reasoning involved. Easy scenarios are single-step, Medium scenarios require two steps with moderate ambiguity, and Hard scenarios involve multi-step reasoning with conflicting constraints. Audio is collected from human speakers in uncontrolled environments to preserve natural speech variation.

Each recording is annotated for five disfluency categories that target distinct failure modes. False starts, where a speaker abandons one intent and begins another, test whether models discard obsolete context without hallucinating tool calls. Self-corrections, where parameters are updated mid-sentence, evaluate dynamic state rollback. Fillers such as "um" and "uh" probe whether redundant tokens degrade accuracy or inflate latency. Pauses and hesitations, including filler-repetition combinations, test the robustness of end-of-turn detection.

In addition to the benchmark design, the authors analyze a cascaded reference pipeline consisting of Whisper for automatic speech recognition, GPT-4o for language understanding, and OpenAI TTS for speech output. This pipeline isolates the cost of a traditional modular architecture. Although it shares the same underlying large language model as GPT-Realtime, its Pass@1 is lower, indicating that ASR-introduced errors propagate downstream. The effect is most pronounced on self-corrections: the cascaded system scores substantially lower, while GPT-Realtime handles rollback far more effectively. Because Whisper can finalize an incorrect transcription before the user's correction arrives, the downstream model has no opportunity to revise the original state.

The cascaded design provides reliable turn-taking, with perfect engagement and no silent-worker failures. However, this reliability comes at the cost of high task-completion latency, driven mainly by a large first-word delay. The sequential Whisper, language model, and TTS chain creates an irreducible bottleneck that end-to-end speech models can sidestep through concurrent processing. This contrast quantifies the trade-off between modular reliability and native speech-model speed.

Experiment

The benchmark evaluates real-time voice agents using deterministic mock APIs across four task domains and three difficulty tiers, with naturalistic disfluent speech collected from human speakers. Six systems are compared, including five end-to-end models and a cascaded Whisper to GPT-4o to TTS baseline, on tool selection, argument accuracy, task completion, response quality, and turn-taking latency. GPT-Realtime emerges as the most balanced overall performer, while the cascaded baseline is reliable but slow and Gemini Live 3.1 is fast yet often fails to produce speech despite executing tool calls. Self-corrections remain difficult for all systems, and qualitative case studies reveal a fundamental trade-off between eager tool invocation for speed and conservative commitment for correctness.

The benchmark is organized around four task domains that use locally executed mock APIs with deterministic, zero-latency responses. Each domain provides a small set of callable functions for activities such as travel booking, financial operations, housing search, and e-commerce support. This design isolates model reasoning and parameter-passing and enables automatic scoring. The four task domains are Travel & Identity, Finance & Billing, Housing & Location, and E-Commerce Support. Each domain offers a small set of callable functions, typically two or three, covering booking, financial, housing, and order management tasks. All mock APIs run locally with deterministic, zero-latency responses, which isolates reasoning and parameter-passing and enables automatic scoring.

GPT-Realtime is the strongest overall performer, leading on tool selection, argument accuracy, response quality, Pass@1, and lowest interruption rate. Gemini Live 3.1 has the fastest latency but the lowest turn-take rate, missing responses in many scenarios. The Cascaded baseline always responds but is slowest, while Ultravox combines high turn-taking with the most interruptions and filler. GPT-Realtime leads overall performance, combining top accuracy and response quality with the lowest interruption rate. Gemini Live 3.1 achieves the fastest latency but has the lowest turn-take rate, with many scenarios receiving no response. The Cascaded baseline guarantees a response in every scenario but has the highest latency. Ultravox ties for the highest turn-take rate yet has the highest interruption and filler rates, often speaking over users with low-content responses. Gemini Live 2.5 is more conservative, showing higher turn-taking and low interruption but lower overall accuracy.

GPT-Realtime leads or ties on every disfluency category, with a particularly large advantage in self-correction. Gemini Live 3.1 is more balanced than Gemini Live 2.5 on false starts but weaker at self-correction, while the cascaded pipeline has the lowest self-correction score. Pauses are a shared weak point, especially for Grok and Ultravox, indicating difficulty detecting when a user has finished speaking. GPT-Realtime achieves the best or tied best Pass@1 in every disfluency category and leads most strongly on self-correction. The cascaded pipeline is much weaker on self-correction than all other systems, consistent with limited state rollback after transcription finalization. Pause is the lowest-scoring category for Grok and Ultravox and a shared weak point across several systems, suggesting end-of-utterance detection remains challenging. Gemini Live 3.1 improves false-start robustness relative to Gemini Live 2.5 but trails on self-correction, indicating the update improved general robustness while state rollback declined.

Pass@1 declines as scenario difficulty increases across all evaluated models. GPT-Realtime achieves the highest Pass@1 at every difficulty level, sharing the top Medium score with Gemini Live 3.1. Cascaded is relatively strong on Easy tasks but drops sharply on Hard, while Grok records the lowest Hard Pass@1. GPT-Realtime posts the leading Pass@1 at every difficulty level, tied with Gemini Live 3.1 on Medium. All systems lose accuracy as tasks move from Easy to Medium to Hard. Cascaded performs competitively on Easy tasks but degrades sharply on Hard tasks. Grok falls to the lowest Hard Pass@1 among all models.

Finance is the strongest domain for every model, while housing is consistently the weakest. GPT-Realtime leads all four domains, with the closest competition in finance and travel. Ultravox shows a notable imbalance between its stronger finance performance and weaker e-commerce performance. Every model achieves its highest Pass@1 on finance and its lowest on housing. GPT-Realtime leads all four domains, with housing remaining low across all systems. Gemini Live 3.1 closely trails GPT-Realtime on finance but has a wider gap on e-commerce and housing. Ultravox shows notable domain imbalance, with finance roughly double its e-commerce score. Cascaded matches Grok on e-commerce but scores lower on travel.

The benchmark uses four task domains with locally executed deterministic mock APIs, enabling automatic scoring of tool selection and argument passing. GPT-Realtime is the strongest overall model, leading on accuracy, response quality, lowest interruption rate, and performance across difficulty levels and domains, while Gemini Live 3.1 offers fast latency but often fails to respond and Ultravox has high interruption and filler rates. The cascaded baseline responds consistently but is slow and degrades sharply on hard tasks. Across systems, finance is the easiest domain, housing is the hardest, and pause handling and self-correction remain common weaknesses.