One-sentence Summary

Researchers from Tencent, Zhejiang University, Shanghai Jiao Tong University, The Chinese University of Hong Kong, and Nanyang Technological University present Gander, an end-to-end model that unifies omni perception, realtime interaction, and agentic capabilities through a Cerebellum-Brain collaborative framework built on a streaming Thinker-Talker architecture, enabling full-duplex, interruptible multimodal interaction and demonstrating robust performance in noisy, multi-party, and backchannel scenarios.

Key Contributions

• Introduces Gander, an end-to-end model that unifies omni perception, realtime interaction, and agentic execution in a single framework, supporting full-duplex streaming inputs across video, speech, and text with user interruption and proactive model feedback.
• Combines a Cerebellum-Brain collaborative design, where the Cerebellum handles realtime interaction and conversational capabilities and the Brain manages complex reasoning and agentic tasks, with a streaming Thinker-Talker architecture that flattens inputs and outputs into an ordered chunk-level token stream for low-latency continuous interaction.
• Evaluates Gander across conversational ability, omni understanding, interactive capability, and agentic intelligence, showing competitive spoken dialogue performance with SOTA open source models and robustness in background noise, multi-party interactions, and backchannel communication; releases models, code, and data.

Introduction

Large language models are moving beyond text based question answering toward autonomous agents that can perceive, reason, and act in real world environments. However, current human AI interaction remains locked in a turn based, text driven paradigm that differs fundamentally from the fluid, multimodal, and interruptible nature of human communication. Prior systems rely on external modules like VAD to manage interaction timing, which fails to handle spontaneous interruptions, proactive engagement, noisy environments, or multi party conversations, and they typically separate realtime interaction from complex agentic reasoning.

The authors introduce Gander, an Omni Interaction Agent that unifies realtime multimodal interaction with agentic intelligence in a single end to end model. They argue interactivity must be intrinsic to the model rather than an orchestration layer, so Gander processes streaming audio visual inputs and generated text as temporally aligned chunks, explicitly predicting whether to listen or speak in each chunk. To address the trade off between low latency conversation and long horizon reasoning, they adopt a Brain Cerebellum architecture where the Cerebellum handles realtime interaction and the Brain, a training-free plug and play component, performs deeper asynchronous reasoning via tool calls. This decoupled design allows stronger reasoning models to be integrated without retraining, providing a scalable path toward simultaneously improving responsiveness and intelligence.

Dataset

The authors construct a large-scale training corpus for Gander, organized into four data families: speech interaction, audio-visual interaction, agentic interaction, and robustness and negative data. The corpus is designed to teach the model not only general speech and multimodal understanding, but also when to listen or speak, how to react to evolving audio-visual context, and how to coordinate with a separate back brain during long-horizon tasks.

Dataset Composition and Sources

• Speech Interaction Data (approximately 37% of the corpus): Covers general dialogue, spoken instruction following, question answering, full-duplex interaction, and simultaneous speech translation. The full-duplex portion, called InteractionSpeech, is synthesized through a dedicated four-stage pipeline: dialogue collection, interaction event annotation, audio rendering, and quality control.

  • Dialogue sources: Two complementary sources are used. First, 11.2K scene and topic seeds spanning 45 everyday and task-oriented scenarios (e.g., education, healthcare, travel, finance, customer support) are expanded into multi-turn spoken dialogues using DeepSeek-V4-Pro, constrained to 8-18 turns and at most 96 seconds of speaking time. Second, existing multi-turn dialogues from real assistant logs and public corpora are converted to duplex form after passing a spoken suitability filter that rejects turns with markdown, code, URLs, or implausible speech density. Each dialogue is conditioned on one of five interaction profiles: constraint clarification, process control, user correction, failed service entry, and safety or urgency stop.
  • Interaction events: Two events are annotated. Competitive interruption occurs when the user barges in before the assistant finishes, with the assistant's remaining words forming a hidden continuation that overlaps the user speech but is never heard. Supportive backchannel occurs when the user emits a brief acknowledgment while the assistant continues without pausing. Backchannels are accepted only if embedded within the ongoing utterance, below a length threshold (8 Chinese characters or 6 English words), and match a bilingual lexicon of 266 Chinese and 170 English expressions across 11 intent categories.
  • Rendering and timing: Only the user channel is rendered to audio using voice cloning TTS, while assistant turns remain textual as duration placeholders. Every turn gets a global onset, duration, and overlap interval, making timing explicit supervision. Only two markers are retained: interruption and backchannel markers.
  • Quality control: A rule-based gate discards implausible interruption points, degenerate overlaps, leaked content, or unresolved interactions. An LLM judge then rates surviving samples on naturalness, assistant coherence, interruption plausibility, and backchannel plausibility. The final corpus contains 260.8K dialogues with broadly distributed interruption onsets.

• Audio-Visual Interaction Data (approximately 40% of the corpus): Extends realtime interaction to multimodal environments. Collected from JoyAI-VL, LiveCC, and Streamo, organized into streaming video question answering, streaming video narration, and proactive visual response. All samples undergo quality filtering and temporal alignment refinement using Qwen3.5-297B-A17B, yielding approximately 1.1M high-quality audio-visual interaction pairs. Target responses are rewritten with DeepSeek-V4-Pro to normalize response lengths to an average of 8 tokens per second, and user utterances are synthesized into speech using Qwen3-TTS.

• Agentic Interaction Data: Extends interaction to coordinated settings among user, front cerebellum, and back brain. Consists of three categories: audio agentic interaction, omni agentic interaction, and a small set of tool-assisted reasoning data.

  • Audio-agentic data: Built through seed-driven trajectory synthesis. Structured task seeds are sampled across task classes, domains, categories, and task families, covering workflows such as execution, information search, and search-then-execute. Spoken user requests and interaction trajectories are synthesized with DeepSeek-V4-Pro, then filtered for task consistency, interaction validity, and quality.
  • Omni-agentic data: Extends coordination to visually grounded tasks. GUI and video trajectories come from crawled interaction trajectories, existing GUI datasets, and Codex-generated trajectories, totaling approximately 36K examples. Qwen3.5-297B-A17B converts these into structured descriptions of environment states and task progress, and DeepSeek-V4-Pro synthesizes spoken user requests and interactions. Trajectories are filtered for temporal consistency, task validity, and interaction quality.

• Robustness and Negative Data: Covers irrelevant visual context, no-command environments, acoustic and conversational interference, and multi-party interaction. Irrelevant-video negatives pair web-crawled videos with unrelated user instructions. Anti-interference and multi-party examples come from Hy-Realtime production data. These data encourage the model to ignore unrelated events, remain silent without valid requests, resist noise and overlapping distractors, and track speakers and addressees in multi-party settings.

Data Usage in the Model

• The corpus is used to train the model on realtime interaction behavior, including turn-taking, interruption, overlapping speech, and response timing.
• The full-duplex data explicitly supervise interaction timing rather than inheriting it from turn-based corpora.
• Audio-visual data train the model to continuously integrate incoming visual evidence with ongoing interaction and ground responses in current information.
• Agentic data train the front cerebellum to maintain continuous user interaction while coordinating with the back brain throughout asynchronous task execution, spanning the full task lifecycle from initiation to result delivery.
• Robustness and negative data reduce spurious responses and improve reliability in complex real-world environments.

Method

Gander is an end-to-end omni interaction agent architecturally composed of the front cerebellum, the agent orchestration runtime, and the back brain. Refer to the framework diagram:

The front cerebellum is a realtime, full duplex multimodal model based on a thinker-talker architecture, responsible for continuous multimodal perception and interactive communication. The back brain is a general purpose task execution agent that requires no task specific training, instantiated by general purpose agents such as Claude Code and Codex. The agent orchestration runtime serves as the coordination layer between the front cerebellum and back brain, jointly managing realtime multimodal inference and the orchestration of asynchronous background tasks. Through this division of responsibilities, the three components collectively support both continuous realtime interaction and complex long horizon task execution.

The front cerebellum dynamically determines whether a request can be resolved locally or requires delegation to the back brain. As shown in the figure below:

Simple conversational and short horizon tasks are handled directly by the front cerebellum, whereas complex workflows involving multi step reasoning, external tool use, or long horizon execution are delegated to the back brain. This delegation mechanism is formalized through structured tool calls that expose the front cerebellum's task-level decisions to the agent orchestration runtime. The interface defines three primary operations: task_start, task_send, and task_resolve, corresponding to task creation, incremental task interaction, and deterministic tool state control respectively. The agent orchestration runtime supports two control modes: lean and coordinator. In the lean mode, the runtime directly executes the task actions classified by the front cerebellum. The coordinator mode introduces an independent control plane model between the front cerebellum and the gateway to generate execution directives. The Gateway serves as the persistent orchestration core of the runtime, organizing backend execution around five persistent entities and managing task state transitions, worker scheduling, and permission handling.

The front cerebellum builds on a thinker-talker architecture that receives streaming audio and video input and generates both textual and audio output. Refer to the detailed architecture diagram:

Building on this design, the authors introduce a streaming chunk flattening mechanism for end-to-end interaction, which flattens perceptual inputs and generated outputs into a unified chunk stream and enables the model to dynamically determine whether to listen or speak at each chunk. The front cerebellum senses its environment through two modality encoders that run concurrently, one over the visual stream and one over the acoustic stream. The visual pathway follows an any resolution partitioning scheme where each frame is decomposed into slices, encoded by a SigLIP vision transformer, and condensed by a query based resampler. The acoustic stream is processed by a streaming and chunk wise speech encoder, followed by a lightweight MLP projector that applies temporal downsampling. To let the front cerebellum perceive and respond within a single autoregressive process, the continuous interaction is partitioned into fixed windows of one second and assembled into one chunk per window. Each chunk is the flattened concatenation of encoded audio and visual tokens, a predicted control token, and text tokens. The control token takes one of three values: a listen token, a speak token, or an interrupt token. The context is managed through a fixed budget of 128 chunks using a sliding window. For speech generation, the front cerebellum decouples semantic planning from acoustic realization, delegating waveform production to a speech token decoder followed by a streaming flow matching decoder.

To train Gander, the authors construct a large scale corpus comprising realtime speech interaction, audio-visual interaction, and agentic interaction data. The construction pipelines for agentic interaction data are illustrated in the figure below:

The agentic interaction data extend the setting to coordinated interaction among the user, front cerebellum, and back brain. Audio-agentic interaction data are constructed through a seed-driven trajectory synthesis pipeline, sampling structured task seeds and diverse interaction patterns. Omni-agentic interaction extends such coordination to visually grounded tasks, assembling underlying GUI and video trajectories from crawled interaction trajectories, existing GUI datasets, and GUI trajectories generated with Codex. These trajectories are converted into structured descriptions of environment states, task progress, and interaction context, followed by synthesis of spoken user requests and corresponding interactions.

Experiment

Gander is evaluated across three axes: full-duplex interaction with tool use, spoken conversation, and omni understanding. In full-duplex interaction, it achieves the best turn-taking behavior, with no missed turns and few premature interruptions, though it slightly trails baselines on task accuracy; a back-brain-only condition shows the execution tier is not the limiting factor. In spoken conversation, Gander leads the full-duplex group on knowledge-oriented QA and matches turn-based systems, indicating the streaming formulation does not inherently cost knowledge retention. In omni understanding, it retains competitive performance despite interaction-only training, with the ablation showing genuine audio-visual fusion rather than reliance on a single modality, though fine-grained perceptual reasoning shows some regression.

The gateway in the agent orchestration runtime manages core entities such as projects, tasks, runs, worker events, and deliveries, which collectively support long-lived workflows and result delivery. The runtime offers two control modes, lean and coordinator, balancing latency and determinism against flexibility and oversight. The coordinator mode adds a separate control plane for dynamic execution directives, while the lean mode relies on preconfigured task handling. The gateway manages persistent projects, logical tasks, concrete run instances, worker events, and delivery records for user-facing results. Lean mode provides lower latency and higher determinism but limits dynamic adaptation of reasoning intensity or supervision policies. Coordinator mode introduces an independent control plane that generates declarative execution directives, adding latency and nondeterminism while enabling flexible supervision and permission policies.

The training corpus for Gander is dominated by streaming video QA and narration data, which together make up the largest fraction, while speech interaction data covers turn-taking, interruption, and spoken QA. The corpus also includes robustness-oriented and negative supervision data to improve reliability in challenging conditions, such as multi-party scenarios and cases where the model should remain silent. Streaming video QA and narration form the largest data family, accounting for over 40% of the corpus. Speech interaction data includes full-duplex turn-taking, interruption, and multi-turn interaction, comprising nearly 10% of the corpus. The corpus includes negative supervision data to teach the model when to remain silent or suppress unnecessary responses.

Gander trails most full-duplex baselines on tool selection, argument accuracy, response quality, and pass@1, but achieves the lowest interruption rate among full-duplex models. Its take-turn rate is tied with the cascaded pipeline, though the latter is text-driven and not directly comparable on interaction metrics. Gander has the lowest interruption rate among full-duplex models, despite lower overall quality scores. Gander matches the cascaded pipeline's take-turn rate, but the cascaded baseline is not directly comparable due to its text-driven nature. Gander's tool selection and argument accuracy are below all full-duplex baselines, indicating a trade-off between interactivity and task performance.

Gander leads the full-duplex group on SpokenQA subsets but trails on VoiceBench metrics, and its scores are competitive with turn-based models despite operating under a streaming constraint. The evaluation did not invoke the back brain, so results reflect only the front cerebellum's performance. Gander outperforms Audio-Interaction and Moshi on both SpokenQA subsets within the full-duplex group. Gander's SpokenQA scores rank second overall among all nine systems, close behind leading turn-based models. On VoiceBench, Gander places second in its group on AlpacaEval and SD-QA, slightly behind Audio-Interaction. The back brain was never used during the evaluation, so all scores come from the front cerebellum alone.

Gander, trained exclusively for interaction, retains competitive omni understanding, scoring 49.62 on WorldSense and 78.53 on Daily-Omni. It stays within 1.67 points of its MiniCPM-o 4.5 initialization on Daily-Omni but drops 6.08 points on WorldSense, with the regression attributed to training that rewards temporal audio-visual reasoning over static attribute inspection. Gander remains ahead of Qwen3-Omni on Daily-Omni (78.53 vs. 70.70) while staying close to its own base model. The WorldSense drop is not due to visual encoding, as the vision tower is frozen and unchanged. Interaction training preserves temporal audio-visual reasoning (Daily-Omni) more than fine-grained perceptual attribute tasks (WorldSense).

The experiments evaluate Gander's full-duplex interaction capabilities against turn-based and cascaded baselines. Gander excels at interaction metrics such as interruption and take-turn rates but trades off tool selection and argument accuracy, and it remains competitive on spoken QA and omni understanding despite a streaming constraint. The runtime architecture supports lean and coordinator modes, balancing latency and determinism against flexibility, while the training corpus prioritizes streaming video QA and speech interaction data, including negative supervision to suppress unnecessary responses.