HyperAIHyperAI

Command Palette

Search for a command to run...

Beyond Answering: Learning to Speak First and Act in Streaming Video

Abstract

Most AI assistants respond only when prompted, remaining silent even when events in the video stream offer meaningful opportunities for conversation. To move beyond purely user-driven interaction, we present AURA 2.0, a streaming video large language model that introduces three core upgrades to AURA. The key advance is Model-Initiated Conversation (MIC), which enables the model to identify topics worth discussing in the evolving visual stream and initiate conversations at appropriate moments without a user query. Stream-Aware Tool Calling (SATC) lets the model act on what it sees by invoking external tools using the current visual and conversational context and incorporating returned information into the ongoing interaction. Interactive Visual Grounding (IVG) supports visual grounding in both user inputs and model outputs. The model can identify objects marked by users on video frames and produce frame-aligned bounding boxes to link its responses to specific objects and regions. To develop these capabilities, we construct a dedicated streaming interaction dataset and adopt a three-stage training scheme to integrate the new behaviors with AURA’s existing streaming interaction abilities. We further incorporate Long-Horizon Interaction Memory (LHIM) to maintain contextual continuity over extended video streams. Evaluations on public streaming benchmarks show that AURA 2.0 achieves leading performance among open-source streaming models, while targeted evaluations further validate its new interaction capabilities.

One-sentence Summary

Researchers from Huawei Research, CUHK MMLab, and HKUST propose AURA 2.0, a streaming video large language model that extends AURA with model-initiated conversation, stream-aware tool calling, interactive visual grounding, and long-horizon interaction memory; trained on a dedicated streaming interaction dataset, it achieves leading performance among open-source streaming models on public benchmarks.

Key Contributions

  • AURA 2.0 is a streaming video large language model whose core advance is Model-Initiated Conversation: it identifies topics worth discussing from evolving visual observations and conversational history, decides when to initiate a conversation or remain silent, and moves beyond purely user-prompted interaction.
  • The model contributes Stream-Aware Tool Calling, which decides when to invoke external tools and incorporates returned results into ongoing responses, and Interactive Visual Grounding, which interprets user-marked objects and produces frame-aligned bounding boxes for referenced objects and regions.
  • To develop these capabilities, a dedicated streaming interaction dataset and a three-stage training scheme are constructed, with Long-Horizon Interaction Memory used for context continuity. Evaluations on four public streaming benchmarks show leading open-source performance, and targeted evaluations show useful MIC openings, improved answer accuracy with tool access, and fewer missing bounding-box outputs.

Introduction

Streaming VideoLLMs process incoming frames incrementally and coordinate continuous visual perception with response decisions, letting models stay silent during observation and answer using only what has been seen so far. Context management and memory mechanisms help maintain continuity across long video streams. However, prior interaction remains largely user-driven: models respond to user queries or proactive triggers based on standing instructions and monitoring rules, and existing systems do not initiate open-ended conversation without a prior request. The authors address this gap with AURA 2.0, whose main contribution is Model-Initiated Conversation, enabling the model to identify discussion-worthy topics in the evolving visual stream and speak first at appropriate moments, while also adding stream-aware tool calling and interactive visual grounding.

Dataset

The authors use the AURA 2.0 data pipeline to extend AURA’s streaming interaction data with three new supervision types: MIC, SATC, and IVG. Source videos are re-encoded, resized, and divided into clips; an MLLM then generates clip-level descriptions of objects, actions, and scene context.

Dataset composition and sources

  • The verified training mixture includes inherited Real-Time QA, Proactive QA, and Multi-Response QA, plus MIC, SATC, IVG, and mixed-capability data.
  • The pipeline contains 484,459 samples and 11.43 billion tokens, covering about 5,400 hours of video and 40 million training frames.
  • Sample and token counts include replayed examples across training stages and do not represent distinct source videos.

Subset details

  • Real-Time QA: 34.2% of samples. Supervises immediate answers.
  • Proactive QA: 22.1% of samples. Supervises delayed responses when sufficient evidence becomes available.
  • Multi-Response QA: 12.4% of samples. Supervises multiple responses to a single user query.
  • MIC: 11.8% of samples. An MLLM decides when to start a conversation and what to say. Each opening has a timestamp and a short message, supported by visual observations available up to that timestamp. Openings may draw attention to a visual detail, provide an alert, give contextual information, or suggest an activity.
  • IVG: 9.8% of samples. Built from about 55,000 video grounding and tracking trajectories sourced from MeViS, Refer-YouTube-VOS, DanceTrack, and SportsMOT. An MLLM rewrites annotations into streaming grounding interactions with bounding boxes aligned to video frames. Six patterns are covered: responses with boxes, multi-candidate disambiguation, subjective selection with an explanation, multi-turn follow-up and target switching, box-only tracking, and target-absent responses.
  • SATC: 5.3% of samples. Combines real execution records with MLLM-generated tool-calling traces. Each trajectory includes tool-call arguments, the returned result, and the subsequent response. For 50% of non-tool-calling streaming samples, one to four random tool schemas are added while retaining the original targets, which teaches selective tool use.
  • Mixed-capability data: 4.5% of samples. Combines multiple capabilities.

Processing and filtering

  • Synthesized trajectories are aligned to video chunks so that queries, silent observations, responses, tool calls, and bounding boxes share a consistent timeline.
  • Structural checks verify chronological consistency and correct alignment between bounding boxes and video frames.
  • MLLM validators assess whether responses are supported by available visual evidence and occur at appropriate times. For memory-related QA, relevant video history is used.
  • MIC openings are reviewed for timing and scene consistency using clips extending five seconds beyond the proposed initiation time.
  • For IVG, bounding boxes are rendered on the corresponding frames to verify target correctness.
  • Samples that fail these checks are excluded from training.

Training usage

  • Samples are split across three training stages: 51.6%, 33.9%, and 14.5% of samples, and 49.4%, 30.5%, and 20.0% of tokens.
  • Stage 3 has a higher token share than sample share, indicating a higher average token count per sample.
  • Silent observations dominate response-control targets in all three stages: 96.3%, 94.9%, and 96.7%. This imbalance motivates the response-control objective used during training.

Method

AURA 2.0 follows a streaming architecture where incoming video is organized into temporal chunks and processed incrementally alongside user inputs and interaction history. The context retains recent visual observations and textual interactions through sliding windows, allowing the model to continuously observe the video stream and determine whether to generate a response. For each video chunk, the assistant outputs <|silent|> when it remains silent, or <|respond|> followed by response content, which can contain a conversational utterance or a tool call. This format explicitly distinguishes silent observation from response generation and accommodates different interaction behaviors within the same chunk-wise structure.

As shown in the figure below:

The main architectural addition is the Long-Horizon Interaction Memory (LHIM) module, implemented using an auxiliary summarization model. This model processes visual content leaving the active visual window together with all available textual context. It organizes information at the event level and produces summaries in QA form, which are inserted back into the interaction context to preserve historical information about past events after their visual observations are no longer available.

The data pipeline extends streaming interaction data with three new forms of supervision: Model-Initiated Conversation (MIC), Stream-Aware Tool Calling (SATC), and Interactive Visual Grounding (IVG). It proceeds from video preparation to task-specific synthesis and finally to streaming assembly and quality verification.

As shown in the figure below:

Source videos are re-encoded, resized, and divided into clips, with an MLLM generating clip-level descriptions of objects, actions, and scene context. MIC data is constructed by asking an MLLM to determine when to initiate a conversation and what to say, ensuring openings are supported by visual observations and avoid repeating earlier discussions. SATC data combines real execution records with MLLM-generated tool-calling traces, including tool-call arguments, returned results, and subsequent responses. To encourage selective tool use, randomly selected tool schemas are added to non-tool-calling samples. IVG examples are constructed from video grounding and tracking trajectories, where an MLLM rewrites annotations into streaming grounding interactions with bounding-box coordinates. Structural checks and task-specific MLLM verification filter synthesized trajectories to ensure chronological consistency and visual evidence support.

AURA 2.0 is initialized from Qwen3.5-9B and employs a three-stage curriculum to progressively introduce new capabilities without degrading existing response policies.

As shown in the figure below:

Stage 1 establishes a general streaming interaction and tool-use foundation using general streaming QA and SATC data. Stage 2 introduces visual grounding and model-initiated conversation with MIC and IVG data, while replaying earlier tasks to prevent degradation. Stage 3 focuses on capability composition and preference refinement, training on examples that combine multiple capabilities, such as Proactive QA with SATC and MIC with SATC.

To address the imbalance where silent steps account for over 94 percent of supervised control decisions, the authors combine class-balanced normalization with focal weighting for the control tokens <|silent|> and <|respond|>. Let pip_ipi​ denote the predicted probability of the ground-truth token at position iii, and let ℓi=−log⁡pi\ell_i = -\log p_iℓi​=−logpi​. Using SSS and RRR to denote the sets of supervised <|silent|> and <|respond|> positions, and OOO for all other supervised positions, the training objective is:

L=14(1∣S∣∑i∈S(1−pi)γℓi+1∣R∣∑i∈R(1−pi)γℓi)+1∣O∣∑j∈Oℓj\mathcal{L} = \frac{1}{4} \left( \frac{1}{|S|} \sum_{i \in S} (1 - p_i)^\gamma \ell_i + \frac{1}{|R|} \sum_{i \in R} (1 - p_i)^\gamma \ell_i \right) + \frac{1}{|O|} \sum_{j \in O} \ell_jL=41​(∣S∣1​i∈S∑​(1−pi​)γℓi​+∣R∣1​i∈R∑​(1−pi​)γℓi​)+∣O∣1​j∈O∑​ℓj​

with γ=2\gamma = 2γ=2. The focal factor reduces the contribution of confidently predicted control tokens, placing greater emphasis on difficult decisions. Response text and serialized tool-call content use standard cross-entropy without focal weighting. The losses of the two control classes are averaged separately to ensure equal base weights regardless of their respective numbers of supervised positions.

To support long-horizon streaming video understanding without continuously increasing visual context, LHIM compresses historical video observations into timestamped textual descriptions. The context is organized into a sliding window preserving the most recent 30 seconds of video and dialogue, and a context history storing compressed visual memories. Video preceding the sliding window is divided into 30-second slices and processed by a separate summarization MLLM, which partitions each slice into semantically coherent events. These events form level-1 memory records stored as text-only QA pairs. As the session progresses, older records are progressively merged into coarser level-2 and level-3 summaries, maintaining a fixed 30-second visual context while preserving recent history at event resolution.

Inference latency is optimized through session history caching, fast paths for silent input and output, and sentence-level speech handoff. Each request carries only the frames of the current video chunk, cached in a per-session history. Turns outputting <|silent|> end after a single decoding step without running speech synthesis, while speech synthesis starts before the MLLM finishes generating the full response by splitting text at sentence boundaries and forwarding completed segments.

Experiment

AURA 2.0 is evaluated across four public streaming benchmarks that assess proactive interaction, historical reasoning, real-time perception, tool use, and temporal understanding, along with targeted tests of model-initiated conversation, stream-aware tool calling, interactive visual grounding, and inference efficiency. The results show strong proactive and forward temporal reasoning, improved historical retrieval, and competitive real-time perception, with external tools and visual grounding contributing clear gains over the backbone. Efficiency measurements indicate low response latency and stable real-time speech delivery even under 16 concurrent sessions, with serving optimizations substantially reducing time to first audio.

Representative streaming interaction models consistently support real-time question answering, proactive question answering, and multi-response question answering. AURA 2.0 is the only listed model with reported support for all six compared capabilities, adding multimodal interaction context, stream-aware tool calling, and interactive visual grounding. Among earlier systems, these added capabilities are largely absent, with stream-aware tool calling reported only for JoyAI-VL-Interaction. All compared models support real-time QA, proactive QA, and multi-response QA. AURA 2.0 is the only listed system with reported support for MIC, SATC, and IVG. Earlier models generally lack MIC and IVG; SATC is reported only for JoyAI-VL-Interaction among prior systems.

AURA 2.0 demonstrates strong proactive interaction performance on IPIBench, particularly in proactive task management, where its category average is the highest among all evaluated models, including proprietary offline systems. In proactive monitoring, its timing accuracy outperforms several recent baselines and Gemini 3 Pro while approaching GPT-4o, and its repeated proactiveness score approaches Gemini 3 Pro. Human-level scores remain far above all model results across every subtask. AURA 2.0 achieves the highest proactive task management average among all evaluated models, with multi-task management exceeding Gemini 2.5 Pro and GPT-5.4. AURA 2.0 outperforms Qwen3.5-9B, Qwen3-VL-8B, and Gemini 3 Pro in proactive monitoring timing, approaching GPT-4o, and its repeated proactiveness score approaches Gemini 3 Pro. All evaluated models remain far below human-level performance, especially in interleaved reactive-proactive interaction and repeated proactiveness.

AURA 2.0 improves over its predecessor across the main StreamArena dimensions, with the largest gain in proactive interaction and a notable improvement on long-horizon historical retrospection. It also outperforms the other evaluated open-source streaming models in real-time perception and historical retrospection, and its tool-use accuracy surpasses the listed MiniCPM-o 4.5 and JoyAI-VL-Interaction results. The human reference remains far ahead of all models in every reported dimension. AURA 2.0 advances past JoyAI-VL-Interaction and MOSS-VL-Realtime in real-time perception and historical retrospection, with the strongest long-horizon L4 result among the listed open-source streaming models. Proactive interaction accuracy rises from 3.7% to 12.8%, overtaking JoyAI-VL-Interaction and MOSS-VL-Realtime. Tool-use accuracy also surpasses the listed JoyAI-VL-Interaction and MiniCPM-o 4.5 results, reaching 18.5%.

AURA 2.0 demonstrates strong proactive video understanding across the four ProactiveVideoQA categories. At the default emphasis, it outperforms JoyAI-VL-Interaction in every category and surpasses MOSS-VL-Realtime in three of the four categories. Its WEB performance is notably strong, exceeding all listed models at every emphasis level. AURA 2.0 outperforms JoyAI-VL-Interaction on WEB, EGO, TV, and VAD at the default emphasis. Compared with MOSS-VL-Realtime at the default emphasis, AURA 2.0 scores higher on WEB, EGO, and VAD, while MOSS-VL-Realtime remains slightly ahead on TV. AURA 2.0 achieves the best WEB PAUC among all listed models at every emphasis level, including proprietary offline baselines.

On PhoStream, AURA 2.0 achieves the highest Overall score among reported models and leads open-source streaming models on Instant, Backward, and Forward tasks. Its largest relative advantage appears in Forward reasoning, where it also reports the lowest Early Response rate and the highest Partly Correct rate. A remaining No Response rate indicates some responses are still missed. AURA 2.0 attains the best Overall score and exceeds the strongest proprietary baseline by over nine points. It leads open-source streaming models across Instant, Backward, and Forward tasks, with the greatest margin in Forward reasoning. AURA 2.0 records the lowest Early Response rate among all listed models, far below MOSS-VL-Realtime, Qwen3.5-9B, and JoyAI-VL-Interaction. It also achieves the highest Partly Correct rate, while a nonzero No Response rate points to some missed responses.

The evaluation covers streaming capability coverage, proactive interaction, stream understanding, proactive video question answering, and streaming reasoning benchmarks. AURA 2.0 is the only model reporting all six streaming interaction capabilities, and it consistently leads open-source baselines while also showing strong results against proprietary systems, including the highest proactive task management and overall reasoning scores plus the strongest WEB performance. Its main remaining limitations are a nonzero no-response rate on some streaming reasoning tasks and a large gap to human-level performance across interactive and proactive settings.


Build AI with AI

From idea to launch — accelerate your AI development with free AI co-coding, out-of-the-box environment and best price of GPUs.

AI Co-coding
Ready-to-use GPUs
Best Pricing

HyperAI Newsletters

Subscribe to our latest updates
We will deliver the latest updates of the week to your inbox at nine o'clock every Monday morning
Powered by MailChimp