HyperAIHyperAI

Command Palette

Search for a command to run...

응답을 넘어서: 스트리밍 비디오에서 먼저 말하고 행동하는 법 학습

초록

대부분의 AI 비서는 사용자가 요청할 때만 응답하며, 비디오 스트림 속 사건이 의미 있는 대화 기회를 제공하더라도 침묵을 유지한다. 순수하게 사용자 주도로 이루어지는 상호작용을 넘어서기 위해, 우리는 AURA에 세 가지 핵심 개선 사항을 도입한 스트리밍 비디오 대규모 언어 모델인 AURA 2.0을 제시한다. 핵심적인 진전은 모델 주도 대화(MIC)로, 이를 통해 모델은 변화하는 시각 스트림에서 논의할 가치가 있는 주제를 식별하고 사용자 질의 없이도 적절한 순간에 대화를 시작할 수 있다. 스트림 인지 도구 호출(SATC)은 모델이 현재의 시각 및 대화 맥락을 사용하여 외부 도구를 호출하고 반환된 정보를 진행 중인 상호작용에 통합함으로써 본 것을 바탕으로 행동할 수 있게 한다. 대화형 시각 접지(IVG)는 사용자 입력과 모델 출력 모두에서 시각적 접지를 지원한다. 모델은 사용자가 비디오 프레임에 표시한 객체를 식별하고, 프레임 정렬 경계 상자를 생성하여 응답을 특정 객체 및 영역에 연결할 수 있다. 이러한 능력을 개발하기 위해 우리는 전용 스트리밍 상호작용 데이터셋을 구축하고, 새로운 행동을 AURA의 기존 스트리밍 상호작용 능력과 통합하기 위한 3단계 훈련 방식을 채택한다. 나아가 장기 상호작용 메모리(LHIM)를 통합하여 확장된 비디오 스트림 전반에 걸쳐 맥락적 연속성을 유지한다. 공개 스트리밍 벤치마크에 대한 평가 결과, AURA 2.0은 오픈소스 스트리밍 모델 중 선도적인 성능을 달성했으며, 표적 평가를 통해 새로운 상호작용 능력이 추가로 검증되었다.

One-sentence Summary

Researchers from Huawei Research, CUHK MMLab, and HKUST propose AURA 2.0, a streaming video large language model that extends AURA with model-initiated conversation, stream-aware tool calling, interactive visual grounding, and long-horizon interaction memory; trained on a dedicated streaming interaction dataset, it achieves leading performance among open-source streaming models on public benchmarks.

Key Contributions

  • AURA 2.0 is a streaming video large language model whose core advance is Model-Initiated Conversation: it identifies topics worth discussing from evolving visual observations and conversational history, decides when to initiate a conversation or remain silent, and moves beyond purely user-prompted interaction.
  • The model contributes Stream-Aware Tool Calling, which decides when to invoke external tools and incorporates returned results into ongoing responses, and Interactive Visual Grounding, which interprets user-marked objects and produces frame-aligned bounding boxes for referenced objects and regions.
  • To develop these capabilities, a dedicated streaming interaction dataset and a three-stage training scheme are constructed, with Long-Horizon Interaction Memory used for context continuity. Evaluations on four public streaming benchmarks show leading open-source performance, and targeted evaluations show useful MIC openings, improved answer accuracy with tool access, and fewer missing bounding-box outputs.

Introduction

Streaming VideoLLMs process incoming frames incrementally and coordinate continuous visual perception with response decisions, letting models stay silent during observation and answer using only what has been seen so far. Context management and memory mechanisms help maintain continuity across long video streams. However, prior interaction remains largely user-driven: models respond to user queries or proactive triggers based on standing instructions and monitoring rules, and existing systems do not initiate open-ended conversation without a prior request. The authors address this gap with AURA 2.0, whose main contribution is Model-Initiated Conversation, enabling the model to identify discussion-worthy topics in the evolving visual stream and speak first at appropriate moments, while also adding stream-aware tool calling and interactive visual grounding.

Dataset

The authors use the AURA 2.0 data pipeline to extend AURA’s streaming interaction data with three new supervision types: MIC, SATC, and IVG. Source videos are re-encoded, resized, and divided into clips; an MLLM then generates clip-level descriptions of objects, actions, and scene context.

Dataset composition and sources

  • The verified training mixture includes inherited Real-Time QA, Proactive QA, and Multi-Response QA, plus MIC, SATC, IVG, and mixed-capability data.
  • The pipeline contains 484,459 samples and 11.43 billion tokens, covering about 5,400 hours of video and 40 million training frames.
  • Sample and token counts include replayed examples across training stages and do not represent distinct source videos.

Subset details

  • Real-Time QA: 34.2% of samples. Supervises immediate answers.
  • Proactive QA: 22.1% of samples. Supervises delayed responses when sufficient evidence becomes available.
  • Multi-Response QA: 12.4% of samples. Supervises multiple responses to a single user query.
  • MIC: 11.8% of samples. An MLLM decides when to start a conversation and what to say. Each opening has a timestamp and a short message, supported by visual observations available up to that timestamp. Openings may draw attention to a visual detail, provide an alert, give contextual information, or suggest an activity.
  • IVG: 9.8% of samples. Built from about 55,000 video grounding and tracking trajectories sourced from MeViS, Refer-YouTube-VOS, DanceTrack, and SportsMOT. An MLLM rewrites annotations into streaming grounding interactions with bounding boxes aligned to video frames. Six patterns are covered: responses with boxes, multi-candidate disambiguation, subjective selection with an explanation, multi-turn follow-up and target switching, box-only tracking, and target-absent responses.
  • SATC: 5.3% of samples. Combines real execution records with MLLM-generated tool-calling traces. Each trajectory includes tool-call arguments, the returned result, and the subsequent response. For 50% of non-tool-calling streaming samples, one to four random tool schemas are added while retaining the original targets, which teaches selective tool use.
  • Mixed-capability data: 4.5% of samples. Combines multiple capabilities.

Processing and filtering

  • Synthesized trajectories are aligned to video chunks so that queries, silent observations, responses, tool calls, and bounding boxes share a consistent timeline.
  • Structural checks verify chronological consistency and correct alignment between bounding boxes and video frames.
  • MLLM validators assess whether responses are supported by available visual evidence and occur at appropriate times. For memory-related QA, relevant video history is used.
  • MIC openings are reviewed for timing and scene consistency using clips extending five seconds beyond the proposed initiation time.
  • For IVG, bounding boxes are rendered on the corresponding frames to verify target correctness.
  • Samples that fail these checks are excluded from training.

Training usage

  • Samples are split across three training stages: 51.6%, 33.9%, and 14.5% of samples, and 49.4%, 30.5%, and 20.0% of tokens.
  • Stage 3 has a higher token share than sample share, indicating a higher average token count per sample.
  • Silent observations dominate response-control targets in all three stages: 96.3%, 94.9%, and 96.7%. This imbalance motivates the response-control objective used during training.

Method

AURA 2.0 follows a streaming architecture where incoming video is organized into temporal chunks and processed incrementally alongside user inputs and interaction history. The context retains recent visual observations and textual interactions through sliding windows, allowing the model to continuously observe the video stream and determine whether to generate a response. For each video chunk, the assistant outputs <|silent|> when it remains silent, or <|respond|> followed by response content, which can contain a conversational utterance or a tool call. This format explicitly distinguishes silent observation from response generation and accommodates different interaction behaviors within the same chunk-wise structure.

As shown in the figure below:

The main architectural addition is the Long-Horizon Interaction Memory (LHIM) module, implemented using an auxiliary summarization model. This model processes visual content leaving the active visual window together with all available textual context. It organizes information at the event level and produces summaries in QA form, which are inserted back into the interaction context to preserve historical information about past events after their visual observations are no longer available.

The data pipeline extends streaming interaction data with three new forms of supervision: Model-Initiated Conversation (MIC), Stream-Aware Tool Calling (SATC), and Interactive Visual Grounding (IVG). It proceeds from video preparation to task-specific synthesis and finally to streaming assembly and quality verification.

As shown in the figure below:

Source videos are re-encoded, resized, and divided into clips, with an MLLM generating clip-level descriptions of objects, actions, and scene context. MIC data is constructed by asking an MLLM to determine when to initiate a conversation and what to say, ensuring openings are supported by visual observations and avoid repeating earlier discussions. SATC data combines real execution records with MLLM-generated tool-calling traces, including tool-call arguments, returned results, and subsequent responses. To encourage selective tool use, randomly selected tool schemas are added to non-tool-calling samples. IVG examples are constructed from video grounding and tracking trajectories, where an MLLM rewrites annotations into streaming grounding interactions with bounding-box coordinates. Structural checks and task-specific MLLM verification filter synthesized trajectories to ensure chronological consistency and visual evidence support.

AURA 2.0 is initialized from Qwen3.5-9B and employs a three-stage curriculum to progressively introduce new capabilities without degrading existing response policies.

As shown in the figure below:

Stage 1 establishes a general streaming interaction and tool-use foundation using general streaming QA and SATC data. Stage 2 introduces visual grounding and model-initiated conversation with MIC and IVG data, while replaying earlier tasks to prevent degradation. Stage 3 focuses on capability composition and preference refinement, training on examples that combine multiple capabilities, such as Proactive QA with SATC and MIC with SATC.

To address the imbalance where silent steps account for over 94 percent of supervised control decisions, the authors combine class-balanced normalization with focal weighting for the control tokens <|silent|> and <|respond|>. Let pip_ipi​ denote the predicted probability of the ground-truth token at position iii, and let ℓi=−log⁡pi\ell_i = -\log p_iℓi​=−logpi​. Using SSS and RRR to denote the sets of supervised <|silent|> and <|respond|> positions, and OOO for all other supervised positions, the training objective is:

L=14(1∣S∣∑i∈S(1−pi)γℓi+1∣R∣∑i∈R(1−pi)γℓi)+1∣O∣∑j∈Oℓj\mathcal{L} = \frac{1}{4} \left( \frac{1}{|S|} \sum_{i \in S} (1 - p_i)^\gamma \ell_i + \frac{1}{|R|} \sum_{i \in R} (1 - p_i)^\gamma \ell_i \right) + \frac{1}{|O|} \sum_{j \in O} \ell_jL=41​(∣S∣1​i∈S∑​(1−pi​)γℓi​+∣R∣1​i∈R∑​(1−pi​)γℓi​)+∣O∣1​j∈O∑​ℓj​

with γ=2\gamma = 2γ=2. The focal factor reduces the contribution of confidently predicted control tokens, placing greater emphasis on difficult decisions. Response text and serialized tool-call content use standard cross-entropy without focal weighting. The losses of the two control classes are averaged separately to ensure equal base weights regardless of their respective numbers of supervised positions.

To support long-horizon streaming video understanding without continuously increasing visual context, LHIM compresses historical video observations into timestamped textual descriptions. The context is organized into a sliding window preserving the most recent 30 seconds of video and dialogue, and a context history storing compressed visual memories. Video preceding the sliding window is divided into 30-second slices and processed by a separate summarization MLLM, which partitions each slice into semantically coherent events. These events form level-1 memory records stored as text-only QA pairs. As the session progresses, older records are progressively merged into coarser level-2 and level-3 summaries, maintaining a fixed 30-second visual context while preserving recent history at event resolution.

Inference latency is optimized through session history caching, fast paths for silent input and output, and sentence-level speech handoff. Each request carries only the frames of the current video chunk, cached in a per-session history. Turns outputting <|silent|> end after a single decoding step without running speech synthesis, while speech synthesis starts before the MLLM finishes generating the full response by splitting text at sentence boundaries and forwarding completed segments.

Experiment

AURA 2.0 is evaluated across four public streaming benchmarks that assess proactive interaction, historical reasoning, real-time perception, tool use, and temporal understanding, along with targeted tests of model-initiated conversation, stream-aware tool calling, interactive visual grounding, and inference efficiency. The results show strong proactive and forward temporal reasoning, improved historical retrieval, and competitive real-time perception, with external tools and visual grounding contributing clear gains over the backbone. Efficiency measurements indicate low response latency and stable real-time speech delivery even under 16 concurrent sessions, with serving optimizations substantially reducing time to first audio.

Representative streaming interaction models consistently support real-time question answering, proactive question answering, and multi-response question answering. AURA 2.0 is the only listed model with reported support for all six compared capabilities, adding multimodal interaction context, stream-aware tool calling, and interactive visual grounding. Among earlier systems, these added capabilities are largely absent, with stream-aware tool calling reported only for JoyAI-VL-Interaction. All compared models support real-time QA, proactive QA, and multi-response QA. AURA 2.0 is the only listed system with reported support for MIC, SATC, and IVG. Earlier models generally lack MIC and IVG; SATC is reported only for JoyAI-VL-Interaction among prior systems.

AURA 2.0 demonstrates strong proactive interaction performance on IPIBench, particularly in proactive task management, where its category average is the highest among all evaluated models, including proprietary offline systems. In proactive monitoring, its timing accuracy outperforms several recent baselines and Gemini 3 Pro while approaching GPT-4o, and its repeated proactiveness score approaches Gemini 3 Pro. Human-level scores remain far above all model results across every subtask. AURA 2.0 achieves the highest proactive task management average among all evaluated models, with multi-task management exceeding Gemini 2.5 Pro and GPT-5.4. AURA 2.0 outperforms Qwen3.5-9B, Qwen3-VL-8B, and Gemini 3 Pro in proactive monitoring timing, approaching GPT-4o, and its repeated proactiveness score approaches Gemini 3 Pro. All evaluated models remain far below human-level performance, especially in interleaved reactive-proactive interaction and repeated proactiveness.

AURA 2.0 improves over its predecessor across the main StreamArena dimensions, with the largest gain in proactive interaction and a notable improvement on long-horizon historical retrospection. It also outperforms the other evaluated open-source streaming models in real-time perception and historical retrospection, and its tool-use accuracy surpasses the listed MiniCPM-o 4.5 and JoyAI-VL-Interaction results. The human reference remains far ahead of all models in every reported dimension. AURA 2.0 advances past JoyAI-VL-Interaction and MOSS-VL-Realtime in real-time perception and historical retrospection, with the strongest long-horizon L4 result among the listed open-source streaming models. Proactive interaction accuracy rises from 3.7% to 12.8%, overtaking JoyAI-VL-Interaction and MOSS-VL-Realtime. Tool-use accuracy also surpasses the listed JoyAI-VL-Interaction and MiniCPM-o 4.5 results, reaching 18.5%.

AURA 2.0 demonstrates strong proactive video understanding across the four ProactiveVideoQA categories. At the default emphasis, it outperforms JoyAI-VL-Interaction in every category and surpasses MOSS-VL-Realtime in three of the four categories. Its WEB performance is notably strong, exceeding all listed models at every emphasis level. AURA 2.0 outperforms JoyAI-VL-Interaction on WEB, EGO, TV, and VAD at the default emphasis. Compared with MOSS-VL-Realtime at the default emphasis, AURA 2.0 scores higher on WEB, EGO, and VAD, while MOSS-VL-Realtime remains slightly ahead on TV. AURA 2.0 achieves the best WEB PAUC among all listed models at every emphasis level, including proprietary offline baselines.

On PhoStream, AURA 2.0 achieves the highest Overall score among reported models and leads open-source streaming models on Instant, Backward, and Forward tasks. Its largest relative advantage appears in Forward reasoning, where it also reports the lowest Early Response rate and the highest Partly Correct rate. A remaining No Response rate indicates some responses are still missed. AURA 2.0 attains the best Overall score and exceeds the strongest proprietary baseline by over nine points. It leads open-source streaming models across Instant, Backward, and Forward tasks, with the greatest margin in Forward reasoning. AURA 2.0 records the lowest Early Response rate among all listed models, far below MOSS-VL-Realtime, Qwen3.5-9B, and JoyAI-VL-Interaction. It also achieves the highest Partly Correct rate, while a nonzero No Response rate points to some missed responses.

The evaluation covers streaming capability coverage, proactive interaction, stream understanding, proactive video question answering, and streaming reasoning benchmarks. AURA 2.0 is the only model reporting all six streaming interaction capabilities, and it consistently leads open-source baselines while also showing strong results against proprietary systems, including the highest proactive task management and overall reasoning scores plus the strongest WEB performance. Its main remaining limitations are a nonzero no-response rate on some streaming reasoning tasks and a large gap to human-level performance across interactive and proactive settings.


AI로 AI 구축

아이디어에서 출시까지 — 무료 AI 코코딩, 즉시 사용 가능한 환경, 최적의 GPU 가격으로 AI 개발을 가속화하세요.

AI 협업 코딩
바로 사용 가능한 GPU
최적의 가격

HyperAI Newsletters

최신 정보 구독하기
한국 시간 매주 월요일 오전 9시 에 이번 주의 최신 업데이트를 메일로 발송합니다
이메일 서비스 제공: MailChimp