One-sentence Summary

Researchers from The Hong Kong University of Science and Technology et al. introduce StreamArena, a benchmark for hour-scale interactive streaming video understanding containing 243 full-length videos and 3,646 open-ended question-answer pairs, and StreamMind, a two-tier architecture that offloads latency-critical interaction to frontend workers while backend workers asynchronously build persistent multimodal memory to improve long-horizon comprehension and reduce query latency.

Key Contributions

• StreamArena is a benchmark for hour-scale streaming video understanding, containing 243 videos averaging 88.8 minutes and 3,646 open-ended question-answer pairs that jointly evaluate real-time perception, historical retrospection, proactive interaction, and multimodal tool use.
• Experiments across five system classes reveal that existing designs support only subsets of these capabilities, exposing specific failure conditions such as lost visual evidence from text conversion, inability to recover distant events from short-term memory, and degraded fine details under repeated compression.
• The StreamMind architecture decouples latency-critical interaction from asynchronous memory construction and retrieval via independently scheduled frontend and backend workers, outperforming the strongest streaming baselines by 53.7% to 228.1% across all four capabilities while reducing query-to-answer latency by 66.2% through persistent state reuse.

Introduction

The authors address the challenge of building always-on multimodal agents for continuous, hour-scale scenes such as embodied robotics and wearables. Traditional turn-based models passively process pre-segmented clips and wait for user prompts, missing fleeting events and failing to intervene proactively. Existing benchmarks exacerbate this by using short videos and multiple-choice formats that allow language priors and recency shortcuts to inflate scores, while contemporary streaming systems each target only a subset of required capabilities (real-time perception, historical retrospection, proactive interaction, and tool use) and show stark trade-offs. The authors contribute StreamArena, a benchmark with 243 full-length videos (~89 minutes each) and 3,646 open-ended, causally timestamped tasks, and StreamMind, a two-tier streaming architecture that decouples frontend interaction from asynchronous memory construction and retrieval, yielding substantial gains across all four evaluation dimensions.

Dataset

The authors introduce StreamArena, a benchmark for evaluating always-on streaming assistants across hour-long videos with multi-turn interactions.

Dataset composition and sources

• 243 videos sourced from YouTube, spanning seven domains (e.g., vlogs, tutorials, news).
• Each video is at least 60 minutes long (average 88.8 minutes, maximum 134.2 minutes), has 1080p or higher resolution, and contains English or Chinese audio. Sensitive, inappropriate, and political content is excluded.
• 3,646 question–answer pairs are finalized after a three-stage annotation pipeline: PhD-level annotators draft ~20 multi-turn pairs per video, two cross-validators refine them, and a third annotator audits the question, answer, evidence, and timestamps. Approximately 73% of drafts survive, yielding around 15 final pairs per video.
• All questions and supporting evidence segments carry explicit timestamps; consecutive questions from the same video maintain conversational continuity.

Sub-task breakdown (based on evaluated capabilities)

• Real-time multimodal perception: 263 pairs requiring joint reasoning over synchronized audio and video.
• Historical retrospection: 877 pairs querying past events from the stream. Median evidence-to-query gap is 12.1 minutes (IQR 5.7–25.7 minutes); 49 questions have the nearest evidence more than one hour before the query. 20.6% of retrospection questions fall beyond a 30-minute gap.
• Multimodal tool utilization: 1,732 pairs that demand external search (e.g., Google) for information unavailable from the video stream or the assistant’s parametric memory.
• Proactive interaction: 774 pairs where the agent must monitor autonomously and alert when a specified event occurs, without receiving a new prompt at the moment. 12.0% of proactive tasks require monitoring for more than 4 minutes.
• Additionally, 189 questions (21.6%) require joint attention to at least two distinct evidence segments; the most demanding question spans 15 segments.

How the dataset is used

• StreamArena serves as a zero-shot evaluation benchmark. No training or fine-tuning split is provided; models are tested directly on the released set.
• For explicitly triggered tasks (retrospection and tool use), factual answer accuracy is the primary metric, judged by Gemini 3.1 Pro via a strict binary check against the ground truth.
• For tool-utilization questions, accuracy reflects end-to-end tool-enabled answers, not tool invocation traces.
• Proactive tasks are evaluated on both content correctness and temporal vigilance: a response is considered correct only if the alert’s content matches the ground truth and the detection timestamp falls within [-0.5 s, +2.0 s] of the annotated event time.
• Response latency is measured from query arrival to answer delivery for reactive tasks, excluding continuous stream processing done before the query.

Method

The authors introduce StreamMind, an architecture designed to decouple latency-sensitive interaction from long-horizon cognition for always-on streaming assistants. To support the diverse capabilities required by the StreamArena benchmark, such as real-time multimodal perception, historical retrospection, multimodal tool utilization, and proactive interaction (as illustrated in the framework diagram below), StreamMind employs a two-tier architecture.

As shown in the figure below, the system is divided into a frontend tier that interfaces with users and performs task dispatch, and a backend tier that maintains persistent memory and executes retrieval-intensive reasoning.

Frontend Interaction and Dispatch The Front Worker serves as the interaction gateway and high-level dispatcher. Given a user turn, recent causal observations, and the conversation state, it selects among three execution paths. It answers immediately when the context is sufficient, formulates a retrieval brief for the backend when the request requires historical or external evidence, or instantiates a Monitor Worker when the user asks the system to watch for a future condition. Each Monitor Worker maintains an independent lifecycle for its assigned condition and notifies the frontend only when that condition is satisfied, ensuring that persistent vigilance does not occupy the Front Worker or delay unrelated interactions.

Asynchronous Memory Construction In parallel with frontend interaction, the Memory Writer continuously transforms incoming frames and speech observations into the Memory Bank. Its hierarchical event layer organizes local actions into micro events and progressively aggregates them into macro and super events. A complementary entity relation graph links recurring people, objects, locations, and their relations across time. Events retain key entities, temporal boundaries, textual descriptions, and representative frames, preserving both compact semantic structure and retrievable visual evidence. Because memory construction is independent of user queries, the stream is consolidated before a later question reveals which evidence will be needed.

Backend Retrieval and Coordination When the Front Worker delegates a mission, the Router Worker acts as the coordinator. It decomposes the request into Recall and Search subtasks, executes independent subtasks concurrently, inspects their returned evidence, and either issues another round of retrieval or synthesizes the final response. The Recall Worker performs content-addressable retrieval over the event hierarchy and entity graph, returning relevant descriptions, relations, and historical key frames. The Search Worker grounds visible entities through image retrieval and acquires information unavailable from the stream through external text search. Their composition supports multistage requests: the Router can first recover an earlier event and its key frame through Recall, then use that frame as the visual anchor for Search. Recall and Search remain specialized and do not communicate directly; the Router mediates their evidence exchange and global reasoning state.

Decoupled Execution The architecture supports two execution modes. Continuous ingestion drives monitoring and memory construction, while event-driven messages create Monitors or activate backend recall or external search only when required. Every query-driven memory read is bounded by the corresponding query time, preserving causal access to the stream. By pairing a non-thinking Front Worker with thinking-enabled backend workers, StreamMind supports low-latency interaction, proactive monitoring, historical recall, and external tool use.

Experiment

The evaluation is conducted on StreamArena, a benchmark of 3,646 multi-turn question-answer pairs spanning 243 long videos and requiring real-time multimodal perception, historical retrospection, tool use, and proactive monitoring. Five architecture groups are compared under query-triggered protocols, with StreamMind—the only system combining continuous ingestion, persistent memory, retrieval, and proactive monitoring—substantially outperforming streaming baselines across all capabilities, particularly in tool use and long-horizon comprehension. StreamMind also reduces online response latency by reusing precomputed state, and diagnostic analyses confirm that multimodal evidence, sufficient temporal coverage, and memory retrieval are critical for robust streaming performance.

Existing streaming video benchmarks suffer from recency shortcuts and lack true continuous monitoring, while offline benchmarks support long, omni-modal video but omit interactive capabilities. StreamMind introduces explicit worker coordination for tool use and proactive triggering, achieving substantial relative gains over streaming baselines and cutting response latency by roughly two-thirds through state reuse. Diagnostic analyses confirm that visual input is critical for real-time perception, with speech providing complementary improvements. StreamMind’s worker coordination yields a 228.1% relative improvement on tool use compared to the strongest streaming baseline. On real-time perception tasks, visual-only input reaches 26.8% accuracy; adding speech boosts it to 32.4%, showing complementary multimodal benefit.

StreamArena temporal strata reveal distinct demand profiles: historical retrospection questions are spread across horizons up to more than 30 minutes, with the largest share in the 5–15 minute range, whereas proactive interaction questions overwhelmingly concentrate within the 30-second to 4-minute window, highlighting the need for rapid, unprompted event detection. Over half of proactive interaction questions fall into the 30-second to 4-minute layer, with only 12% beyond 4 minutes, emphasizing continuous short-latency vigilance. Historical retrospection questions are distributed across four strata, peaking at 5–15 minutes (37.6%) but with similar proportions below 5 minutes and above 30 minutes, indicating broad temporal recall demands. One third of proactive questions require response within 30 seconds, reinforcing the challenge of autonomous monitoring where no explicit query is issued.

StreamMind consistently achieves the highest accuracy among streaming systems on all four StreamArena capabilities, with relative improvements over the strongest streaming baselines reaching 58.4% for real-time perception, 53.7% for historical retrospection, 228.1% for tool utilization, and 54.7% for proactive monitoring. Its retrieval-augmented multimodal memory yields gains on every long-horizon retrospection gap (24.0–73.0%), while human performance drops from 80.7% HR with rewatching to 63.4% without it, underscoring the difficulty of hour-scale streaming recall. The system uniquely unifies continuous ingestion with all four capability types, whereas offline turn-based models cannot perform proactive interaction or maintain state across the video. StreamMind improves tool use accuracy by 228.1% over the strongest streaming baseline, driven by explicit coordination among router, recall, and search workers. Human historical retrospection degrades from an 80.7% average offline to 63.4% in a streaming condition, showing the inherent challenge of retaining evidence over long intervals without rewatching.

StreamMind drastically reduces query-to-answer latency compared to the same backbone running offline inference, while retaining most of its accuracy. Across question types, relative latency reductions reach 84.6% for real-time perception, 73.9% for historical retrospection, and 55.3% for tool use, and the system achieves a weighted average latency of 27.5 seconds versus the backbone's 81.4 seconds. StreamMind also records the lowest latencies among all tested models, including closed-source alternatives. Using the shared Qwen backbone, StreamMind reduces latency by 84.6% on real-time perception, 73.9% on historical retrospection, and 55.3% on tool questions, while preserving 89.7% of the pooled accuracy. StreamMind’s proactive memory construction yields a weighted average latency of 27.5 seconds, down from 81.4 seconds for the offline backbone, giving it the fastest response times among all compared systems.

Visual input is essential across all tasks, with ASR-only input yielding particularly low accuracy for real-time perception and historical retrospection but moderate accuracy for tool utilization. Adding audio to visual input provides complementary gains for real-time perception and a small boost for historical retrospection, while tool utilization accuracy remains unchanged. The multimodal condition achieves the highest overall accuracy, driven primarily by visual evidence. ASR-only input yields 4.2% accuracy on real-time perception, 16.7% on historical retrospection, but a relatively high 42.5% on tool-utilization questions. Visual-only input raises real-time perception to 26.8% and historical retrospection to 37.8%, demonstrating that visual evidence is critical for these tasks. Adding audio to visual input further improves real-time perception by 5.6 percentage points to 32.4%, and historical retrospection by 3.3 points to 41.1%. Tool-utilization accuracy with visual input (60.5%) is essentially the same when audio is added (60.0%), indicating speech provides no additional benefit for this category.

StreamMind is evaluated on StreamArena, which covers real-time perception, historical retrospection, tool utilization, and proactive monitoring. Its worker coordination and state reuse yield substantial accuracy improvements and the lowest latency among streaming systems. Ablation studies reveal that visual information is crucial, with speech providing complementary multimodal benefits, and proactive tasks demand immediate detection. The system uniquely integrates continuous ingestion with all capability types, while human recall degrades without rewatching, underscoring the difficulty of live streaming memory.