One-sentence Summary

Flash-VStream is a memory-based video-language model that simulates human memory to process long video streams in real time and answer asynchronous user queries, achieving significant reductions in inference latency and VRAM consumption compared to existing models while introducing the VStream-QA benchmark for online video streaming question answering.

Key Contributions

• Flash-VStream is introduced as a video-language model that processes online video streams in real time through a learnable visual memory mechanism. This architecture disentangles visual encoding from answer decoding to enable simultaneous processing of continuous visual content and asynchronous user queries.
• VStream-QA is proposed as a novel question-answering benchmark specifically designed for online video streaming evaluation. Its question-answer-timestamp triplet design and extended video durations allow for comprehensive assessment of model capabilities on nearly infinite-length streams.
• The framework achieves significant reductions in inference latency and VRAM consumption compared to existing offline models. These efficiency improvements enable real-time understanding without the linear memory scaling observed in prior methods.

Introduction

Online video streaming underpins critical applications like autonomous robotics and continuous surveillance, where systems must continuously interpret visual data and respond to dynamic queries. Existing multimodal video-language models, however, are optimized for offline, finite-length videos and fail to handle streaming efficiently. Their reliance on uncompressed frame tokens quickly exhausts GPU memory and forces computationally expensive re-encoding whenever a new question arrives, making real-time interaction impractical. To overcome these bottlenecks, the authors introduce Flash-VStream, a model that mimics human cognitive processing by decoupling continuous visual encoding from question-driven decoding. By integrating a learnable Spatial-Temporal-Abstract-Retrieved memory mechanism, the system compresses redundant visual features and updates them in real time, enabling instant responses to user queries on extremely long streams while drastically reducing inference latency and VRAM consumption. The researchers also release VStream-QA, a dedicated benchmark designed to evaluate performance in realistic online streaming environments.

Dataset

• Dataset Composition and Sources: The authors introduce VStream-QA, a benchmark designed for online video stream understanding, divided into two distinct subsets. VStream-QA-Ego comprises 10 one-hour ego-centric clips sourced from the Ego4D dataset, while VStream-QA-Movie provides 22 thirty-minute movie clips from MovieNet. Combined, the dataset spans 21 hours of video covering practical daily activities and a wide range of film genres.

• Subset Details and Filtering Rules: The ego-centric subset contains 1.5K question-answer-timestamp triplets, and the movie subset holds 2K triplets. Questions fall into five balanced categories that assess scene understanding, event detection, and temporal reasoning, with a roughly even distribution between yes and no answers. A dedicated human review phase removes ambiguous or irrelevant queries, questions requiring external knowledge, items answerable without visual context, and incorrect or unclear responses.

• Data Usage and Evaluation Strategy: Rather than serving as a training corpus, the authors deploy VStream-QA as a comprehensive evaluation benchmark. It tests model capabilities in both online streaming and traditional offline configurations, with zero-shot video question answering metrics reported to establish performance baselines against existing methods.

• Processing Pipeline and Metadata Construction: The dataset follows a structured five-step generation process. Videos are temporally segmented into 30-second intervals, with eight frames sparsely sampled per interval to feed GPT-4V for dense captioning. GPT-4 subsequently deduplicates and condenses these captions into timestamped scene summaries, which directly inform question generation. Every QA pair is explicitly anchored to a specific timestamp, guaranteeing that answers rely exclusively on visual information available prior to that moment.

Method

The authors leverage a dual-process architecture to enable real-time online video stream understanding, as illustrated in the framework diagram. This architecture separates the system into two asynchronous processes: the frame handler and the question handler. The frame handler continuously processes incoming video frames, encoding them and writing the resulting features into a shared memory structure. The question handler, triggered by user queries, reads from this memory and generates responses in real time. The only inter-process communication occurs through the shared memory, which is written by the frame handler and read by both processes. This design allows for continuous encoding and immediate response generation, overcoming the latency limitations of conventional offline video LLMs that require all frames to be processed before any inference can begin.

The core of the system is the streaming visual encoder, which continuously processes video frames to generate embedded features. The authors use a pre-trained CLIP ViT-L model as the visual encoder, extracting patch tokens from each frame $V^{t} \in \mathbb{R}^{H \times W \times 3}$Vt∈RH×W×3 to produce a feature map $e^{t} \in \mathbb{R}^{P \times P \times D}$et∈RP×P×D. This feature map is then fed into the STAR memory mechanism, which is designed to handle information at different levels of granularity. The STAR memory consists of four components: spatial, temporal, abstract, and retrieved memory, along with a feature buffer that stores the most recent frames. The feature buffer acts as an intermediate storage, from which the spatial memory is updated using a First-In-First-Out (FIFO) queue, ensuring that the most recent and detailed spatial information is readily accessible.

The temporal memory integrates dynamic information over time, crucial for long-term retention. When the size of the temporal memory exceeds its capacity, the Weighted K-means Clustering algorithm is applied to condense the memory content into a fixed number of clusters, representing key events in the video. This process is illustrated in the diagram, where the feature map $e^t$et is processed through Weighted K-means Clustering to generate the updated temporal memory. The abstract memory supports high-level semantic concept interpretation through a Semantic Attention model, which synthesizes insights from both spatial and temporal memories into a more abstract, actionable knowledge. The retrieved memory focuses on recalling precise spatial details by identifying and retrieving the most substantial frame features, which are then used to supplement the temporal memory with more detailed spatial information.

The question handler process manages the real-time LLM decoder, which provides responses to user queries. When a question $Q^{t}$Qt is received, the LLM decoder first calculates the text embedding $I_{\text{text}}^{t} = f_{\text{embed}}(Q^{t})$Itextt​=fembed​(Qt). It then maps the STAR memory $M^{t}$Mt to the embedding space using a projector, resulting in $I_{\text{vision}}^{t} = f_{\text{proj}}(M^{t})$Ivisiont​=fproj​(Mt). The LLM decoder then generates the answer $A^{t} = f_{\text{LLM}}(I_{\text{text}}^{t}, I_{\text{vision}}^{t}).\text{decode}()$At=fLLM​(Itextt​,Ivisiont​).decode() in real time. This process ensures that the system can provide immediate responses to user questions, leveraging the most relevant information stored in the STAR memory. The entire system is designed to handle the continuous flow of video data and user interactions efficiently, enabling real-time online video stream understanding.

Experiment

Evaluated on real-time and standard video question answering benchmarks using GPT-3.5 for response quality and latency measurements for efficiency, the experiments validate the model's ability to process continuous video streams without frame-by-frame bottlenecks. The results demonstrate that the proposed STAR memory mechanism achieves state-of-the-art long video comprehension while significantly reducing computational overhead compared to existing architectures. Ablation studies and feature visualizations further confirm that temporal memory and semantic attention are critical for consolidating visual context and accurately tracking complex narratives over extended durations. Ultimately, the findings establish Flash-VStream as a highly efficient and accurate solution for real-time online video understanding.

The authors conduct an ablation study to evaluate the impact of different memory components and updating strategies on video understanding performance. Results show that the proposed Semantic Attention method outperforms other approaches, and the integration of temporal memory is crucial for maintaining high performance in long video comprehension tasks. The study also highlights the importance of balancing spatial and temporal memory sizes to achieve optimal results. Semantic Attention outperforms Q-Former and Sequential Q-Former in video understanding tasks. Temporal memory plays a vital role in maintaining high performance for long video comprehension. Balancing spatial and temporal memory sizes leads to optimal performance in video understanding tasks.

The authors provide training details for Flash-VStream, outlining the configuration for two distinct training stages. The model undergoes modality alignment followed by instruction tuning, with specific hyperparameters and component settings defined for each phase. Training is divided into two stages with distinct hyperparameters and component configurations. The model uses a Cosine decay learning schedule and AdamW optimizer for both stages. Components such as the Semantic attention and Projector are open for both stages, while the LLM and Visual encoder are frozen in Stage-1 and open in Stage-2.

The authors evaluate Flash-VStream against several competitive models on real-time video question answering benchmarks, focusing on accuracy and computational efficiency. Results show that Flash-VStream achieves superior performance with lower memory usage compared to other models. The model demonstrates strong capabilities in long video understanding while maintaining low inference latency. Flash-VStream outperforms other models in accuracy on real-time video question answering benchmarks. Flash-VStream achieves lower memory consumption compared to competing models. The model maintains high performance while enabling real-time inference through an efficient memory mechanism.

The authors investigate the impact of spatial and temporal memory dimensions on model performance in long video understanding tasks. Results show that varying these dimensions affects both accuracy and score, with optimal configurations yielding the best results. The model achieves strong performance with balanced memory settings, indicating the importance of memory design for effective video comprehension. Performance varies with different spatial and temporal memory configurations, with optimal settings yielding the best results. The model achieves high accuracy and score on both datasets with balanced memory dimensions. Memory design significantly influences long video understanding, with specific configurations outperforming others.

The authors conduct an ablation study to evaluate the impact of different components in the STAR memory mechanism on video question answering performance. Results show that the presence of all memory types contributes to optimal performance, with the full configuration achieving the highest accuracy on both datasets. The temporal memory component appears to be particularly important, as its absence leads to a significant drop in performance. All components of the STAR memory mechanism contribute to performance, with the full configuration achieving the highest accuracy. The absence of temporal memory leads to a significant performance drop, indicating its importance in long video understanding. The full STAR memory configuration achieves the best results on both VS-ego and VS-movie datasets.

The experiments evaluate the Flash-VStream model through ablation studies on memory components, dimensional tuning of spatial and temporal memory, and comparative benchmarks on real-time video question answering. These evaluations validate the effectiveness of the proposed Semantic Attention mechanism and demonstrate that temporal memory is essential for maintaining high accuracy in long-form video comprehension. The results further confirm that carefully balancing spatial and temporal memory dimensions is critical for optimizing overall system performance. Ultimately, the complete STAR memory configuration consistently achieves the best results while enabling efficient real-time inference with substantially reduced memory consumption compared to competing approaches.