One-sentence Summary

Nanjing University researchers propose Light-Omni, a reflexive multimodal agent framework that employs dual contextual states—a globally consolidated multimodal script and a parametric latent state—to achieve semantically aligned retrieval and instant actions in a single forward pass, thereby eliminating iterative reasoning and outperforming M3-Agent with an average $2.4\%$2.4% accuracy gain, a $12.1\times$12.1× speedup, and a $2.6\times$2.6× improvement in GPU memory efficiency.

Key Contributions

• The paper introduces Light-Omni, a multimodal agent framework that replaces slow, iterative “detective-style” reasoning with a reflexive single-forward-pass design for long-horizon video understanding.
• The framework constructs dual contextual states: a global state consolidated from episodic memory via hierarchical merging to preserve recent details and summarize past events, and a parametric latent state conditioned on the global context that directly drives actions and retrieval embeddings, bridging the semantic gap between queries and memory without iterative reasoning.
• Across multiple long-video benchmarks, Light-Omni outperforms M3-Agent with a 2.4% accuracy gain, a 12.1× speedup, and a 2.6× GPU memory reduction. It also serves as a memory system to enhance the performance and efficiency of existing multimodal large language models.

Introduction

Multimodal large language models (MLLMs) suffer from a fundamental amnesia when processing continuous, long-horizon video streams because their finite context windows discard earlier events. To compensate, memory-augmented video agents store clips as discrete entries with coarse textual metadata and retrieve them via similarity search, but a persistent semantic gap between user queries and these sparse representations degrades retrieval accuracy and forces expensive, multi-step reasoning workflows. The authors argue that the root cause is the lack of an explicit global context and the misalignment between query and memory distributions. They introduce Light-Omni, a framework that replaces heavy reasoning with reflexive responses by constructing dual contextual states: a Global State that consolidates a non-parametric, compact multimodal script from episodic memory, and a Latent State that produces semantically aligned embeddings, enabling direct, accurate retrieval without reasoning. This design bridges the semantic gap, yielding strong accuracy improvements while dramatically reducing latency and memory footprint.

Dataset

The authors create a custom training dataset by automatically processing publicly available long videos. Key points:

• Source and composition: Offline, publicly available long videos serve as raw material.
• Automated synthesis pipeline: Textual and auditory instructions are injected at varied timestamps to provoke synchronized responses and dense intermediate supervisory signals (e.g., memory updates and executed actions). To increase diversity and mimic multi‑session interactions, heterogeneous video clips are concatenated.
• Scale: 43 k training samples.
• Schema and metadata: The samples carry time‑aligned instructions, generated responses, and step‑by‑step supervisory labels such as memory state changes and action decisions.
• Usage in training: The dataset is employed to train the three‑way multi‑LoRA adapters.
• Memorization and generation adapters are optimized with a standard next‑token‑prediction (NTP) loss.
• The reaction adapter uses a hybrid objective: a discrete classification loss over the action space (search, speech) plus a contrastive retrieval alignment term that links the predicted retrieval embedding to the correct memory embedding.
• Processing / cropping: No explicit cropping strategy is described; instead, the pipeline concatenates clips and injects events at multiple time steps, effectively creating long, instruction‑rich sequences.

Method

The authors propose Light-Omni, a framework designed to equip general Multimodal Large Language Models (MLLMs) with long-term memory and low-latency inference capabilities for long video understanding. Unlike recent video agents that rely on a detective-style workflow involving iterative reasoning and tool invocation, which introduces prohibitive latency, Light-Omni achieves near-constant latency through a dual contextual state design. This architecture decouples the slow, incremental consolidation of global context from the fast, real-time reflexive response driven by the latent state.

As illustrated in the framework diagram, the system operates via two main phases: Real-time Reflex and Sleep-time Consolidation. Given an omni-modal input segment at timestamp $t$t, $\pmb{\mathcal{Z}}^t = \{\pmb{V}^t, \pmb{A}^t, \pmb{T}^t\}$Zt={Vt,At,Tt}, comprising visual frames, audio signals, and textual instructions, the model produces actions conditioned on the current context while updating memory in a post-response consolidation phase. This process is mathematically formalized as: $\{\boldsymbol{\mathcal{A}}^t, \boldsymbol{\mathcal{S}}^t, \boldsymbol{\mathcal{M}}^t\} = \Phi(\pi; \boldsymbol{\mathcal{I}}^t, \boldsymbol{\mathcal{S}}^{t-1}, \boldsymbol{\mathcal{M}}^{t-1}), \quad t = 0, 1, 2, \dots${At,St,Mt}=Φ(π;It,St−1,Mt−1),t=0,1,2,… where $\mathcal{A}^t$At denotes reflexive actions (e.g., response generation, search), $\pmb{S}^t$St represents the dual contextual states, and $\mathcal{M}^t$Mt is the structured multimodal long-term memory. $\Phi(\cdot)$Φ(⋅) defines the system dynamics, parameterized by the model $\pi$π.

As shown in the figure below, this coupled design contrasts sharply with iterative reasoning paradigms. While traditional agents suffer from high latency and local vision due to the semantic gap between queries and offline-constructed memory representations, Light-Omni leverages dual contextual states to maintain a global context and directly generate semantically aligned embeddings for evidence aggregation. By incorporating sleep-time memory consolidation, it enables seamless online interaction with both global vision and near-constant latency.

Multimodal Long-Term Memory Memory serves as the cornerstone of Light-Omni. The authors decompose the multimodal memory $\mathcal{M}$M into three distinct components:

1. User Profile ($\mathcal{M}_p$Mp​): Stores fundamental attributes of individuals, including visual avatars, personal preferences, and personality traits, enabling personalized interactions.
2. Semantic Memory ($\mathcal{M}_s$Ms​): Distills abstracted facts, concepts, and relationships from streaming inputs, stored as textual key-value pairs for quick recall.
3. Episodic Memory ($\mathcal{M}_e$Me​): Serves as a chronological ledger of past events, logging historical interactions as structured, omni-modal scripts comprising explicit timestamps, visual scene descriptions, auditory cues, and assistant responses.

For efficiency, Light-Omni processes the continuous video stream sequentially. Given the current input $\mathcal{Z}^t$Zt and the global state $S_g^{t-1}$Sgt−1​, the model directly generates $\text{Topic}^t = \{\mathcal{M}_s^t, \mathcal{M}_e^t\}$Topict={Mst​,Met​} via $p(\text{Topic}^t \mid \mathcal{T}^t, S_g^{t-1}; \pi)$p(Topict∣Tt,Sgt−1​;π). The user profile $\mathcal{M}_p$Mp​ is updated only when the number of buffered topics exceeds a predefined capacity $\tau_t$τt​.

Dual-State Design At the core of Light-Omni lies a novel dual-state design, $\pmb{S}^t = \{\pmb{S}_g^t, \pmb{S}_l^t\}$St={Sgt​,Slt​}, where both states are derived from and grounded in the multimodal memory $\mathcal{M}^{t-1}$Mt−1. This mechanism ensures that $S_g^t$Sgt​ endows the agent with a global context for response generation, memory retrieval, and consolidation, while $S_l^t$Slt​ enables action execution via fast, implicit reflexes rather than heavy, explicit reasoning.

• Global State ($S_g$Sg​): Acting as a non-parametric contextual backbone, the global state $S_g$Sg​ provides a compact, hierarchical representation of historical context. Rather than naively concatenating the entire memory bank, $S_g$Sg​ is constructed via a hierarchical merging strategy that exhibits a resolution-decaying property. Specifically, the episodic scripts $\mathcal{M}_e$Me​ are progressively summarized and merged based on a capacity factor $k$k. When the number of level-$i$i nodes $N_i$Ni​ reaches $k+1$k+1, the oldest $k$k nodes are consolidated into a single higher-level node $m_{i+1}$mi+1​: $\boldsymbol{S}_g^t = \text{Update}(\boldsymbol{\mathcal{M}}_e^{t-1} \cup \Delta\boldsymbol{\mathcal{M}}_e^t, k)$Sgt​=Update(Met−1​∪ΔMet​,k) $\text{s.t.} \quad N_i \geq k+1 \implies m_{i+1} \leftarrow \text{Merge}(\{m_i\}_{1:k}).$s.t.Ni​≥k+1⟹mi+1​←Merge({mi​}1:k​). This operation ensures that recent fine-grained details and distant high-level summaries coexist within a bounded context window.

• Latent State ($S_l$Sl​): Conditioned on $S_g^t$Sgt​, the latent state directly generates semantically aligned embeddings for accurate retrieval. The authors introduce learnable soft prompts $P_{soft} \in \mathbb{R}^{N \times D}$Psoft​∈RN×D appended to the input sequence. Within a single forward pass, the backbone model $\pi_\theta$πθ​ processes the augmented sequence $[S_g^t, \pmb{\mathcal{T}}^t, P_{soft}]$[Sgt​,Tt,Psoft​] and directly yields corresponding hidden features $H^t$Ht. These specific hidden states are then decoded in parallel via task-specific heads: $\boldsymbol{z}_{ret}^t = \operatorname{Pro}(\boldsymbol{h}_{ret}^t) \in \mathbb{R}^{D'}, \quad (\text{Retrieval Embedding})$zrett​=Pro(hrett​)∈RD′,(Retrieval Embedding) $\boldsymbol{a}_{act}^t \sim \operatorname{Bernoulli}(\boldsymbol{p}_{act}^t) \in \{0, 1\}^2, \quad (\text{e.g., speech, search})$aactt​∼Bernoulli(pactt​)∈{0,1}2,(e.g., speech, search) where $\boldsymbol{p}_{act}^t = \text{Sigmoid}(\text{FC}(\boldsymbol{h}_{act}^t))$pactt​=Sigmoid(FC(hactt​)) represents the independent trigger probability for each action. The projection head $\text{Pro}(\cdot)$Pro(⋅) is an MLP-ReLU-MLP module. The retrieval embedding $\boldsymbol{z}_{ret}^t$zrett​ is added to the original retrieval embedding to rectify the query representation to match the memory distribution, bridging the semantic gap between queries and memory representations.

Learning Strategy and Optimization

• Dataset Construction: The authors develop an automated pipeline to synthesize a customized training dataset from publicly available offline long videos. They inject textual and auditory instructions at diverse time steps to generate synchronized responses and dense intermediate supervisory signals, yielding 43k training samples.
• Multi-LoRA Design: To mitigate optimization conflicts between memorization, generation, and reaction tasks, the authors employ a multi-LoRA architecture to decouple these capabilities. During inference, the system dynamically switches adapters based on the current state.
• Training: For memorization and generation tasks, the corresponding adapters are optimized using the standard Next Token Prediction (NTP) objective: $\mathcal{L}_1 = -\sum_{t=1}^T \log P_\theta(y_t \mid x, y_{<t})$L1​=−∑t=1T​logPθ​(yt​∣x,y<t​) In contrast, the reaction adapter is optimized via a hybrid objective combining discrete classification and contrastive retrieval alignment: $\mathcal{L}_2 = \sum_{c \in \mathcal{C}} \mathcal{L}_{CE}(\boldsymbol{p}_c, \boldsymbol{y}_c) - \lambda \log \frac{\exp(\boldsymbol{z}_{ret} \cdot \boldsymbol{k}^+ / \tau)}{\sum_{j=1}^B \exp(\boldsymbol{z}_{ret} \cdot \boldsymbol{k}_j / \tau)}$L2​=∑c∈C​LCE​(pc​,yc​)−λlog∑j=1B​exp(zret​⋅kj​/τ)exp(zret​⋅k+/τ)​ where $\mathcal{C} = \{\text{search}, \text{speech}\}$C={search,speech} denotes the action space, $\boldsymbol{z}_{ret}$zret​ is the predicted retrieval embedding, and $\boldsymbol{k}^+$k+ is the corresponding positive memory embedding.
• Efficiency Optimization: To further minimize latency, the authors implement two critical optimizations. First, Feature Caching ensures that visual and auditory signals are encoded without repetition across multiple stages, yielding an initial 8% reduction in latency. Second, Redundancy Pruning leverages the inherent temporal redundancy in long videos to dynamically perform token pruning based on feature similarity.

As shown in the figure below, these strategies cumulatively reduce inference latency by 42% without compromising accuracy.

Experiment

Light-Omni is evaluated on long-video benchmarks including VideoMME-long, LVBench, HippoVlog, and OVO-Bench against general MLLMs, memory-augmented methods, and reasoning agents. The experiments validate its strong performance on long videos, its ability to serve as a universal memory module that boosts existing MLLMs, the dual-state mechanism's retrieval robustness under textual and audio noise, and its significant latency reduction for interactive use. The dual-state design ensures precise, noise-resistant retrieval, while the system maintains near-constant memory and inference time regardless of video duration, enabling scalable long-term multimodal assistance.

Light-Omni reaches 58.0% average accuracy, surpassing larger commercial and open-source models, while delivering drastic speed and memory gains over its baseline. When used as a memory module, it consistently boosts accuracy and reduces resource consumption across multiple foundation models. Light-Omni outperforms GPT-4o, Gemini-2.0-Flash, and Qwen2.5-VL-72B with an average accuracy of 58.0% on long-video benchmarks. Compared to its Qwen2.5-Omni-7B baseline, Light-Omni achieves a 9.5% accuracy gain, roughly 20× faster inference, and over 3× lower memory usage. Adding Light-Omni to Qwen2.5-VL-7B, Qwen3-VL-8B, and Gemini-2.0-Flash improves their average accuracy by 4.9%, 2.5%, and 3.8% respectively, while for Qwen2.5-VL-7B it accelerates inference by 7.2× and reduces memory 2.5× on LVBench.

On VideoMME-long, Light-Omni delivers the highest accuracy (66.1%) while achieving per-query latency of only 2.28 seconds, over an 8× speedup compared to the iterative M3-Agent. Its offline memorization cost per video is just 1.52 seconds, and the total processing time for the full benchmark is 1,753 seconds, nearly half of M3-Agent’s 3,313 seconds. This efficiency comes without sacrificing quality, as Light-Omni’s accuracy marginally surpasses the runner-up WorldMM. Light-Omni attains the best accuracy (66.1%) on VideoMME-long, outperforming M3-Agent (61.8%) and WorldMM (66.0%). Per-query latency for Light-Omni is 2.28 s, more than 8× faster than M3-Agent’s 25.91 s. Light-Omni’s offline memorization cost for an entire video is only 1.52 s, compared to 9.82 s for M3-Agent. Despite making more LLM calls (153.5 vs. 81.2), Light-Omni completes the full video evaluation in 1,753 s, nearly half the time of M3-Agent.

An ablation on VideoMME-long compares strategies for global context. The hierarchical global state achieves the highest accuracy, outperforming the retrieval-only baseline by 2.99 points and uniform frame sampling by 1.99 points. Short-term memory as context yields only a marginal gain of 0.56 points, indicating that hierarchical memory more effectively captures long-term temporal dependencies. Hierarchical global state (S_g) surpasses the retrieval-only baseline by 2.99 percentage points and uniform sampling by 1.99 points on VideoMME-long. Short-term memory as global context adds only 0.56 points over the baseline, trailing the 1.0-point gain from uniform frame sampling.

On the LVBench benchmark, jointly fusing text embeddings with latent features via element-wise addition yields the highest accuracy (49.90%), slightly above the text-only retrieval baseline. Using only latent embeddings for retrieval reduces accuracy by 2.78 percentage points, while prepending latent features as soft prompts causes a larger 5.81 point drop, showing that naive integration of latent representations can impair retrieval performance. Fusing text and latent embeddings improves accuracy by 0.64 percentage points over the text-only retrieval baseline. Relying solely on latent embeddings for retrieval leads to a 2.78 percentage point accuracy decline relative to the baseline. Attaching latent features as soft prompt tokens causes the largest drop, falling 5.81 percentage points below the text-only baseline.

Light-Omni is evaluated on long-video understanding benchmarks, where it surpasses larger commercial and open-source models while delivering substantial speed and memory improvements over its baseline. As a memory module, it consistently enhances accuracy and reduces resource consumption when integrated into various foundation models. Ablation studies further demonstrate that a hierarchical global state effectively captures long-term temporal dependencies, and that jointly fusing text with latent embeddings improves retrieval accuracy, whereas naive latent-only integration degrades performance.