One-sentence Summary

VOICEMEM, developed by Nanyang Technological University, National University of Singapore, Tsinghua University, and The Chinese University of Hong Kong, introduces a streaming dual-brain memory architecture with parallel informational and emotional components that enables memory-aware speech language model training, long-horizon evaluation, and decoupled deployment, outperforming MEM0 by nearly $30$30 in $\text{top-}5$top-5 retrieval, achieving state-of-the-art persona benchmarks with a $1.89$1.89 aggregate gain, and completing retrieval in $134\,\text{ms}$134ms without adding conversational delay, thereby providing a practical foundation for real-time, personalized, and emotionally aware speech interaction.

Key Contributions

• Introduces VOICEMEM, a streaming dual-brain memory architecture that separates factual memory (left brain) from affective memory (right brain), with cross-brain associations and streaming memory I/O designed for real-time speech language models.
• The left brain uses a two-level schema–entity index with query-driven cluster emergence to maintain accuracy under a strict top-5 retrieval budget, while the right brain models the user via independent and cross-entity persona nodes with short- and long-horizon affective attribution, achieving state-of-the-art results on three persona benchmarks and a 1.89-point aggregate improvement over the prior best system.
• Hides retrieval inside standard VAD latency via a four-stage streaming query, completing retrieval in 134 ms with 430 memory tokens, reaching 91.2 on LoCoMo, and improving three different memory backends by 15.8–29.5 points, demonstrating practical deployment without added conversational delay.

Introduction

Memory is what transforms a conversational system from an intelligent tool into a human-centered partner, yet recent advances in memory systems and speech dialogue models have not been merged into a complete solution. Three obstacles block this integration: unified architecture for both informational and emotional intelligence, high information density under zero latency (where existing retrieval breaks the 500 ms real-time budget and top-100 outputs overwhelm speech models), and infrastructure evolvability as both fields move rapidly. The authors present VOICEMEM, a streaming dual-brain memory framework for real-time spoken interaction. The left brain uses a two-level schema-entity architecture with an emergence mechanism to manage factual memory, while the right brain models emotion and persona through independent and cross-entity nodes with short- and long-term affective attribution. A four-stage streaming query hides retrieval within standard VAD latency, and the framework is decoupled for model adaptation with SLM-verified blackbox OPD training and a standalone memory engine that transfers across backends. Experiments show consistent gains over prior state-of-the-art across information memory (+46.1% over Mem0), persona memory (+16.8%), and long-horizon audio memory (+41.3%), while supporting low-latency streaming retrieval at a top-5 budget.

Dataset

Dataset Composition and Sources

• The authors construct a large-scale, memory-dependent conversational dataset called CHATMEM-400K, used to train and evaluate long-horizon voice assistants.
• The dataset is built through a black-box online distillation pipeline, where proprietary models (Qwen2.5-Omni, Qwen3-Omni, and Step-Audio2-Mini) act as teachers, and the target models are distilled from them (Qwen3.5-Omni and Step-Audio2, respectively).
• The construction process is organized into four stages, each contributing distinct components to the final dataset.

Key Details for Each Subset

• CHATMEM-400K: A large-scale corpus of memory-dependent conversations, generated via an iterative pipeline. It includes:
  • Synthetic user histories built from a core persona, progressively instantiated as: Persona → Background → Events → Messages → Memory.
  • Conversations that cover knowledge-, emotion-, or persona-oriented goals, with challenges such as recall, reference resolution, contradiction, and multi-memory reasoning.
  • Contrastive distillation pairs that compare responses with different memory-use behaviors.
  • Verification steps that filter for necessity, faithfulness, and quality.
• CHATMEM-BENCH: A human-curated benchmark subset, consisting of 316 questions drawn from 15,314 turns and 53 hours of dialogue. Every session is curated by humans, from topic selection to memory-world construction and challenge design. It covers four dimensions with 14 fine-grained categories: Information, Persona, Affective Attribution, and Paralinguistics & Environment.

How the Paper Uses the Data

• The training pipeline uses CHATMEM-400K for supervised fine-tuning (SFT) in an iterative loop, where the generator is updated after each verification pass.
• Human-curated harder samples, with richer emotional and personality dynamics and multimodal inputs (e.g., acoustic questions), are added to augment training.
• CHATMEM-BENCH is used as a primary evaluation benchmark for the final models, since existing benchmarks do not fully capture real-world voice assistant requirements.

Processing and Metadata Construction

• The authors use online distillation to mitigate catastrophic forgetting during adaptation, meaning the training data is generated and verified in a loop rather than being a static corpus.
• The verification step filters generated conversations for necessity, faithfulness, and quality, ensuring only high-quality memory-dependent dialogues are retained.
• The benchmark construction requires substantial human effort, and the authors note that a detailed introduction of the benchmark and annotation pipeline is deferred to a separate follow-up technical report.
• The benchmark evaluates models across four dimensions, with categories such as update tracking, temporal reasoning, synthesis, abstention, profile constraints, preference-aligned recommendation, implicit-need inference, emotion-target attribution, attachment-guided decisions, affect-aware phrasing, background-sound recall, acoustic-scene inference, multi-user conversational memory, and acoustic style adaptation.

Method

3 VoiceMem: Streaming Dual-brain Architecture

The authors instantiate the proposed streaming dual-brain architecture as VOICEMEM to meet three requirements: maximize information density under a limited memory budget, develop a long-term understanding of personality while learning to perceive and adapt to emotions and attitudes, and add no perceptible latency to real-time spoken dialogue. The architecture comprises two parallel cognitive structures: a left brain that handles factual memory with efficient access, and a right brain that maintains persona and affective knowledge. These two brains operate concurrently, connected by cross-brain associations that enable joint retrieval during conversation.

3.1 Left Brain: Efficient Memory Access

The left brain in VOICEMEM supports efficient memory storage and retrieval under the tight context and latency budgets of online spoken interaction. Existing systems face two key challenges. First, retrieving up to top-100 candidates improves coverage but overwhelms the limited context capacity of speech-language models, whereas restricting retrieval to top-5 risks omitting relevant memories. Second, conventional memory pipelines often require 2 to 3 seconds for retrieval and processing, while real-time dialogue permits only 100 to 200 milliseconds of additional latency to preserve natural turn-taking.

To address these challenges, the authors decouple the backend memory items stored in MEM0 from a lightweight semantic index constructed above them. This index narrows retrieval to a compact and semantically coherent candidate set before accessing the underlying memories. The left-brain index adopts a two-level hierarchy comprising schemas for coarse-grained semantic routing and entities for locating concrete people, events, or concepts. Formally, the left brain is defined as:

$$\mathcal {G} ^ {L} = (\mathcal {S}, \mathcal {V}, \mathcal {E}), \qquad v = (d _ {v}, \mathcal {N} _ {v} ^ {\mathrm{micro}}, \mathcal {I} _ {v}), \qquad s = (d _ {s}, \mathcal {N} _ {s} ^ {\mathrm{macro}}, \mathcal {V} _ {s}).$$GL=(S,V,E),v=(dv​,Nvmicro​,Iv​),s=(ds​,Nsmacro​,Vs​).

Each entity $v \in \mathcal V$v∈V belongs to exactly one schema $s \in \mathcal {S}$s∈S, where $d _ { v }$dv​ and $d _ { s }$ds​ are textual descriptions, $\mathcal {I} _ { v }$Iv​ indexes the associated backend memory items, and $\mathcal {V} _ { s }$Vs​ contains the entities assigned to schema $s$s. The edge set $\mathcal {E} = \mathcal {E} _ {\mathrm{micro}} \cup \mathcal {E} _ {\mathrm{macro}}$E=Emicro​∪Emacro​ supports lightweight semantic expansion: $\mathcal {N} _ { v } ^ {\mathrm{micro}}$Nvmicro​ links related entities, while $\mathcal {N} _ { s } ^ {\mathrm{macro}}$Nsmacro​ links related schemas. The authors encode schema membership directly rather than introducing explicit schema-entity edges, thereby avoiding recursive traversal and keeping retrieval both compact and semantically focused.

During retrieval, a streaming matcher identifies relevant schemas and entities from the partial transcript while the user is speaking, then expands the matched entities and those contained in the matched schemas through strong and weak one-hop connections:

$$(\mathcal {V} _ {t}, \mathcal {S} _ {t}) = \mathrm{Match} (x _ {\leq t}, \mathcal {V}, \mathcal {S}), \quad \mathcal {Z} _ {t} = \mathcal {V} _ {t} \cup \mathcal {V} _ {\mathcal {S} _ {t}} \cup \mathcal {N} _ {1} ^ {\mathrm{strong}} (\mathcal {V} _ {t} \cup \mathcal {V} _ {\mathcal {S} _ {t}}) \cup \mathcal {N} _ {1} ^ {\mathrm{weak}} (\mathcal {V} _ {t} \cup \mathcal {V} _ {\mathcal {S} _ {t}}).$$(Vt​,St​)=Match(x≤t​,V,S),Zt​=Vt​∪VSt​​∪N1strong​(Vt​∪VSt​​)∪N1weak​(Vt​∪VSt​​).

Here, $\mathcal {V} _ { t }$Vt​ and $\mathcal {S} _ { t }$St​ denote the matched entities and schemas, respectively; $\mathcal {V} _ { \mathcal {S} _ { t } }$VSt​​ denotes the entities assigned to the matched schemas; and $\mathcal {Z} _ { t }$Zt​ is the expanded entity set obtained through one-hop strong and weak edges. The corresponding memory items are then indexed and searched as:

$$\mathcal {C} _ {t} ^ {L} = \bigcup_ {z \in \mathcal {Z} _ {t}} \mathcal {I} _ {z}, \qquad \mathcal {R} _ {t} ^ {L} = \mathrm{MemSearch} (q _ {t}, \mathcal {C} _ {t}; K).$$CtL​=z∈Zt​⋃​Iz​,RtL​=MemSearch(qt​,Ct​;K).

The backend searches only this candidate pool rather than the full memory $\mathcal {M}$M. By retaining memories associated with matched and neighboring entities while sharply reducing the search space, the proposed index enables accurate and efficient top-5 retrieval.

After each turn, an asynchronous updater extracts new facts and jointly reconciles them with nearby memories through ADD, UPDATE, DELETE, or KEEP operations. It then updates the associated schemas, entities, and relations off the critical path.

The authors further introduce a cluster emergence mechanism to handle growing memory clusters. As a cluster grows, its information density decreases, while frequent rule-based splitting may fragment related memories and reduce retrieval coverage. Instead, coherent subclusters emerge from repeated retrieval patterns. Let $\mathcal{Q}$Q be the queries observed within a session, $A _ { q }$Aq​ the entities activated by query $q$q, and $H$H a connected subset of entities in the current cluster. The query coherence is measured by:

$$\rho (H) = \frac {1}{| \mathcal {Q} |} \sum_ {q \in \mathcal {Q}} \frac {| A _ {q} \cap H |}{| A _ {q} \cup H |}.$$ρ(H)=∣Q∣1​q∈Q∑​∣Aq​∪H∣∣Aq​∩H∣​.

A high score indicates that the entities in $H$H are repeatedly retrieved together. If the largest qualifying subgraph exceeds threshold $\alpha$α, an LLM judge further evaluates its relevance, importance, and completeness before promoting it to a new cluster.

3.2 Right Brain: Knowing the Person

While the left brain records what happened, the right brain captures who the user is. Operating in parallel, it maintains persona memory: stable dispositions, affective tendencies, and attitudes grounded in specific people, events, or concepts. Such evidence spans multiple timescales, from immediate reactions within a turn to persistent properties consolidated across sessions.

The right brain maintains two complementary types of persona nodes:

$$\mathcal {G} ^ {R} = \left(\mathcal {V} ^ {I}, \mathcal {V} ^ {C}\right), \qquad v ^ {I} = \left(d _ {v} ^ {I}, \mathcal {I} _ {v} ^ {I}\right) \in \mathcal {V} ^ {I}, \qquad v _ {e} ^ {C} = \left(d _ {v, e} ^ {C}, \mathcal {I} _ {v, e} ^ {C}, \rho_ {v, e}\right) \in \mathcal {V} ^ {C}, \quad e \in \mathcal {V}.$$GR=(VI,VC),vI=(dvI​,IvI​)∈VI,veC​=(dv,eC​,Iv,eC​,ρv,e​)∈VC,e∈V.

Here, $d$d is a persona description and $\mathcal{I}$I indexes its supporting backend memory items. Independent nodes $v ^ { I }$vI encode user-intrinsic properties, including enduring dispositions, behavioral regularities, and affective tendencies supported by longitudinal evidence. Cross-entity nodes $v _ { e } ^ { C }$veC​ instead capture context-dependent affect, with $\rho _ { v, e }$ρv,e​ linking each node to a left-brain entity $e \in \mathcal { V }$e∈V. This distinction is fundamental: $v ^ { I }$vI explains persistent user characteristics, whereas $v _ { e } ^ { C }$veC​ preserves whom or what an emotion concerns. Collapsing the two would either mistake situational reactions for stable traits or remove the real-world causes that give affect its meaning.

A streaming matcher jointly activates independent and cross-entity persona nodes from the partial transcript. These matches are combined with cross-entity nodes linked to the entities activated by the left brain:

$$\begin{array}{c} (\mathcal {V} _ {t} ^ {I}, \mathcal {V} _ {t} ^ {C}) = \mathrm{Match} \left(x _ {\leq t}, \mathcal {V} ^ {I}, \mathcal {V} ^ {C}\right), \qquad \mathcal {Z} _ {t} ^ {R} = \mathcal {V} _ {t} ^ {I} \cup \mathcal {V} _ {t} ^ {C} \cup \left\{v _ {e} ^ {C} \in \mathcal {V} ^ {C}: e \in \mathcal {Z} _ {t} \right\}, \\ \mathcal {C} _ {t} ^ {R} = \bigcup_ {z \in \mathcal {Z} _ {t} ^ {R}} \mathcal {I} _ {z}, \qquad \mathcal {R} _ {t} ^ {R} = \mathrm{MemSearch} \left(q _ {t}, \mathcal {C} _ {t} ^ {R}; K\right). \end{array}$$(VtI​,VtC​)=Match(x≤t​,VI,VC),ZtR​=VtI​∪VtC​∪{veC​∈VC:e∈Zt​},CtR​=⋃z∈ZtR​​Iz​,RtR​=MemSearch(qt​,CtR​;K).​

Here, $\mathcal { Z } _ { t }$Zt​ is the expanded entity set produced by the left brain. The resulting candidate pool combines persona evidence directly implied by the conversation with attitudes grounded in the currently relevant real-world entities, without searching the full persona memory.

For short-horizon attribution, an affect estimator produces an emotion representation $e _ { t } = \phi ( x _ { t } )$et​=ϕ(xt​) for each conversational input $x _ { t }$xt​. The pair $( x _ { t }, e _ { t } )$(xt​,et​) provides immediate evidence for adding, editing, or merging persona nodes:

$$e _ {t} = \phi (x _ {t}), \qquad \mathcal {G} _ {t} ^ {R} = \text {Modify} \left(\mathcal {G} _ {t - 1} ^ {R}; x _ {t}, e _ {t}\right), \qquad t = 1, \dots, T.$$et​=ϕ(xt​),GtR​=Modify(Gt−1R​;xt​,et​),t=1,…,T.

Short-horizon attribution preserves the current affect together with its situational target and cause, allowing the persona graph to adapt within the ongoing interaction.

After each session, long-horizon attribution jointly analyzes the sequence $( x _ { 1 }, e _ { 1 } ), ( x _ { 2 }, e _ { 2 } ), \ldots, ( x _ { T }, e _ { T } )$(x1​,e1​),(x2​,e2​),…,(xT​,eT​) and consolidates recurrent evidence into stable independent persona nodes:

$$\mathcal {V} ^ {I} \leftarrow \text {Consolidate} \left(\mathcal {V} ^ {I}; (x _ {1}, e _ {1}), (x _ {2}, e _ {2}), \dots, (x _ {T}, e _ {T})\right).$$VI←Consolidate(VI;(x1​,e1​),(x2​,e2​),…,(xT​,eT​)).

Rather than accumulating every transient state, this process identifies persistent affective and behavioral patterns that provide a stable account of who the user is.

3.3 Streaming Dual-Brain Retrieval

The streaming retrieval process of VOICEMEM is the core of its near-zero added latency. Four stages are introduced: listening, speech tail, anticipation, and searching. During listening and speech tail (0 to 200 ms), while the user speaks, the system obtains the transcript $x _ { \le t }$x≤t​, the matched entities and schemas of both brains, and the speaker identity $p_t$pt​ in a streaming manner:

$$\underbrace {x _ {\leq t} = \operatorname{ASR} \left(a _ {\leq t}\right), \left(\mathcal {V} _ {t}, \mathcal {S} _ {t}\right), \left(\mathcal {V} _ {t} ^ {I}, \mathcal {V} _ {t} ^ {C}\right) = \operatorname{Match} \left(x _ {\leq t}\right)} _ {\text {in real time}} \mid \underbrace {p _ {t} = \psi \left(a _ {\leq t}\right)} _ {\text {(i) if delayed}}.$$in real timex≤t​=ASR(a≤t​),(Vt​,St​),(VtI​,VtC​)=Match(x≤t​)​​∣(i) if delayedpt​=ψ(a≤t​)​​.

In the anticipation stage (200 to 400 ms), the system assumes a reply is coming once the silence reaches 200 ms. It extracts the query embedding and expands the upper-layer graphs of both brains:

$$q _ {t} = \mathrm{Embed} (x _ {\leq t}), \qquad \mathcal {Z} _ {t} = \mathcal {G} ^ {L} (\mathcal {V} _ {t}, \mathcal {S} _ {t}), \qquad \mathcal {Z} _ {t} ^ {R} = \mathcal {G} ^ {R} (\mathcal {V} _ {t} ^ {I}, \mathcal {V} _ {t} ^ {C}, \mathcal {Z} _ {t}).$$qt​=Embed(x≤t​),Zt​=GL(Vt​,St​),ZtR​=GR(VtI​,VtC​,Zt​).

In the searching stage (400 to 500 ms), only the backend search remains. Both brains are searched and merged:

$$\mathcal {R} _ {t} ^ {L} = \operatorname{MemSearch} \left(q _ {t}, \mathcal {C} _ {t} ^ {L}; K\right), \quad \mathcal {R} _ {t} ^ {R} = \operatorname{MemSearch} \left(q _ {t}, \mathcal {C} _ {t} ^ {R}; K\right), \quad R _ {t} = \operatorname{Prompt} \left(\mathcal {R} _ {t} ^ {L}, \mathcal {R} _ {t} ^ {R}\right).$$RtL​=MemSearch(qt​,CtL​;K),RtR​=MemSearch(qt​,CtR​;K),Rt​=Prompt(RtL​,RtR​).

A VAD threshold is commonly set to 500 ms, leaving a 400 ms window before the reply must start. VOICEMEM meets this budget with margin, as the dense dual-brain retrieval itself costs only 134 ms.

3.4 Audio Memory Extension

VOICEMEM extends memory beyond text to audio, supporting multi-speaker discrimination, paralinguistic analysis, and environmental sound memory. When enabled, agents selectively retain audio as speaker voiceprints, acoustic embeddings, or raw waveforms, and attach them as multimodal nodes to the corresponding entity nodes.

4 Infrastructure: Model Training, Validation and Deployment

The authors provide implementation details of VOICEMEM in real-world settings, focusing on two aspects: model training, where they convert the Qwen2.5-Omni, Qwen3-Omni, and Step-Audio2-Mini model families, originally designed for speech-input/text-output, into memory-augmented speech language models through online black-box on-policy distillation, yielding the first speech language models with explicit memory access; and deployment, where they design an evolvable two-layer system architecture that decouples the memory layer from the underlying search engine.

4.1 Black-box OPD Training

To maximize model performance, the authors employ proprietary models as teachers and adopt online distillation to mitigate catastrophic forgetting during adaptation. Specifically, Qwen2.5-Omni and Qwen3-Omni are distilled from Qwen3.5-Omni, while Step-Audio2-Mini is distilled from Step-Audio2. The training pipeline consists of four stages.

Stage I constructs a coherent long-term memory world for each synthetic user. Starting from a core persona, the pipeline progressively instantiates background information, life events, messages, and online memories following the order Persona to Background to Events to Messages to Memory. This yields temporally consistent user histories for downstream dialogue generation.

Stage II performs SLM-verified online distillation. Memory-dependent conversations are generated through an iterative pipeline: Task and Topic sampling selects knowledge-, emotion-, or persona-oriented goals; Scene Initialization constructs the current context; Challenge and Strategy introduces recall, reference resolution, contradiction, and multi-memory reasoning; Contrastive Distillation compares responses with different memory-use behaviors; Verification filters for necessity, faithfulness, and quality; and SFT Update updates the generator for the next iteration. Repeating this loop yields CHATMEM-400K, a large-scale corpus of memory-dependent conversations.

Stage III involves human curation. Human annotators further refine the corpus and construct harder samples with richer emotional and personality dynamics and multimodal inputs such as acoustic questions. These examples augment training, while a challenging subset is curated into CHATMEM-BENCH.

Stage IV performs validation using CHATMEM-BENCH as one of the primary benchmarks for evaluating the final models, as existing benchmarks do not fully capture the requirements of real-world voice assistants. It covers four dimensions, Information, Persona, Affective Attribution, and Paralinguistics and Environment, across 14 fine-grained question categories.

4.2 Decoupled Upper-level Routing-Lower-level Engine Architecture

The underlying memory engine is evolving rapidly, with parameterized and latent memory mechanisms also emerging as promising directions. The authors therefore avoid tightly coupling the real-time memory architecture with any particular backend, which would otherwise hinder timely adoption of newer memory algorithms. To this end, they substantially redesigned VOICEMEM around a decoupled architecture: the focus lies on upper-level routing, the parallel dual-brain organization, and streaming reasoning, while the underlying engine abstracted by MemSearch remains fully interchangeable. In the current implementation, MEM0 is used as the backend due to its generality and strong state-of-the-art performance.

Experiment

VoiceMem is evaluated across text, persona, and long-horizon audio benchmarks against ten baseline systems, consistently outperforming them on most categories, particularly in temporal reasoning and acoustic-related tasks where text-only baselines fail. The system achieves high accuracy at a small retrieval budget (K=5), with latency staying flat as K increases due to schema routing, and ablations confirm that every component, especially the upper-layer index and the affective right brain, contributes meaningfully across datasets. The index also transfers across different memory backends, improving all three tested stores by 15.8 to 29.5 points without retuning.

VOICEMEM outperforms existing memory engines and full-context processing on factual memory benchmarks, leading in most information sub-categories and achieving the highest average score. Its gains are largest when multiple memories must be integrated, such as temporal reasoning, and smallest on tasks where a single recent memory suffices, like update tracking. VOICEMEM leads in seven of eleven information sub-categories and beats full-context processing by a large margin on average. The largest improvement over baselines occurs on temporal reasoning, while the smallest is on update tracking. Compared to its own storage backend, Mem0, VOICEMEM improves the average factual memory score by over 24 points.

VOICEMEM outperforms all baselines on persona memory benchmarks, achieving the highest average score. Its advantage is most pronounced on conflict detection and user modeling tasks, where full-context processing performs poorly. The method also shows consistent gains over memory engines like Mem0 and Zep across most persona sub-categories. VOICEMEM achieves the highest average persona memory score, surpassing the strongest baseline MemOS by a small margin. On ES-MemEval, VOICEMEM scores much higher than full-context on conflict detection and user modeling, while full-context scores are very low on those tasks. Compared to Mem0, VOICEMEM shows large improvements on several sub-categories, including information extraction, temporal reasoning, and conflict detection. The method also outperforms Zep and LangMem by wide margins on most persona sub-categories.

VoiceMem outperforms text-based memory systems on most ChatMem-Bench categories, with the largest gains in acoustic and paralinguistic recall where transcripts provide no evidence. Text baselines remain competitive on affective attribution because word choice carries much of the signal, while VoiceMem's advantage appears in attachment-guided decisions and affect-aware phrasing. VoiceMem leads on 11 of 14 categories, with the widest margin on Paralinguistics & Environment. Text systems score between 3.23 and 26.92 on acoustic categories, whereas VoiceMem reaches 45.16 to 53.84. Affective Attribution is more balanced, with text baselines staying competitive and VoiceMem improving specifically on attachment-guided decisions and affect-aware phrasing.

The proposed method consistently improves accuracy across all tested backends, with the largest gain on Mem0. The approach also achieves higher accuracy with far fewer memory tokens than the strongest baseline, and its performance remains stable across different retrieval budgets. Ablation studies show that every component contributes, with the upper-layer index being the most critical. Accuracy gains range from about 16 to 30 points across backends, with the largest improvement on Mem0. At the same retrieval budget, the method outperforms the strongest baseline by over 8 points while using roughly 4 times fewer memory tokens. Retrieval latency stays nearly constant as the budget increases, because the search space is limited before ranking. The method achieves its best accuracy at a modest budget; increasing the budget further yields only marginal gains. Removing the upper-layer index causes the largest accuracy drop, followed by the removal of the affective/persona component. Emergent clustering and dual-horizon updating matter most on datasets with longer sessions, and least on a shorter-session dataset.

VOICEMEM consistently outperforms existing memory engines and full-context processing across factual, persona, and acoustic benchmarks, with the largest gains on tasks requiring memory integration or paralinguistic cues, and smaller gains on single-recent-memory tasks like update tracking. It leads in most information sub-categories and shows strong improvements over Mem0, especially on conflict detection and temporal reasoning. The method also improves accuracy across all storage backends, most notably on Mem0, while using far fewer memory tokens and maintaining stable retrieval latency. Ablations show the upper-layer index is the most critical component, with emergent clustering and dual-horizon updating most beneficial on longer sessions.