One-sentence Summary

A team from MemTensor, University of Science and Technology of China, and collaborating institutions propose MSCE, a training-free Memory–Skill Co-Evolution framework that converts agent experience into evidence-grounded, callable skills and uses reflection-weighted value backfilling to govern memory and skill evolution, significantly outperforming baselines on EvoAgentBench and LoCoMo while exhibiting strong cross-domain transferability.

Key Contributions

• MSCE is a training-free framework that organizes agent experience into a three-level memory hierarchy of grounded traces, reusable policies, and environmental cognition, and that promotes evidence-backed policies with positive gain into callable skills capturing applicability boundaries, decision guidance, verification rules, and reliability estimates.
• The framework introduces reflection-weighted value backfilling, which propagates sparse terminal feedback through dense local self-reflections to produce evidence-calibrated trace values for governing memory retention and skill evolution.
• Experiments on EvoAgentBench and LoCoMo show that MSCE significantly outperforms state-of-the-art skill-augmented and memory-driven baselines, while exhibiting strong cross-domain transferability and lifelong-evolution capabilities.

Introduction

Large language model agents are increasingly tasked with long-horizon, multi-step interactions that require persistent memory to maintain coherence, personalize behavior, and reuse experience across sessions. Prior memory systems typically store factual logs or experiential summaries, but they treat these records as passive context that the agent must repeatedly reason over, leading to token bloat and missed opportunities for automation. While skills offer a more executable form of reusable knowledge, distilling them directly from raw trajectories is unreliable because traces contain failed attempts, sparse feedback, and environment-specific noise, causing brittle overfitting. The authors propose MSCE, a training-free memory–skill co-evolution framework that introduces a governed three-level memory hierarchy (trace, policy, and environmental cognition) and a reflection-weighted value backfilling mechanism. This design converts noisy interaction histories into verifiable abstractions and promotes only evidence-backed, stable procedural patterns into deployable skills, bridging the gap between passive retrieval and active skill execution.

Method

The authors propose MSCE, a framework that maintains a governed memory hierarchy and a downstream skill library to enable traceable, evidence-backed skill acquisition and invocation.

Refer to the framework diagram:

As shown in the figure below, the system maintains a memory hierarchy $\mathcal{M} = (\mathcal{M}^{(1)}, \mathcal{M}^{(2)}, \mathcal{M}^{(3)})$M=(M(1),M(2),M(3)) where L1 traces preserve auditable evidence, L2 policies abstract revisable procedures, and L3 environmental cognition stores declarative environment knowledge. The skill library $\mathcal{K}$K sits downstream of this memory hierarchy, exposing verified procedures for invocation. This separation keeps skills traceable to evidence while preventing traces, policies, environmental facts, and callable actions from being merged into a single uncontrolled pool.

Three-level Governed Memory L1 Trace Memory stores grounded decision units collected during interaction:

$$f_{i, t}^{(1)} = (s_{i, t}, a_{i, t}, o_{i, t}, \rho_{i, t}, V(f_{i, t}^{(1)}))$$fi,t(1)​=(si,t​,ai,t​,oi,t​,ρi,t​,V(fi,t(1)​))

where $s_{i, t}$si,t​, $a_{i, t}$ai,t​, $o_{i, t}$oi,t​, and $\rho_{i, t}$ρi,t​ denote the semantic state, action, observation, and self-reflection at step $t$t. The trace value $V(f_{i, t}^{(1)})$V(fi,t(1)​) is initialized as unavailable and filled after terminal feedback arrives. L1 serves as the evidence layer; higher-level memories and skills must maintain links to their supporting traces. To bound storage and reduce privacy risk, MSCE stores normalized evidence rather than unbounded raw observations.

L2 Policy Memory stores reusable policies induced from recurring trace evidence:

$$f^{(2)} = (\phi, \pi, \kappa, \mathcal{B}, \{f^{(1)}\})$$f(2)=(ϕ,π,κ,B,{f(1)})

where $\phi$ϕ is the trigger condition, $\pi$π is a natural-language procedure, $\kappa$κ specifies verification or fallback criteria, $\mathcal{B}$B records applicability boundaries, and $\{f^{(1)}\}${f(1)} is the supporting L1 evidence set. Policy extraction occurs when a valued L1 trace satisfies $V(f_{i, t}^{(1)}) \geq v_{\min}$V(fi,t(1)​)≥vmin​. Eligible traces are matched against existing L2 policies using embedding similarity and structured evidence. If no policy matches, the trace enters a candidate pool. A new L2 policy is induced only when the same signature bucket contains evidence from at least $n_{\min}$nmin​ distinct episodes. For each touched policy, MSCE computes a heuristic utility gain:

$$G(f^{(2)}) = \bar{V}_{\mathrm{with}} - \bar{V}_{\mathrm{blend}}(S_{\mathrm{without}})$$G(f(2))=Vˉwith​−Vˉblend​(Swithout​)

This gain determines whether a policy remains active, becomes eligible for skill crystallization, or is retired.

L3 Environmental Cognition Memory compresses environment-level cognition as:

$$f^{(3)} = (\mathcal{E}, \mathcal{I}, \mathcal{C}, \{f^{(2)}\})$$f(3)=(E,I,C,{f(2)})

where $\mathcal{E}$E, $\mathcal{I}$I, and $\mathcal{C}$C are entity or structure facts, action response regularities, and environmental constraints, each linked to supporting L2 policies. Unlike L2 policies, L3 describes how a particular environment is organized. Abstraction is triggered when multiple active policies share a domain context, extracting declarative facts while excluding imperative procedural instructions.

Skill Crystallization and Invocation A skill is a callable object derived from an eligible L2 policy:

$$k = (\phi, \pi, \kappa, \mathcal{B}, \mathcal{A}, \mathcal{D}, \eta)$$k=(ϕ,π,κ,B,A,D,η)

It inherits the trigger, procedure, verification rule, and boundary from L2, adding evidence anchors $\mathcal{A}$A, decision guidance $\mathcal{D}$D, and reliability $\eta$η. Skill promotion requires two gates: the source L2 policy must satisfy the positive gain condition $G(f^{(2)}) > \theta_G$G(f(2))>θG​, and it must be stable. The crystallization prompt conditions on the policy, evidence anchors, a whitelist of tools, and decision guidance seeds. A deterministic verifier checks the schema, enforces evidence grounding, and runs lightweight coverage tests. Drafts failing any check are discarded.

After deployment, MSCE estimates skill reliability with a smoothed success rate:

$$\eta = \frac{n_{\mathrm{pass}} + 1}{n_{\mathrm{trial}} + 2}$$η=ntrial​+2npass​+1​

High reliability skills enter active retrieval. During inference, if the context matches an L3 cognition, retrieved L3 facts provide environmental priors that instantiate parameters and interpret applicability boundaries without overwriting the skill procedure.

Online Update via Dual-Signal Value Backfilling MSCE updates the external cognitive state $\mathcal{C}_i = (\mathcal{M}^{(1)}, \mathcal{M}^{(2)}, \mathcal{M}^{(3)}, \mathcal{K})_i$Ci​=(M(1),M(2),M(3),K)i​ through dense step level reflections and sparse terminal feedback. To distribute the sparse terminal feedback $R_i$Ri​ to each step trace, the terminal step inherits this value: $V(f_{i, H_i}^{(1)}) = R_i$V(fi,Hi​(1)​)=Ri​. For earlier steps, MSCE applies reflection weighted value backfilling:

$$V(f_{i, t}^{(1)}) = \alpha_{i, t} R_i + (1 - \alpha_{i, t}) \gamma V(f_{i, t+1}^{(1)})$$V(fi,t(1)​)=αi,t​Ri​+(1−αi,t​)γV(fi,t+1(1)​)

where $\gamma \in [0, 1)$γ∈[0,1) is a discount factor and $\alpha_{i, t}$αi,t​ is a reflection weight. High $\alpha$α steps provide locally informative evidence for the final outcome. The weight $\alpha$α is estimated based on the self reflection $\rho_{i, t}$ρi,t​ and its local context. After an episode closes, MSCE backfills step values, associates or induces L2 policies, triggers L3 abstraction, and crystallizes eligible policies into skills.

Experiment

The evaluation uses EvoAgentBench for multi-domain skill self-evolution and LoCoMo for long-term dialogue memory, comparing MSCE against memory-based and trajectory-to-skill baselines under identical tool access and interaction budgets. MSCE achieves the best Pass@1 across all EvoAgentBench domains while reducing inference cost in most settings, and it shows transferable cross-domain gains and monotonic lifelong improvement, demonstrating effective "learning by using." Ablation results confirm that hierarchical memory, skill crystallization, value-calibrated applicability filtering, and reflection-weighted learning jointly drive these performance and efficiency improvements.

MSCE achieves the best or tied-best Pass@1 across all five EvoAgentBench domains, outperforming the strongest non-MSCE baselines by substantial margins, most notably a 15.39-point gain in Software Engineering. On Code, it matches the top baseline accuracy while cutting cost nearly in half, and cost remains competitive or improves in other domains, confirming that gains are not bought with excessive inference budgets. MSCE obtains the highest or tied Pass@1 in every domain, with a 15.39-point lead over the best non-MSCE baseline on Software Engineering. On Code tasks, MSCE ties the best baseline at 61.54% Pass@1 yet reduces cost from 3.9 to 2.0 turns. Cost is lowered on Information Retrieval, Math, Code, and Knowledge Work compared to the strongest non-MSCE baselines, with especially large reductions on Math and Code.

MSCE achieves the best overall judge score and F1 on LoCoMo, outperforming the strongest baseline SkillFlow-Evolve by 2.01 and 1.18 points respectively. It also obtains the highest scores on single-hop, multi-hop, and temporal reasoning questions, while EverOS performs best on open-domain questions when excluding Vanilla Agent. The hierarchical memory design helps preserve factual evidence and abstract long-range dependencies. MSCE outperforms SkillFlow-Evolve by 2.01 points in overall judge score and 1.18 points in F1. MSCE leads on single-hop, multi-hop, and temporal reasoning; EverOS is best on open-domain questions (excluding Vanilla Agent).

Removing any component of the full MSCE method reduces accuracy or efficiency, confirming that hierarchical memory, skill crystallization, and applicability filtering all contribute to performance. The flat memory variant suffers the largest drop in Pass@1 and incurs the highest cost on code, while disabling skill crystallization degrades Pass@1 and increases cost in every domain. Each ablation shows that effective reuse depends on jointly organizing, selecting, and refining skills rather than simply retrieving past experiences. Flat memory reduces Pass@1 by over 15 points on information retrieval, mathematical reasoning, and software engineering, and nearly triples the cost on code implementation from 2.0 to 5.3 turns. Removing skill crystallization causes the most consistent component-level degradation, lowering Pass@1 by 6.15 to 11.54 points across all five domains and raising cost in every domain. Disabling value calibration lowers Pass@1 and increases cost in every domain, demonstrating that applicability-aware filtering prevents harmful skill injection. All ablations underperform the full MSCE, with hierarchical abstraction, skill crystallization, and calibrated selection each essential for maintaining both accuracy and efficiency.

MSCE is evaluated on the five-domain EvoAgentBench and the LoCoMo long-context reasoning benchmark, with an ablation study dissecting its components. It achieves the best Pass@1 across all EvoAgentBench domains, most notably a 15-point gain in Software Engineering, while simultaneously reducing inference cost on Code and other tasks, and it leads LoCoMo overall judge score and F1, excelling on single-hop, multi-hop, and temporal reasoning thanks to hierarchical memory for long-range dependencies. Removing any component (hierarchical memory, skill crystallization, or applicability filtering) markedly degrades both accuracy and efficiency, confirming that their joint design is essential for effective and cost-efficient skill reuse.