One-sentence Summary

Mila, Université de Montréal, and Google DeepMind study whether self-evolving large language model agents are effective continual learners by evaluating ReasoningBank, GEPA, and Meta-Harness on four drifting task streams built from BIRD, BFCL, and ScienceWorld, finding that global self-modification improves plasticity but harms stability while similarity-based memory retrieval suffers retrieval collisions across shifting topics, workflows, and domains.

Key Contributions

• This paper frames nonparametric self-evolving LLM agents as continual learners by keeping model weights frozen and updating harness components, such as prompts, memory, and skills, from task feedback.

• This work formalizes harness adaptation over sequential task streams and uses a policy-sequence objective plus adapted plasticity and stability metrics to compare experience retrieval and self-modification methods such as ReasoningBank, GEPA, and Meta-Harness.

• Experiments on four drifting streams built from BIRD, BFCL, and ScienceWorld show that these methods improve aggregate performance over the seed harness in most settings, while exposing a plasticity-stability trade-off through interference from global prompt edits, overwriting of earlier knowledge, foreign memory injection, and retrieval collision.

Introduction

Large language model agents are increasingly used to operate software tools, write code, query databases, and act in simulated environments. This matters because real-world use involves new tasks, tools, and workflows after model weights are frozen, so agents need to keep learning during deployment without forgetting earlier skills. The authors leverage nonparametric self-evolving agents, which keep the model weights frozen and instead update the agent harness, including prompts, memory, skills, and control code.

Prior work shows that self-evolution can improve agent performance on single-task or single-domain benchmarks. However, it remained unclear whether this approach supports continual learning over heterogeneous task streams. A shared harness may become biased toward recent or task-specific shortcuts, and updates for one task can interfere with later tasks. Existing evaluations also do not directly test continual learning under topic drift, workflow drift, domain drift, and combined drift.

The main contribution is a framework for evaluating self-evolving LLM agents as continual learners. The authors formalize the evolver that updates the harness from experience, construct four drifting task streams from BIRD, BFCL, and ScienceWorld, and test ReasoningBank, GEPA, and Meta-Harness. They find that current methods face a tradeoff between plasticity and stability: shared self-modification can cause cross-task interference, while similarity-based retrieval does not fully prevent foreign memory injection or retrieval collisions.

Dataset

1. The authors construct task streams from three agentic benchmarks: BIRD for text-to-SQL querying over relational databases, BFCL for multi-turn user requests served by calling stateful APIs, and ScienceWorld for interactive science experiments in a text environment.

2. Each stream is organized into consecutive blocks of 50 training tasks per group. After training, each group is evaluated on a disjoint holdout set of 50 tasks, except for ScienceWorld, which uses a holdout set of 90 tasks.

3. For topic drift, the authors use three BIRD database groups in sequence: BIRD-F1 based on the formula_1 database, BIRD-Code based on codebase_community, and BIRD-TP based on thrombosis_prediction. The text-to-SQL task remains fixed, but each database introduces a different schema and local conventions.

4. For workflow drift, the authors use BFCL-Miss followed by BFCL-Base. Both groups share the same API environment, but BFCL-Miss requires the agent to clarify missing arguments, while BFCL-Base provides complete arguments and penalizes unnecessary clarification.

5. For domain drift, the authors use ScienceWorld, BFCL-Base, and BIRD-F1 in sequence. This stream changes the environment, observation space, and action space across interactive science experiments, multi-turn tool use, and database querying.

6. For compound drift, the authors combine multiple shift types over a longer sequence: BIRD-Code, BIRD-F1, BFCL-Miss, and BFCL-Base. This tests both cross-domain transitions and within-domain changes, including topic, domain, and workflow shifts.

7. The paper uses the data as online task streams rather than standard train-validation-test partitions. The agent trains on groups in sequence, receives binary task-success feedback after each training task, and is evaluated on disjoint holdout sets from every group represented in the stream.

8. Evaluation checkpoints occur every 50 stream positions. At each checkpoint, the frozen system is evaluated on the holdout sets of all K groups, and each evaluation averages eight rollouts per instance.

9. The dataset setup mainly relies on grouping, sequencing, and disjoint holdout construction. The paper does not describe additional filtering, cropping, or metadata processing steps beyond benchmark selection, task block formation, group-specific holdout assignment, and stream sequencing.

Method

The authors structure the methodology around two interacting components: the evolver $f$f and the task stream weight schedule $w$w. The evolver represents the adaptive mechanism being studied, while the weight schedule determines how the task stream is weighted when measuring performance. Together, these components define the evaluation objective for the system.

The key evaluation criterion is the improvement of the evolver over a seed harness. Rather than assessing the evolver in isolation, the authors measure how much it increases the weighted task-stream objective $\bar{J}_{w}(f)$Jˉw​(f) relative to the baseline harness. This design places emphasis on the interaction between the evolver and the task distribution weighting, allowing the method to be judged by its practical effect on the weighted performance objective.

Experiment

Experiments instantiate task-free drifting streams from text-to-SQL, API-calling, and science-environment benchmarks, evaluating a frozen evolving harness at checkpoint intervals across topic, workflow, domain, and compound drift to measure whether self-evolving methods improve performance while balancing plasticity and stability. Across methods, self-evolution generally yields net gains, but higher plasticity on active tasks consistently comes with greater cross-task interference and forgetting, especially when updates overwrite shared prompts or workspaces. Retrieval-based memory improves stability when domains are clearly separated, yet similarity alone fails to distinguish conflicting workflows or closely related tasks, causing pre-block interference and post-block displacement. Overall, current self-evolving agents struggle to integrate new capabilities without erasing or misapplying earlier ones under continual drift.

Across drift streams, self-evolution methods generally improve terminal and average performance, with the clearest gains on workflow, domain, and compound drift. Topic drift is the exception, producing near-zero or slightly negative net change for the methods tested. The results show that stronger learning on the active task block usually comes with weaker preservation of earlier tasks. GEPA produces the strongest active-block gains but the largest cross-block losses because updates rewrite a shared prompt. ReasoningBank avoids post-block forgetting when domains are separable through localized retrieval, but it still causes pre-block interference by injecting foreign memories. Higher within-block plasticity consistently corresponds to lower cross-block stability across the four drift scenarios.

The experiments compare self-evolution methods under multiple drift streams, showing that they generally improve terminal and average performance, especially under workflow, domain, and compound drift, while topic drift yields little or negative gain. They also reveal a plasticity-stability tradeoff: methods with higher within-block plasticity, such as GEPA, learn active tasks better but suffer larger cross-block losses from shared prompt rewriting, whereas ReasoningBank reduces post-block forgetting through localized retrieval yet introduces pre-block interference. Overall, the results indicate that stronger adaptation to the current task often comes at the cost of preserving earlier tasks.