One-sentence Summary

A survey by researchers from Jilin University, KAUST, et al. proposes a system-level framework that formalizes self-improving agents as adaptive systems coupling a foundation model with an operational scaffold, defines self-improvement as a self-induced update operator on model parameters or scaffold components, and organizes prior work by update target and signals while reviewing applications, evaluation, and future directions.

Key Contributions

• A system-level framework models a self-improving agent as a foundation model coupled with an operational scaffold (prompts, memory, tools, control logic) and formalizes self-improvement as a self-induced update operator acting on model parameters or scaffold components.
• Prior self-improvement methods are organized into two pathways: parametric foundation-model updates (a slow loop) and non-parametric scaffolding updates (a fast loop), categorized by the signals that drive change such as intrinsic generative demonstrations or extrinsic exploratory experience.
• Evaluation is reframed as a resource optimization problem; open challenges include multi-agent cooperative co-evolution and non-stationary simulators for open-world drift, and future directions point to safe, self-referential architectures that modify their own reasoning structures.

Introduction

The authors examine the rise of foundation-model-based agents, where large language and vision-language models enable a natural-language interface for reasoning and action, bringing the longstanding vision of machine self-improvement into practical reach. Earlier self-referential learning and meta-learning systems were limited by their need to search low-level code or synaptic spaces, preventing scalable open-ended improvement. Modern research has quickly produced a fragmented landscape in which closely related self-improvement techniques appear under inconsistent labels, and existing surveys often treat model fine-tuning and agent scaffolding as separate concerns without a unified historical or formal perspective. This survey bridges that gap by proposing a unified taxonomy that organizes self-improvement into two distinct pathways—permanent updates to the foundation model itself and faster, reversible modifications to the surrounding scaffold—tracing their conceptual roots from classical AI to current practice and providing a systematic roadmap for future self-improving agent systems.

Method

The authors formalize a foundation-model-based agent at time step $t$t as $\mathcal{A}_t = (\theta_t, \Sigma_t)$At​=(θt​,Σt​), where $\theta_t$θt​ represents the neural parameters of the foundation model and $\Sigma_t$Σt​ denotes the dynamic operational scaffold. The scaffold specifies how the model is conditioned, grounded, and connected to the external world, decomposing into structured prompts, memory mechanisms, external tools, and control logic. Self-improvement is conceptualized as a self-induced operator that updates this intrinsic configuration based on execution signals, bifurcating into two complementary paradigms: foundation model improvement and scaffolding improvement.

Foundation model improvement targets the parameter set $\theta_t$θt​ to internalize new behaviors and reasoning patterns. The agent acts as its own source of supervision, generating learning signals to update its weights through gradient-based optimization. The authors categorize this process based on the nature of the self-induced learning signal.

As illustrated in the figure above, the agent improves the foundation model through three distinct parameter-update loops. In the first loop, Intrinsic Generative Demonstrations, the agent synthesizes training instances such as instruction-response pairs or reasoning trajectories. A quality-control operator filters these candidate demonstrations before they are used for parameter updates. In the second loop, Intrinsic Evaluative Feedback, the agent or an auxiliary judge produces scores, preferences, or critiques from model outputs. This feedback is aggregated to guide optimization and alignment. In the third loop, Extrinsic Exploratory Experience, the agent interacts with real environments or world models to collect experience trajectories. These trajectories are processed to refine the policy via reinforcement learning or preference optimization.

Scaffolding improvement keeps the foundation model parameters fixed while modifying the operational scaffold $\Sigma_t$Σt​. This allows for fast, reversible, and task-specific adaptation without the risks of catastrophic forgetting. The authors analyze this paradigm by decomposing the scaffold into its core components.

Prompt optimization serves as a highly accessible form of scaffolding improvement, refining the agent's core behavioral prior. The authors categorize prompt refinement methods based on the form and richness of the learning signal.

As shown in the figure above, the refinement process operates as a loop where an initial prompt is executed and evaluated to produce candidate prompts and feedback. This yields four paradigms. Black-Box Optimization relies on scalar performance scores to navigate the discrete text space. Iterative Refinement uses qualitative natural-language critiques to provide interpretable revision guidance. Evolutionary Algorithms treat prompts as genes in a population, applying selection, crossover, and mutation operators to evolve better instructions. Textual Gradient Optimization formalizes the feedback as a structured directional message, analogous to a gradient in continuous optimization, which explicitly prescribes how the prompt should be revised.

Memory evolution equips the agent with an evolving internal cognitive resource, shifting from memoryless execution to cumulative learning. The authors decompose memory-based improvement into memory objects, memory structures, and memory processing.

As depicted in the figure above, the memory system comprises explicit objects like processed interaction trails and curated raw content, alongside implicit objects like latent embeddings. These objects are organized into flat, hierarchical, graph-based, or vector retrieval structures. These components culminate in a unified, signal-driven memory loop. The agent observes and detects saliency from new interactions, creates compact objects through self-driven processes, and organizes them via structured organization. It then reads on demand to plan and act, evaluates outcomes to assign credit and derive learning signals, and finally updates or deletes entries based on feedback to consolidate high-value knowledge and prune noise.

Beyond prompts and memory, the authors describe tool governance, where agents engage in dynamic tool routing, iterative tool refinement, and autonomous tool creation to expand their capability boundaries. At the deepest level of architectural intervention, full scaffolding improvement treats the agent's entire operational logic and codebase as a mutable substrate. In this regime, the improvement procedure is implemented within the current scaffolding, enabling recursive self-modification where candidate program patches are gated by verifiers before being committed to the agent's configuration.

Experiment

The surveyed experiments evaluate self-improving agents that iteratively update either their foundation model parameters or their scaffolding components—including prompts, memory, tools, and control logic—across domains such as software engineering, web navigation, games, scientific discovery, embodied AI, and general computer control. Scaffolding-based improvements tend to be fast and modular but risk overfitting to specific interfaces, while parameter-level improvements internalize skills more durably but are expensive and susceptible to reward hacking and evaluation artifacts. A recurring qualitative conclusion is that rigorous evaluation must track performance trajectories under fixed resource budgets, measure transfer to held-out tasks, and monitor regressions and safety violations over time, rather than relying on single peak scores, to distinguish genuine capability gains from exploitation of benchmark feedback.

This survey uniquely combines a comprehensive organizational framework with a detailed focus on agent update mechanisms, distinguishing it from prior works that emphasize evaluation or signals but lack coverage of historical roots or definitional scope. The analysis of evaluation methodologies across domains reveals a shared imperative to move beyond static scoring toward protocols that track learning trajectories, transfer, overhead, and safety under realistic interactive constraints. This survey is the only one to give primary emphasis to all seven organizing dimensions, including historical roots and update substrate, which are secondary or absent in other surveys. Domain evaluation protocols consistently warn against overfitting, with web and game benchmarks advocating controlled or randomized environments and held-out strategies to separate generalization from memorization. Embodied AI and general computer control evaluations uniquely require tracking safety constraint violations and error-recovery behavior, reflecting physical safety limits and fragile user interfaces. Scientific discovery evaluation prioritizes execution-based checks and reporting of wall-clock time and operational cost alongside artifact quality and reproducibility. The overarching takeaway calls for reporting learning trajectories under fixed budgets, transfer beyond the training signal, overhead costs, and regression indicators over time.

Prompt optimization paradigms range from scalar feedback to textual gradients, with each step adding richer structure and more automated, interpretable refinement. Scalar methods are simple but inefficient, critique-based approaches enable targeted edits but risk noise, population evolution explores strongly yet is compute-heavy, and textual gradients offer efficient directional updates whose reliability depends on the underlying model. Scalar-feedback optimization is model-agnostic and easy to deploy, but it is sample-inefficient and offers low interpretability. Qualitative-feedback refinement provides interpretable, targeted corrections, though critiques can be noisy, drift, and require a strong validator. Population-based evolution strongly explores and escapes local optima, but it is computationally heavy, with fitness functions tuned to specific domains. Textual-gradient optimization delivers sample-efficient, directional updates with high automation, yet the gradients can be brittle and vary in quality across different large language models.

Processed interaction trails and curated raw content represent two distinct memory object strategies with opposing trade-offs: processed trails offer high compactness and interpretability but risk summary bias and stale heuristics, while curated raw content preserves high fidelity and auditability at the cost of context bloat and retrieval noise. Integrated external knowledge and latent embeddings provide alternative profiles, with external knowledge excelling in shared grounding and latent embeddings enabling fast associative recall but introducing risks of silent corruption and hard-to-debug retrieval. Processed trails achieve the highest ratings for compactness and interpretability among all object types, yet their most common failure modes include summary bias, stale heuristics, and weak credit assignment. Curated raw content provides the strongest fidelity and auditability, but suffers from context bloat, retrieval noise, and privacy leakage as primary failure modes. Integrated external knowledge shares the top fidelity rating with curated raw content and adds strong grounding, but is uniquely vulnerable to grounding failures, staleness, and tool brittleness. Latent embeddings receive the lowest interpretability and auditability ratings, making their retrieval hard to debug and susceptible to drift, contamination, and silent corruption.

All surveyed agent memory systems store explicit objects and provide full create, read, update, and delete operations as primary mechanisms. Structural choices range from flat lists in earlier works to hierarchical or graph-based layouts plus vector retrieval in more recent designs, while every system treats memory selection and maintenance as primary governance functions. Every system relies on explicit memory objects and treats create, read, update, and delete as primary operations. Earlier systems adopt flat memory structures, whereas SAGE uses a hierarchical layout and Mem0 employs a graph with vector retrieval.

Different application domains shape self-improvement loops through characteristic sandboxes and learning signals, which in turn determine the main bottlenecks, the target of improvement (scaffolding versus model parameters), and the iteration mode. For instance, deterministic and reversible environments like software engineering support online debugging per issue, while noisy, expensive evaluation in scientific discovery leads to propose-run-critique-revise cycles. Across domains, improvements often focus on scaffolding components such as planning and tool orchestration, except in gaming and embodied AI where model parameters are directly updated via self-play or data flywheels. Domains with low-cost, reversible resets (software engineering, games) enable rapid online iteration, whereas scientific discovery and embodied AI rely on mixed offline cycles due to expensive or noisy evaluation. Improvement targets are predominantly scaffolding-based (e.g., perception-action grounding in web, tool orchestration in science), but gaming and embodied settings directly update policy and model parameters through self-play or data collection and retraining.

The survey evaluates agent update mechanisms by analyzing evaluation methodologies, prompt optimization paradigms, memory object strategies, system architectures, and self-improvement loops across domains. It finds that evaluation must move beyond static scoring to track learning trajectories, transfer, overhead, and safety, with domain-specific concerns such as overfitting in web and game benchmarks and safety violations in embodied AI. Prompt optimization methods trade off simplicity, efficiency, and interpretability, while memory objects balance compactness against fidelity and auditability. Self-improvement loops adapt to domain characteristics, enabling online scaffolding updates in deterministic environments and offline model updates in noisy settings.