One-sentence Summary

Researchers from Tianjin University and Shanxi University propose Generalized Agent Iteration (GAI), a formal framework that unifies iterative policy improvement and recursive self-improvement through two dials—whether the improving mechanism is part of the agent and whether its evaluation standard is external—thereby enabling the placement and defect analysis of existing systems along these axes.

Key Contributions

• Introduces Generalized Agent Iteration (GAI), a formal framework that unifies iterative policy improvement and recursive self-improvement as two cases of a single learning paradigm, defined by a cycle of agent evaluation and agent improvement.
• GAI specifies two dials for distinguishing system instances: whether the improving mechanism lies inside the agent (delineating GPI from RSI) and whether the evaluation standard is grounded outside it (classifying systems as anchored, goal drift, or fully self-referential), which enables placing existing systems on the same two axes.
• The framework provides a defect catalogue that states, one condition at a time, where recursive self-improvement departs from classical guarantees, with each defect marked structural and observed in existing systems where applicable, and identifies three open targets for formal theorems: characterizing the solution set of self-consistency conditions, formalizing the reduction from RSI to GPI as a proposition, and deciding whether an internal monitor could verify its own modification critic.

Introduction

The authors address a long-standing gap in artificial intelligence: the absence of a unified formal account for self-improving systems. While classical iterative policy improvement has a rigorous foundation in generalized policy iteration (GPI), which assumes both the improvement mechanism and the reward lie outside the agent, modern recursive self-improvement (RSI) systems violate this setup by allowing the agent to modify its own improvement mechanism or evaluator. Prior work, including Gödel machines and empirical agentic loops, lacks precise criteria for what constitutes self-improvement and how failures arise when classical assumptions are broken.

To fill this gap, the authors propose Generalized Agent Iteration (GAI), a formal framework that treats iterative policy improvement and recursive self-improvement as two settings of a single alternating learning cycle. The framework centers on two dials: whether the improving mechanism is part of the agent itself, and whether the evaluation standard remains grounded outside the agent. By positioning existing systems, from Gödel machines to co-evolving evaluators, on these axes, the authors provide a uniform reading of diverse approaches. They also identify four structural defects of recursive self-improvement, each tied to a condition of classical GPI that a self-referential system violates, and outline open questions for future formal characterization.

Method

3 The Generalized Agent Iteration (GAI) Framework

The authors propose Generalized Agent Iteration (GAI) as a formal learning paradigm that extends the classical Generalized Policy Iteration (GPI) framework to describe agent-based systems whose improving components can themselves be edited. While GPI captures a broad class of policy-improvement methods, it fails to account for systems that improve recursively, where the mechanism responsible for improving the agent is itself part of what the agent can modify. GAI retains the alternating cycle of evaluation and improvement at its core, and treats both GPI and recursive self-improvement as specific instances of the same underlying structure. The two instances differ along two dimensions, which the authors call the two dials of the framework: (1) whether the mechanism that improves the agent lies inside the agent, and (2) whether the standard against which improvement is measured stays grounded in what lies outside the agent.

3.1 The Self-Improving System and Agent

A self-improving system acts in a world and is measured against a goal. The authors keep the world as a Markov decision process $\mathcal{W} = (\mathcal{S}, \mathcal{A}, p, r)$W=(S,A,p,r), and write $G$G for an external goal that the system is expected to serve. Both the world and the goal belong to the environment and are not part of the system itself. The objective of the system is to serve the external goal $G$G, whose scalar instance in the world is the reward $r$r. The configuration that serves it best is

which is fixed by the world and the goal alone, not by the system itself. The authors note that some self-improving systems allow the agent to propose its own tasks or goals as a means of learning; they treat this as a particular form of self-improvement, since such self-set tasks remain subordinate to an external goal that is objective and cannot be edited by the agent.

Let $\chi$χ be a system made of components, with $\mathcal{O} = \{\pi, V, m, U, \rho\}$O={π,V,m,U,ρ} as the set of components and $\mathcal{C}_o$Co​ as the content space of component $o$o. The system then lies in the space $\chi \in X = \prod_{o \in \mathcal{O}} \mathcal{C}_o$χ∈X=∏o∈O​Co​. Let $\mathrm{Ag} \subseteq \mathcal{O}$Ag⊆O be the set of components that form the agent; an agent instance is an assignment $\xi \in X_{\mathrm{Ag}} = \prod_{o \in \mathrm{Ag}} \mathcal{C}_o$ξ∈XAg​=∏o∈Ag​Co​ of contents to these components. Moreover, $\chi[\xi]$χ[ξ] denotes the system obtained by replacing the agent part of $\chi$χ with $\xi$ξ. The authors use $\Delta(Y)$Δ(Y) to denote the set of probability distributions over a space $Y$Y.

Over an external world $\mathcal{W}$W and a goal $G$G, a system is a configuration $\chi = (\pi, V, m, U, \rho)$χ=(π,V,m,U,ρ), a tuple of the current contents of five components: $\pi: \mathcal{S} \to \Delta(\mathcal{A})$π:S→Δ(A) is the policy that chooses world actions; $V: \mathcal{S} \to \mathbb{R}$V:S→R is the action critic that evaluates how well $\pi$π acts; $m: X \to \Delta(X_{\mathrm{Ag}})$m:X→Δ(XAg​) is the modifier that proposes how the system should change; $U: X \to \mathbb{R}$U:X→R is the modification critic that evaluates whether a proposed change serves $G$G; and $\rho$ρ is the evaluation base, the standard that the critics measure against. All five roles belong to a system; an instance may leave some of them unused, as the modification critic $U$U is when the modifier is fixed. The agent is the modifiable parts inside the system. Its membership varies across systems, such as $\mathrm{Ag} = \{\pi, V\}$Ag={π,V} when the modifier is fixed, or $\mathrm{Ag} = \{\pi, V, m, U, \rho\}$Ag={π,V,m,U,ρ} when the system may also rewrite its modifier and even the evaluation standard.

A generalized agent iteration is a process that alternates two operations on the agent part of a system. Agent evaluation scores the agent part against the evaluation base $\rho$ρ, with $V$V at the policy level and $U$U at the agent level. Agent improvement produces a new agent instance $\xi \sim m(\cdot|\chi)$ξ∼m(⋅∣χ), which the system adopts as $\chi \leftarrow \chi[\xi]$χ←χ[ξ]. This definition leaves two choices open: whether the mechanism that improves the agent is among these components, and what the evaluation base points at. These are the two dials of the framework, and each formalizes one way in which GPI and recursive self-improvement differ.

The first dial concerns whether the mechanism that improves the agent lies inside the agent. The agent changes only through the modifier: at each step $m$m draws a new agent instance $\xi \sim m(\cdot|\chi)$ξ∼m(⋅∣χ), and the system moves to $\chi[\xi]$χ[ξ]. If $m \notin \mathrm{Ag}$m∈/Ag, its content is fixed and $m$m acts as an external improvement mechanism, the abstract evaluation-and-improvement operator that GPI iterates. If $m \in \mathrm{Ag}$m∈Ag, the system may rewrite its own modifier, so recursion closes at $m$m with no external meta-layer.

The second dial concerns whether the standard that improvement is measured against stays grounded. Each evaluation regresses toward a standard, its evaluation base; the authors treat this base as a single standard $\rho$ρ and write its instances $\rho_V$ρV​ and $\rho_U$ρU​ only where the two critics must be told apart. The base is a component of the system, and what the second dial asks about is where its content comes from and whether the agent may rewrite it. An evaluation is grounded when its base draws its content from outside the system, the world and the goal, and when the base itself lies outside the agent. The agent then cannot rewrite the standard it is measured against. A base that draws its content from outside but that the agent may rewrite can move under the system's own edits; a base with no external content at all constrains the loop by nothing beyond self-consistency.

3.2 GPI and Recursive Self-Improvement as Two GAI Instances

The authors re-introduce GPI and recursive self-improvement as two instances of the system definition above. They differ only in how the two dials are set: for GPI the modifier lies outside the agent and the evaluation base is grounded in the world reward; for recursive self-improvement the modifier is itself an agent component. Everything else in the description is shared.

GPI as the anchored instance. The agent consists of the policy and the action critic, $\mathrm{Ag} = \{\pi, V\}$Ag={π,V}, and the modifier is not part of the agent, $m \notin \mathrm{Ag}$m∈/Ag. The content of $m$m is fixed to an evaluation operator $\mathcal{E}_m$Em​ and an improvement operator $\mathcal{I}_m$Im​, where the subscript $m$m marks that the update principle is the content of $m$m. Since $m$m is fixed, both operators are fixed as well. Through $m$m the agent changes componentwise as

while $m$m itself is unchanged. The only evaluation in use is the action critic; the modification critic $U$U is present but not exercised, since $m$m is fixed and no alternative modification is scored. The base $\rho_V$ρV​ of the action critic is grounded, with content the world reward $r$r. When $\mathcal{E}_m$Em​ is policy evaluation and $\mathcal{I}_m$Im​ is greedy, the two updates form exactly policy or value iteration, which in a finite MDP converges to an optimal policy $\pi^{\star}$π⋆ under standard assumptions. The objective of Equation (1) is then maximized by that same policy, with the action critic at its value $V^{\pi^{\star}}$Vπ⋆.

Recursive self-improvement as the self-modifying instance. Now the modifier is part of the agent, $m \in \mathrm{Ag}$m∈Ag, canonically $\mathrm{Ag} = \{\pi, V, m, U\}$Ag={π,V,m,U}. The step is unchanged and now reads

The operators in the first line are subscripted by $m$m to mark that they are decided by the current modifier rather than fixed externally: the policy-level updates of GPI become, in recursive self-improvement, content that $m$m may realize. The second line is the recursion itself: $(\xi)_m$(ξ)m​ and $(\xi)_U$(ξ)U​ are the modifier and modification-critic slots of the proposal, so the next modifier and the next critic are produced by the current one. Recursion closes at $m$m, and no external meta-layer updates it. Every agent component changes only through $m$m, and $m$m may run the GPI-style alternation above, but it may equally rewrite itself, replace its critic, or change the rule by which it chooses.

Nothing in this description guarantees that $m$m improves toward anything external; improving is a property of $m$m's content, not of the framework. If $m$m improves toward its base, its content $(m, U)$(m,U) must satisfy a pair of self-consistency conditions, which the authors state as a reference ideal. Write $\rho_U(\chi, \xi)$ρU​(χ,ξ) for the value that the base assigns to adopting agent instance $\xi$ξ from $\chi$χ. Then

where the first condition calibrates the modification critic to the base and the second requires the modifier to propose only agent instances that improve in the critic's estimate. These are characterization conditions, not enforced updates: they describe the internal structure a self-improver must have, and no external mechanism guarantees them.

3.3 Polarity of Self-Improvement: Anchored, Goal Drift, and Fully Self-Referential

The second dial applies to the base of the action critic and to the base of the modification critic alike. It separates polarities only when the modifier is part of the agent: in GPI the modification critic is never exercised, so the only base in use is that of the action critic, grounded in the world reward, and the polarity entry for GPI is anchored. For $m \in \mathrm{Ag}$m∈Ag, the second dial decides the polarity of self-improvement, namely whether the evaluation base $\rho$ρ stays grounded. The authors distinguish three states it can take: anchored, goal drift, and fully self-referential.

An external judge or reward model is anchored, whatever data it was trained on: the dial asks where the standard sits, not how faithful it is to the goal. A proxy can be anchored and still be satisfied without serving the goal; the same judge as an agent component is the goal drift case. The middle row, goal drift, describes the drift of the system away from its fixed goal, not a change in the goal itself: what moves is the standard that measures progress toward it. In current practice, most demonstrated self-improving systems fall into the anchored row; the two lower rows are so far represented mainly by boundary cases and by position papers.

The two dials, taken together, place every instance. Setting the modifier outside the agent and fixing the only base in use to the world reward gives the GPI loop with $\mathrm{Ag} = \{\pi, V\}$Ag={π,V}. Keeping the modifier inside the agent while keeping $\rho$ρ grounded gives anchored recursive self-improvement, as in STOP with its fixed validation examples, or the Gödel machine, whose proof gate restricts which rewrites are tried. Letting $\rho$ρ enter the agent, or vanish or point at itself, gives the unanchored end. Freezing the modifier as an external procedure and removing the modification critic and its base $\rho_U$ρU​ recover the classical loop over policy and value.

Experiment

The evaluation setup involves agents modifying their own training-data strategies under a fixed target model and external evaluation, probing the feasibility of recursive self-improvement via search over candidate selves. Results show that while a majority of first attempts improve performance, continuing past the best score frequently leads to lower final performance, demonstrating that the improvement step is non-monotone even with a fixed external standard. The underlying structural defects are identified as the lack of a general decision procedure over an unbounded content space and the loss of external evaluation when the evaluator becomes an agent component, both of which compromise the stability and validity of self-improvement.

The table categorizes policy-iteration methods and self-improvement agents by what they modify, whether the improvement mechanism is internal or external, and whether the evaluation basis is fixed outside the agent. Classical methods and some empirical loops keep the evaluation external, while more advanced self-improving agents internalize both modification and evaluation, trading guarantees for flexibility. Classical methods like policy/value iteration and DQN rely on external, fixed evaluators such as environment reward, while newer self-improvement agents often shift evaluation inward. Reflexion/Self-Refine and STOP improve only the agent's outputs or optimizer, keeping the improvement loop external, whereas Gödel-style agents modify their own code and improvement mechanisms internally. The guarantee weakens as evaluation becomes internal: classical methods converge under assumptions, empirical loops show bounded gains, and co-evolving evaluators offer no guarantee against a fixed basis.

The table categorizes self-improvement systems by the polarity of their evaluation base, which can be anchored outside the agent, drift as an agent-rewritable component, or become fully self-referential with no external grounding. The ordering reflects how much anchoring remains, with most current systems falling in the anchored row and the unanchored rows mostly represented by boundary cases or position papers. Anchored systems keep the evaluation standard external and unmodifiable, so improvement tracks a fixed goal, as in the Gödel machine and STOP. Goal Drift occurs when the agent may rewrite the evaluation base while the goal itself stays fixed, allowing progress measures to move away from the original objective. Fully Self-Referential systems have no external signal in the evaluation loop, so the critic reduces to internal self-consistency, as in Socratic learning.

The table classifies self-improvement systems by whether the modifier can modify itself and how the evaluation base is treated, revealing three main categories: anchored systems with fixed evaluation, drifting systems where the evaluator co-evolves, and fully self-referential systems with no external signal. Most existing systems, including fixed loops, the Gödel machine, and its descendants, are anchored because the evaluation benchmark remains external and unmodifiable, while only the Red Queen Gödel Machine and closed-system proposals depart from this pattern. Fixed outer loops and the Gödel machine are anchored because the evaluation base stays outside the agent, even when the modifier can edit the agent itself. In the Red Queen Gödel Machine, the evaluator co-evolves with the agent, so the evaluation base changes over time and no fixed external standard remains. Closed-system proposals allow any component, including the evaluation base, to be rewritten, leaving the system with no external signal. Gödel-agent descendants, such as Hyperagents, remain anchored because only the scaffold is modifiable and the benchmark is external.

The table identifies four defects in recursive self-improvement, each tied to a specific dial setting and the GPI condition it violates. These defects are structural, and some are observed in practice, highlighting where guarantees of Goal-directed Policy Improvement break down. Search over candidate selves occurs when the modifier is an agent component, violating the condition that improvement be cheap and monotone. Self-evaluation defect arises when the utility is an agent component, breaking the requirement that the evaluator stands outside what it evaluates. Ungrounded base results from an ungrounded progress mechanism, violating that the objective is evaluated from outside the agent. Goal drift is triggered when the progress mechanism is an agent component, losing the fixed standard as the base is rewritten. Structural defects are inherent, while observed defects are evidenced by systems like the Red Queen Gödel Machine or benchmarks showing non-monotone improvement.

The experiments use taxonomies and tables to categorize self-improvement agents and policy-iteration methods by what they modify and whether evaluation is internal or external. Classical methods (e.g., DQN, policy iteration) keep evaluation external and offer convergence guarantees, whereas newer agents like Reflexion and STOP alter only outputs or optimizers while retaining external loops. Gödel-style agents shift both modification and evaluation inward, trading formal guarantees for flexibility, and co-evolving evaluators or fully self-referential systems lose any fixed standard, leading to structural defects such as goal drift, self-evaluation bias, and ungrounded progress. The findings highlight that moving evaluation inward progressively weakens guarantees, with anchored systems preserving fixed objectives and unanchored ones risking non-monotone improvement.