One-sentence Summary

Researchers from Aether AI, UC San Diego, and UIC introduce RSIAgent, a training-free multi-agent framework that coordinates curriculum, actor, and verifier agents to autonomously explore new environments, validate outcomes, and build reusable causal memory through a broad-then-deep exploration strategy, enabling open-source models Kimi-K3 and GLM-5.3 to outperform frontier closed-source models including GPT-6 on OSWorld-v2 and Agent’s Last Exam without parameter updates.

Key Contributions

• RSIAgent is a training-free multi-agent framework where curriculum, actor, and verifier agents autonomously explore a digital environment and construct reusable memory containing causal relationships between actions, conditions, and consequences.
• It uses a broad-then-deep recursive self-exploration strategy: broad recursive self-exploration builds diverse environment coverage, and deep recursive self-exploration targets hard cases, hidden constraints, boundary conditions, and unknown causal dependencies.
• Experiments on OSWorld-v2 and Agent’s Last Exam show that the frozen memory improves strong open-source models, enabling Kimi-K3 and GLM-5.3 to outperform frontier closed-source models such as GPT-6 without updating model parameters.

Introduction

Large language and vision-language models have enabled digital agents that automate complex tasks by observing interfaces and executing actions like clicks or programs. However, deploying these agents in new environments is challenging because their pretrained knowledge may not cover unfamiliar interfaces, tools, and failure modes. Traditional adaptation methods require costly, human-assisted data collection and retraining, which is impractical for private or rapidly changing settings. Training-free approaches that manage context and memory offer flexibility but often merely memorize successful trajectories without uncovering causal relationships between actions, environment conditions, and outcomes. The authors introduce RSIAgent, a multi-agent framework that autonomously explores a new environment to discover such causal knowledge and consolidate it into a reusable memory. By coordinating curriculum, actor, and verifier agents and employing a broad-then-deep exploration strategy, RSIAgent enables recursive self-improvement without updating model parameters, allowing open-source models to surpass frontier closed-source models on standard benchmarks.

Method

The authors present RSIAgent, a recursive self-improvement framework designed to enable agents to autonomously explore and adapt to novel environments. To achieve robust control over autonomous exploration and self-improvement, the authors design a multi-agent harness framework that operates through a coordinated recursive loop. This system decomposes the agent architecture into three collaborative roles.

As shown in the figure below:

The first component is the Actor Agent with Evolvable Memory. Serving as the primary policy model, this agent is responsible for understanding the environment and generating executable actions. It is equipped with a persistent memory module that stores environment-specific knowledge, reusable procedures, and lessons learned from prior executions. This memory is highly evolvable. Following an evaluation by the verifier, the actor consolidates the grounded experience by integrating new knowledge and revising or discarding outdated information. Subsequent actor instances inherit this updated memory, allowing useful knowledge to accumulate continuously throughout the exploration phase.

The second component is the Verifier Agent with Environment Feedback. This agent acts as an independent evaluator, determining whether the execution of the actor successfully satisfies the task requirements. To ensure reliable judgments, the verifier directly inspects environmental feedback, including execution results and interface states, grounding its decisions in observable evidence. Crucially, the verifier is isolated from the private reasoning and memory of the actor, which mitigates correlated errors during evaluation. It returns a binary success or failure judgment alongside supporting feedback to guide memory consolidation.

The third component is the Curriculum Agent for Guiding Exploration. Functioning as the high-level coordinator, this agent dictates the direction of exploration by generating suitable practice tasks for the actor and verifier. It selects prerequisite skills, informative variants, failure-driven practice, and stress-test cases based on the current target, accumulated memory, and previous outcomes. This progressive task generation expands the coverage of the evolvable memory until the generated tasks yield diminishing returns in new knowledge acquisition.

Building upon this multi-agent framework, the authors organize the autonomous exploration process into two complementary stages. The first stage is Broad Recursive Self-exploration (BRS), which aims to rapidly construct a comprehensive understanding of a new environment by collecting diverse interaction experiences. In this phase, the curriculum agent proposes multiple tasks spanning different exploration directions, which are executed and verified in parallel. The curriculum agent then analyzes the accumulated experience to identify remaining knowledge gaps, generating more informative tasks for the next iteration. This recursive process progressively expands the coverage of environment-specific knowledge, reusable procedures, and failure patterns.

The second stage is Deep Recursive Self-exploration (DRS), which refines the accumulated memory by targeting critical knowledge gaps, hard cases, and boundary conditions revealed during initial execution. DRS follows a sequential recursive loop that progressively increases exploration difficulty. The curriculum agent proposes a challenging task designed to expose unpredictable issues or weaknesses in the current memory. The actor attempts the task using the accumulated memory, and the verifier evaluates the outcome. Based on the resulting successes, failures, and newly revealed uncertainties, the curriculum agent generates a more challenging follow-up task. Each verified experience is consolidated into memory before the subsequent task is proposed, continuously pushing the agent toward harder and less explored scenarios.

Following the exploration phases, the system transitions to Test-time Memory Reuse. The accumulated memory is frozen and provided to the actor agent for downstream evaluation. During this stage, the curriculum agent and all memory update mechanisms are disabled. Given a target task, the actor agent directly reuses the procedures, discovered constraints, and failure lessons stored in memory to guide its actions. The verifier agent evaluates the resulting outcome against the task requirements, and this action-verification loop continues until the verifier confirms that all task requirements have been fully satisfied.

Experiment

The experiments evaluate RSIAgent on OSWorld 2.0 and ALE Near-term using a shared code-as-policy harness with GLM-5.3 as the actor and Kimi-K3 as the verifier and curriculum agent. The main results show that recursive self-improvement improves both partial-credit and full-completion performance, and the system attains leading partial-credit scores compared with frontier models. Ablations indicate that combining broad and deep recursive self-exploration is more effective than either stage alone, while analysis of RSI rounds shows that accumulated memory can produce discrete task-level gains as it covers critical requirements. Additional game-environment evaluations confirm generalization to autonomous game development, and failure analysis identifies insufficiently targeted exploration, incomplete verification, and unreliable memory consolidation as the main limiting mechanisms.

Recursive Self-Improvement (RSI) enhances an existing agent harness, lifting partial and binary accuracy on OSWorld 2.0 and Agents' Last Exam Near-term. RSIAgent, using GLM-5.3 and Kimi-K3, achieves the top partial scores, surpassing frontier models like GPT-6 Astra and Claude Opus 5. The gains stem from a multi-agent framework with two-stage exploration and memory reuse, strengthening open-source models without parameter updates. RSI lifts OSWorld partial score from 71.97 to 78.98 and binary accuracy from 37.80 to 42.68, with similar improvements on ALE. RSIAgent achieves the highest partial scores, exceeding GPT-6 Astra by 6.38 points on OSWorld and 2.56 points on ALE, and also surpassing Claude Opus 5. Combining a multi-agent harness with two-stage exploration and memory reuse allows open-source models to surpass frontier systems without fine-tuning.

RSIAgent consistently improves game quality across all base generators, with the full version incorporating RSI experience achieving the highest scores. Play2Code helps weaker base games but can degrade already strong ones, while RSIAgent benefits both weak and strong starting points. Adding accumulated RSI experience further boosts all quality dimensions. RSIAgent with RSI experience yields the best overall quality for both strong and weak base generators. Play2Code improves games from weaker generators but reduces quality for already strong games. RSIAgent without RSI already outperforms Play2Code and the baseline across all metrics. Incorporating RSI experience provides additional gains in mechanics, depth, visuals, and art.

The experiments evaluate a recursive self-improvement agent harness on OSWorld 2.0 and Agents' Last Exam Near-term, as well as on game generation with weak and strong base generators. RSIAgent, using GLM-5.3 and Kimi-K3 with two-stage exploration and memory reuse, improves both partial and binary accuracy and achieves top scores above frontier models without parameter updates. In game quality, RSIAgent consistently improves all base generators, while Play2Code helps weaker generators but can degrade already strong ones, and adding accumulated RSI experience further boosts mechanics, depth, visuals, and art. Overall, the multi-agent RSI harness lets open-source models surpass frontier systems and enhances both weak and strong starting points.