One-sentence Summary

The proposed SKILLRL framework evolves LLM agents through recursive skill-augmented reinforcement learning by utilizing an experience-based distillation mechanism to build a hierarchical SKILLBANK and an adaptive retrieval strategy, which transforms redundant trajectories into reusable behavioral patterns to achieve state-of-the-art performance on ALF-World, WebShop, and seven search-augmented tasks.

Key Contributions

• The paper introduces SKILLRL, a framework that bridges the gap between raw experience and policy improvement through automatic skill discovery and recursive evolution.
• This work presents an experience-based distillation mechanism that builds a hierarchical skill library called SKILLBANK, which differentiates between general strategic guidance and task-specific heuristics to reduce token footprint and enhance reasoning utility.
• The method incorporates a recursive evolution mechanism that allows the skill library and the agent policy to co-evolve during reinforcement learning, achieving state-of-the-art performance with a 15.3% improvement over baselines on benchmarks such as ALF-World and WebShop.

Introduction

Large Language Model (LLM) agents are increasingly used for complex tasks like web navigation and deep research, but they often operate in isolation without the ability to learn from past experiences. Current memory-based methods typically store raw trajectories, which are often redundant, noisy, and difficult for models to use for high-level reasoning. The authors propose SKILLRL, a framework that bridges the gap between raw experience and policy improvement through automatic skill discovery and recursive evolution. They leverage an experience-based distillation mechanism to transform trajectories into a hierarchical skill library called SKILLBANK, which contains both general and task-specific skills. By incorporating a recursive evolution mechanism, the authors ensure that the skill library and the agent policy co-evolve during reinforcement learning, significantly improving reasoning utility and performance while reducing the token footprint.

Method

The authors propose SKILLRL, a framework designed to bridge the gap between raw interaction experience and policy improvement through automatic skill discovery and recursive evolution. The framework aims to transform verbose, redundant trajectories into concise, actionable knowledge that can be effectively utilized within the limited context windows of LLM-based agents.

As shown in the figure below:

The SKILLRL framework consists of three core components: experience-based skill distillation, hierarchical skill library construction, and recursive skill evolution.

Experience-based Skill Distillation

To convert raw trajectories into usable knowledge, the authors employ a teacher model $\mathcal{M}_T$MT​ to distill experiences. Unlike traditional methods that only focus on successful episodes, the authors deliberately collect both successful trajectories $\mathcal{T}^{+} = \{\tau_{i} : r(\tau_{i})=1\}$T+={τi​:r(τi​)=1} and failed trajectories $\mathcal{T}^{-} = \{\tau_{i} : r(\tau_{i})=0\}$T−={τi​:r(τi​)=0}. Successful trajectories are processed to extract strategic patterns: $s^{+} = \mathcal{M}_{T}(\tau^{+}, d)$s+=MT​(τ+,d) This allows the teacher model to identify critical decision points and generalizable reasoning. For failed trajectories, the teacher model synthesizes concise failure lessons: $s^{-} = \mathcal{M}_{T}(\tau^{-}, d)$s−=MT​(τ−,d) This transformation identifies the point of failure and provides counterfactual guidance, turning noisy, lengthy failed episodes into actionable principles.

Hierarchical Skill Library (SKILLBANK)

The distilled knowledge is organized into a hierarchical structure called SKILLBANK. It is divided into two levels:

1. General Skills $S_g$Sg​: These capture universal strategic principles, such as exploration strategies and state management, which are applicable across all tasks.
2. Task-Specific Skills $S_k$Sk​: These encode specialized knowledge for specific task categories, including domain-specific action sequences and unique failure modes.

The complete library is defined as $\mathcal{S}_g \cup \bigcup_{k=1}^{K} \mathcal{S}_k$Sg​∪⋃k=1K​Sk​. During inference, general skills are always provided, while task-specific skills are retrieved via semantic similarity: $\mathcal{S}_{\text{ret}} = \text{TopK}(\{s \in \mathcal{S}_k : \text{sim}(e_d, e_s) > \delta\}, K)$Sret​=TopK({s∈Sk​:sim(ed​,es​)>δ},K) where $e_d$ed​ and $e_s$es​ are the embeddings of the task description and the skill, respectively.

Recursive Skill Evolution and RL Training

To ensure the agent can effectively use these skills, the authors implement a cold-start stage using Supervised Fine-Tuning (SFT). The teacher model generates skill-augmented reasoning traces $\mathcal{D}_{\text{SFT}}$DSFT​, and the base model is fine-tuned to minimize the cross-entropy loss: $\theta_{\text{sft}} = \arg \min_{\theta} \mathcal{L}_{\text{CE}}(\mathcal{D}_{\text{SFT}}; \theta)$θsft​=argminθ​LCE​(DSFT​;θ)

Following initialization, the policy $\pi_{\theta}$πθ​ is optimized using Group Relative Policy Optimization (GRPO). For each task, the agent retrieves skills and samples $G$G trajectories. The policy is updated using a clipped objective that incorporates a KL penalty to maintain stability relative to the reference policy $\pi_{\text{ref}}$πref​: $\mathcal{J}(\theta) = \mathbb{E}_{d, \{\tau^{(i)}\}} \left[ \frac{1}{G} \sum_{i=1}^{G} \min(\rho_i A_i, \text{clip}(\rho_i, 1-\epsilon, 1+\epsilon)A_i) - \beta D_{\text{KL}}(\pi_{\theta} || \pi_{\text{ref}}) \right]$J(θ)=Ed,{τ(i)}​[G1​∑i=1G​min(ρi​Ai​,clip(ρi​,1−ϵ,1+ϵ)Ai​)−βDKL​(πθ​∣∣πref​)]

To prevent the skill library from becoming static, the authors introduce recursive evolution. If the success rate for a task category falls below a threshold, the framework collects failed validation trajectories $\mathcal{T}_{\text{val}}^-$Tval−​ and uses the teacher model to generate new skills: $S_{\text{new}} = \mathcal{M}_T(\mathcal{T}_{\text{val}}^-, \text{SKILLBANK})$Snew​=MT​(Tval−​,SKILLBANK) These new skills are then integrated into the library, allowing the skills and the policy to co-evolve.

Experiment

SKILLRL is evaluated across nine benchmarks, including ALFWorld, WebShop, and various search-augmented QA tasks, to compare its performance against closed-source models, prompt-based agents, and reinforcement learning methods. The experiments validate that abstracting experiences into a co-evolving skill library significantly outperforms methods relying on raw trajectory memory or static prompting. Overall, the results demonstrate that structured skill distillation enables smaller open-source models to surpass much larger closed-source counterparts while accelerating training convergence and improving context efficiency.

The authors conduct ablation studies to evaluate the impact of different components within the SKILLRL framework on ALFWorld and WebShop performance. The results demonstrate that the full SKILLRL configuration achieves the highest success rates across both benchmarks. Removing the hierarchical structure or replacing the skill library with raw trajectories leads to significant performance decreases. The cold-start SFT phase is essential for maintaining high success rates in both environments. Dynamic evolution contributes to improved performance compared to a static skill library.

The authors compare SKILLRL against various baseline methods including closed-source LLMs, prompt-based agents, and reinforcement learning approaches on the ALFWorld and WebShop benchmarks. Results show that SKILLRL consistently achieves superior performance across most subtasks and overall metrics compared to all tested baselines. SKILLRL outperforms both closed-source models and standard reinforcement learning methods in overall success rates. The method demonstrates significant improvements in complex ALFWorld subtasks compared to memory-augmented RL baselines. SKILLRL achieves higher average scores and success rates on the WebShop benchmark than prompt-based and RL-based competitors.

The authors evaluate SKILLRL on various search-augmented single-hop and multi-hop QA tasks to demonstrate its reasoning capabilities. Results show that the method achieves a higher average score compared to both specialized search-augmented baselines and memory-augmented reinforcement learning frameworks. SKILLRL achieves superior performance on multi-hop reasoning tasks such as Bamboogle and MuSiQue compared to existing baselines. The method demonstrates strong generalization by maintaining competitive results on out-of-domain datasets. SKILLRL outperforms state-of-the-art search-augmented models in overall average score across the evaluated QA benchmarks.

Through ablation studies, comparative benchmarks on ALFWorld and WebShop, and reasoning evaluations on QA tasks, the authors validate the effectiveness of the SKILLRL framework. The results demonstrate that the full configuration, including its hierarchical structure and dynamic skill library, consistently outperforms closed-source LLMs, prompt-based agents, and standard reinforcement learning methods. Furthermore, the framework exhibits strong reasoning capabilities and generalization across complex multi-hop tasks and out-of-domain datasets.