One-sentence Summary

Researchers from The Chinese University of Hong Kong and LIGHTSPEED propose Skill-$\alpha$α, a reinforcement learning method that frames skill generation as a sequential editing process with a novel rollback reward evaluating each edit by comparing downstream execution under the original and edited skills on anchored queries, significantly outperforming heuristic and pipeline baselines on CL-Bench and tau2-bench.

Key Contributions

• Skill-α formulates skill generation as a progressive sequential editing process that decomposes skill construction into individually evaluable edits applied to a single evolving skill artifact.
• The method introduces a rollback reward that evaluates each edit by comparing downstream agent execution between the original and edited skills on an anchored query, providing an execution-grounded training signal.
• Experiments on CL-Bench, SpreadsheetBench, and tau2-bench show that Skill-α produces more effective skills than heuristic and pipeline baselines in both document-to-skill and experience-to-skill settings, boosting average success rates by up to 6.7 points, and ablations confirm the importance of rollback reward and progressive generation.

Introduction

Large language model agents tackling complex tasks often rely on external skills that condition their reasoning, tool use, and planning. Automatically generating these skills from documents or execution experience is critical for scalable deployment. Prior methods rely on heuristics or separate pipeline designs for different evidence types, providing no learning signal from downstream performance and limiting how evidence can improve a skill. The authors introduce Skill-α, a reinforcement learning framework that unifies document-to-skill and experience-to-skill generation as a progressive sequence of local edits. By introducing a rollback reward that isolates the effect of each edit on a fixed anchored query, Skill-α learns to add, revise, merge, or remove content based on execution-grounded feedback, producing more effective skills than pipeline-based baselines.

Method

The authors introduce Skill-α, a trainable framework designed for progressive skill generation. The core objective is to learn how to construct a skill from source evidence such that a fixed worker agent improves its behavior on held-out target queries. Since the ideal teacher behavior distribution is unavailable, the framework approximates this objective through indirect reward signals and local edit decisions.

As shown in the figure below, the framework operates in two distinct phases: inference and training. During inference, the model reads evidence sequentially and applies a sequence of local edit actions to progressively generate the final skill from an initial state. During training, the skill generator samples a group of candidate actions, constructs the corresponding edited skills, and evaluates them against a controlled baseline using a benchmark-specific verifier to compute rollback rewards for the GRPO update.

To address the limitations of one-shot skill generation, such as context window overflow and weak credit assignment, the authors decompose the process into progressive local edits. Starting from an initial skill state $z_0$z0​, the generator reads evidence units $x_t$xt​ sequentially and updates the skill one step at a time. At each step $t$t, a local edit action $A_t$At​ is sampled conditioned on the current skill state $z_{t-1}$zt−1​ and the evidence $x_t$xt​, and applied to obtain the next state $z_t = \text{Edit}(z_{t-1}, A_t)$zt​=Edit(zt−1​,At​). This formulation unifies document-to-skill and experience-to-skill settings into a single local decision problem.

The edit action space is defined as $\mathcal{A} = \{\text{CREATE, UPDATE, MERGE, PRUNE, NOOP}\}$A={CREATE, UPDATE, MERGE, PRUNE, NOOP}, allowing the model to add, fix, consolidate, remove, or retain skill artifacts. To provide supervision for these local decisions, the authors introduce a rollback reward mechanism. Directly rewarding an edit based on downstream task performance is insufficient because success might stem from the worker's inherent capabilities or query simplicity. Instead, the rollback reward assigns credit by comparing the worker's performance on an evidence-related anchored query $q_t^{\text{anc}}$qtanc​ under the original and edited skills. The fixed worker $\pi_\psi$πψ​ generates control and edited actions, $a_t^{\text{ctrl}} \sim \pi_\psi(\cdot \mid q_t^{\text{anc}}, z_{t-1})$atctrl​∼πψ​(⋅∣qtanc​,zt−1​) and $a_t^{\text{edit}} \sim \pi_\psi(\cdot \mid q_t^{\text{anc}}, z_t)$atedit​∼πψ​(⋅∣qtanc​,zt​), which are evaluated by a verifier $\mathcal{V}_t$Vt​ to yield scalar feedback $r_t^{\text{ctrl}}$rtctrl​ and $r_t^{\text{edit}}$rtedit​.

For training, the authors initialize the skill-editing policy $\pi_\phi$πϕ​ from an instruction-tuned Qwen3-8B model. The training pipeline consists of a supervised fine-tuning warm-up stage followed by reinforcement learning. During the warm-up stage, the policy is trained on synthetic edit trajectories to learn the action syntax and basic evidence-grounded editing behavior. Subsequently, the policy is optimized using Group Relative Policy Optimization (GRPO) while keeping the worker agent fixed as GPT-4o. For a given local editing state, the old policy samples a group of $G$G candidate actions. The rollback rewards are computed for each candidate and used to calculate the group-relative advantage $\widehat{\mathsf{A}}_i$Ai​. The policy is then optimized using the clipped GRPO objective:

where $\rho_i(\phi)$ρi​(ϕ) is the probability ratio between the new and old policies. This design ensures that reward assignment remains tightly aligned with the progressive formulation, training the policy on local edit states rather than directly optimizing a full generation trajectory.

Experiment

Skill-α is evaluated in both document-to-skill and experience-to-skill settings under same-worker and cross-worker transfer, consistently outperforming prior methods by producing reusable, transferable skills rather than mere context compression or trajectory memorization. Ablation experiments demonstrate that training with rollback rewards and structured edit operations like skill merging and pruning is essential for building robust skills, while a purely prompt-based or SFT-only approach yields unstable and reduced gains. Additional analysis reveals that the framework is robust to evidence ordering but benefits from a moderate evidence batch size, which balances pattern abstraction with local edit focus. Overall, Skill-α generalizes across evidence sources and downstream workers, capturing genuine task-solving knowledge instead of worker-specific shortcuts.

Skill-α compresses documents into reusable skills that improve performance on held-out CL-Bench tasks. Under GPT-4o, it delivers a large gain on Procedural Task Execution, achieves best or near-best scores on reasoning categories, and stays close to the no-skill baseline on Rule System Application. When transferred to Claude-Sonnet-4.5, Skill-α leads all four categories and records the highest average, demonstrating robust cross-backbone skill reuse. With GPT-4o, Skill-α lifts Procedural Task Execution pass rate from 4.30 (no skill) to 9.68, far exceeding the next best baseline (Anthropic Skill-Creator at 5.38). On Domain Knowledge Reasoning and Empirical Discovery, Skill-α is best or near-best under GPT-4o, while on Rule System Application it remains close to the no-skill baseline of 21.82. Under the Claude-Sonnet-4.5 backbone, Skill-α is best or tied for best on all four CL-Bench categories and achieves the highest overall average.

Skill-α delivers large, consistent improvements in experience-to-skill settings, outperforming prompt-based and prior experience-based methods on both SpreadsheetBench and tau2-bench. Under GPT-4o, it raises Airline pass rates from 40% to 65% and SpreadsheetBench from 18% to 28%, while the learned skills transfer effectively to Claude-Sonnet-4.5, where Skill-α remains best or tied-best on most tasks. Prompt-only baselines show sharp instability, underscoring the need for a trained skill-editing policy. Skill-α improves GPT-4o Airline performance by 25 percentage points and SpreadsheetBench by nearly 10 points over the no-skill baseline. It achieves the highest GPT-4o tau2-bench average (55.83) among all reported methods. The skills transfer across workers: using GPT-4o-generated skills, Claude-Sonnet-4.5 with Skill-α leads on SpreadsheetBench, Airline, Telecom, and the overall tau2-bench average. Prompt-based baselines are brittle: Anthropic Skill-Creator falls to 7.50 on Telecom, below the no-skill baseline of 12.50, and Progressive Prompt Skill drops below the no-skill baseline on SpreadsheetBench under Claude-Sonnet-4.5. Skill-α's stable gains across tasks and backbones contrast with iterative prompt methods, indicating that training a skill-editing policy is essential.

The full Skill-α model achieves the highest pass rates across all benchmarks, substantially outperforming the SFT-only baseline. Removing rollback reward drops performance to near SFT-only levels, making it the critical signal for learning effective edits. Excluding MERGE/PRUNE causes a sharp tau2-bench decline, showing that skill consolidation and deletion are necessary, while removing NOOP yields strong but consistently lower results, indicating that the ability to abstain from edits is useful but not essential. Full Skill-α outperforms SFT-only by large margins on every benchmark. Without rollback reward, pass rates fall close to the SFT-only baseline, revealing it as the key reward signal. Ablating MERGE/PRUNE causes a notable drop on tau2-bench, confirming that explicit skill consolidation and deletion prevent redundancy. Removing NOOP still leaves a relatively strong model, but the full model remains consistently better, showing that calibrated abstention reduces unnecessary edits.

Evidence order has only a mild effect on pass rates, with shuffled order slightly reducing tau2-bench performance while reverse order behaves similarly to source order. In contrast, evidence batch size strongly controls downstream quality: too few units per step leads to myopic overfitting, too many units overloads the editor, and a moderate batch size of 4 yields the best trade-off between abstraction and local control. Shuffled evidence order lowers tau2-bench pass rate from 55.8% (source order) to 51.7%, while reverse order at 57.5% remains comparable, suggesting robustness with a small benefit from local continuity. Using only 1 evidence unit per step drops tau2-bench pass rate to 47.5%, far below the optimal 4-unit setting, as the editor becomes myopic and fragments the skill.

Skill-α compresses documents into reusable skills and is evaluated on CL-Bench, SpreadsheetBench, and tau2-bench tasks. It yields consistent performance gains across GPT-4o and Claude-Sonnet-4.5, demonstrating robust cross-backbone skill transfer while far surpassing prompt-based baselines, which often exhibit instability. Ablation experiments reveal that a rollback reward signal is essential for learning effective edits, and that explicit skill consolidation and deletion prevent redundancy. The approach is largely insensitive to evidence order, whereas a moderate evidence batch size best balances abstraction and local control, underscoring the need for a trained skill-editing policy.