One-sentence Summary

Researchers from Alibaba and collaborators propose Skill Self-Play (Skill-SP), a co-evolutionary framework that uses agent skills to balance bounded environment feedback with open-ended generation, enabling a proposer, solver, and skill controller to co-evolve through reinforcement learning, thereby bridging structured verification and exploration and consistently improving LLM performance on tool-use and reasoning benchmarks.

Key Contributions

• The Skill Self-Play (Skill-SP) framework co-evolves a proposer, a solver, and a dynamic skill controller in a reinforcement learning loop, where the controller continuously updates a reusable skill library to condition task generation and provide verifiable execution boundaries.
• The framework treats the evolving skill library as a proactive training-time interface, using skill-conditioned generation to guide the proposer toward high-fidelity curricula at the solver's learning frontier while preserving task diversity and rigorous verification.
• Empirical evaluations on tool-use benchmarks (API-Bank, BFCL) and logical reasoning (ZebraLogic) show that Skill-SP yields absolute gains of up to +42.9 points on tool tasks and +12.0 points on reasoning, consistently outperforming unguided self-play and enabling performance turnarounds for initially misaligned models.

Introduction

Self-play approaches allow large language models to improve autonomously by generating tasks and learning through multi-agent reinforcement learning, showing gains in reasoning, search, and tool use. However, prior work either confines training to narrow, externally verified environments or relies on unguided task generation with passive filtering, which suffers from ill-posed tasks, template overfitting, and synthetic data collapse over successive iterations. The authors introduce Skill Self-Play (Skill-SP), a framework that evolves a library of modular skill packages to proactively guide task generation, enforce rigorous verification, and control curriculum difficulty. This design bridges the gap between rigid environment-bound play and chaotic open-ended generation, enabling sustained self-improvement without sacrificing verification reliability. Evaluated on tool-calling and logical reasoning benchmarks, Skill-SP substantially outperforms unguided self-play, demonstrating how co-evolving skills can anchor model training at its learning frontier and open a structured pathway for unbounded LLM self-evolution.

Method

The authors propose Skill Self-Play (Skill-SP), a training-time framework that transforms generic self-play into a proactive curriculum-construction process. At each iteration, Skill-SP orchestrates the joint optimization of a proposer policy $\pi_{\mathrm{propose}}$πpropose​ and a solver policy $\pi_{\mathrm{solve}}$πsolve​, alongside a skill controller managing an evolving library $\mathcal{S}$S.

Refer to the framework diagram:

The framework formalizes a verifiable agent task as a tuple $(x, c)$(x,c), where $x$x is the standard prompt and $c$c is a hidden verification contract. To drive continuous improvement, the proposer targets the solver learning frontier. However, to prevent reward hacking where the proposer synthesizes ill-posed contracts, the authors introduce a gated curriculum reward. This incorporates a binary task quality filter that retains only candidates with structurally sound contracts:

$$\mathcal{R}_{\text{propose}}(\boldsymbol{x}, \boldsymbol{c}; \pi_{\text{solve}}) = \mathbb{1}_{\{(\boldsymbol{x}, \boldsymbol{c}) \text{ is valid}\}} \cdot \left(1 - 2 \left| \nu_{\text{solve}}(\boldsymbol{x}, \boldsymbol{c}; \pi_{\text{solve}}) - \frac{1}{2} \right| \right)$$Rpropose​(x,c;πsolve​)=1{(x,c) is valid}​⋅(1−2​νsolve​(x,c;πsolve​)−21​​)

where $\nu_{\text{solve}}$νsolve​ is the solver expected success rate. Skill-SP is formulated as a bi-level optimization problem where the outer objective jointly optimizes the skill library and the proposer, subject to an inner objective where the solver maximizes execution success.

To operationalize the outer objective, the proposer synthesizes valid and frontier-targeted tasks through skill-conditioned generation. A skill $s \in \mathcal{S}$s∈S serves as a structural interface providing procedural rules, generation hints, and executable validators. The proposer dynamically samples a skill based on historical usage statistics, balancing exploitation and exploration. This is complemented by an open-ended exploration stream without skill constraints to prevent mode collapse. Generated candidates undergo rigorous validity verification, ensuring schema compliance, contract validity, and probe consistency. The proposer policy is then updated via Group Relative Policy Optimization (GRPO) to maximize the expected gated reward.

For the inner objective, the solver is optimized on a dynamic curriculum. The authors construct a high-quality training pool by selecting the most challenging tasks from both the skill and exploration streams. Candidates are ranked by the proposer reward to pinpoint the solver learning frontier, and the top-scoring tasks are selected according to a blending ratio $\alpha$α. The solver policy is subsequently updated via GRPO to maximize the environment verification reward on this constructed curriculum.

To prevent curriculum stagnation, the skill library dynamically co-evolves with the policies. This evolution comprises three operations. First, skill refinement updates tracking statistics and analyzes execution trajectories from invalid attempts to automatically refine core skill content. Second, skill pruning identifies and archives saturated skills that consistently yield trivial tasks, conserving the generation budget. Third, skill induction extracts novel candidates from the exploration stream that meet a minimum frontier reward threshold. A skill controller abstracts the underlying task patterns from these candidates into new skills, filtering them for structural integrity and semantic novelty against the current library. The active library is updated by removing pruned skills and adding newly induced ones, distilling training feedback into reusable structural priors for subsequent task generation.

Experiment

Across tool-use and logical reasoning benchmarks, Skill-SP consistently improves all tested backbones over both the base models and unguided self-play, notably rescuing initially misaligned models that stagnate under naive self-play. Ablation studies validate that proactive skill orchestration, mixed open-ended exploration, and adaptive routing are crucial for generalization and for avoiding mode collapse. Diagnostics further show that the framework constructs a diverse, frontier-targeted curriculum through a dynamically evolving skill library, which steadily expands both active and effective skills to prevent the generator from resting on a few dominant patterns.

Skill-SP consistently improves tool-calling performance across all difficulty levels and programming languages, yielding larger overall gains than Unguided SP. For Qwen3-8B, Unguided SP introduces a minor regression on BFCL Live, whereas Skill-SP delivers strictly positive gains on every subtask for both shown models. Skill-SP raises the overall average by 6.5 points for Qwen3-4B-Instruct and 2.8 points for Qwen3-8B, roughly doubling the gains of Unguided SP on both models. Unguided SP causes a performance drop on BFCL Live for Qwen3-8B (‑0.1), while Skill-SP improves it by 0.8 points and lifts all other metrics without any negative changes.

Skill-SP improves logical reasoning accuracy on the ZebraLogic benchmark across all five tested models. Gains are largest for initially weak models on small- and medium-scale puzzles, where grid-level accuracy moves from near zero to meaningful levels, while strong models achieve steady but smaller improvements. On extreme puzzle scales with very large search spaces, weak base models show minimal progress even with Skill-SP. Overall grid-level accuracy rises for every backbone, with absolute gains spanning from about 1.4 points for a strong model to over 8 points for a weaker one. On small puzzles, Skill-SP delivers over 35 points of gain for a logically misaligned model, whereas a strong model already exceeds 97% accuracy. Cell-level accuracy consistently benefits, with improvements reaching up to 20 points for models starting from very low baselines. At Large and X-Large puzzle scales, models with near-zero base performance stay close to zero even after Skill-SP, indicating a bootstrapping threshold for extreme search spaces.

Ablations reveal that entirely removing skill orchestration (Unguided SP) causes the largest performance regression, while substituting dynamic curriculum allocation with static strategies like uniform routing or frozen skills also consistently degrades results. The findings confirm that performance gains depend on adaptive curriculum and continuous library evolution, not simply on the presence of fixed structural constraints. Unguided self-play suffers the steepest overall accuracy drop, indicating that structural guidance is essential to prevent early training plateaus. Both uniform routing and frozen skill variants underperform the full system, demonstrating that static skill assignment and a non-evolving library fail to match dynamic orchestration.

Freezing the task proposer leads to a notable overall performance decline, while freezing the feedback solver causes an even larger drop. Disabling both updates entirely produces the worst degradation, confirming that continuous co-adaptation of the two policies is essential for effective curriculum generation. Freezing the proposer lowers overall accuracy by 2.1 points, showing the generator must adapt to the evolving skill library. Freezing the feedback solver causes a 3.0-point overall drop, highlighting that outdated difficulty signals severely weaken the proposer's training. Freezing both components yields the largest decline of 3.2 points, proving the necessity of mutual co-evolution for proactive curriculum design.

The evaluation compares Skill-SP against unguided self-play on tool-calling (BFCL) and logical reasoning (ZebraLogic) benchmarks across multiple models. Skill-SP reliably outperforms unguided self-play, delivering strictly positive gains in tool calling and significantly improving reasoning accuracy, particularly for weaker models on smaller puzzle scales. Ablation studies confirm that adaptive skill orchestration and continuous library evolution are essential, as static assignments or frozen skills cause severe degradation. Additionally, co-adaptation of the task proposer and feedback solver is critical, with disabling either component leading to notable performance drops, highlighting that dynamic curriculum generation drives the method's effectiveness.