Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has demonstratedsuccess in enhancing LLM reasoning capabilities, but remains limited tosingle-turn interactions without tool integration. While recent AgenticReinforcement Learning with Tool use (ARLT) approaches have emerged to addressmulti-turn tool interactions, existing works develop task-specific codebasesthat suffer from fragmentation, synchronous execution bottlenecks, and limitedextensibility across domains. These inefficiencies hinder broader communityadoption and algorithmic innovation. We introduce VerlTool, a unified andmodular framework that addresses these limitations through systematic designprinciples. VerlTool provides four key contributions: (1) upstream alignmentwith VeRL ensuring compatibility and simplified maintenance, (2) unified toolmanagement via standardized APIs supporting diverse modalities including codeexecution, search, SQL databases, and vision processing, (3) asynchronousrollout execution achieving near 2times speedup by eliminatingsynchronization bottlenecks, and (4) comprehensive evaluation demonstratingcompetitive performance across 6 ARLT domains. Our framework formalizes ARLT asmulti-turn trajectories with multi-modal observation tokens (text/image/video),extending beyond single-turn RLVR paradigms. We train and evaluate models onmathematical reasoning, knowledge QA, SQL generation, visual reasoning, websearch, and software engineering tasks, achieving results comparable tospecialized systems while providing unified training infrastructure. Themodular plugin architecture enables rapid tool integration requiring onlylightweight Python definitions, significantly reducing development overhead andproviding a scalable foundation for tool-augmented RL research. Our code isopen-sourced at https://github.com/TIGER-AI-Lab/verl-tool.