Abstract

Recently, Agentic Reinforcement Learning (Agentic RL) has made significantprogress in incentivizing the multi-turn, long-horizon tool-use capabilities ofweb agents. While mainstream agentic RL algorithms autonomously explorehigh-uncertainty tool-call steps under the guidance of entropy, excessivereliance on entropy signals can impose further constraints, leading to thetraining collapse. In this paper, we delve into the challenges caused byentropy and propose the Agentic Entropy-Balanced Policy Optimization (AEPO), anagentic RL algorithm designed to balance entropy in both the rollout and policyupdate phases. AEPO comprises two core components: (1) a dynamicentropy-balanced rollout mechanism that adaptively allocate global and branchsampling budget through entropy pre-monitoring, while imposing a branch penaltyon consecutive high-entropy tool-call steps to prevent over-branching issues;and (2) Entropy-Balanced Policy Optimization that inserts a stop-gradientoperation into the high-entropy clipping term to preserve and properly rescalegradients on high-entropy tokens, while incorporating entropy-aware advantageestimation to prioritize learning on high-uncertainty tokens. Results across 14challenging datasets show that AEPO consistently outperforms 7 mainstream RLalgorithms. With just 1K RL samples, Qwen3-14B with AEPO achieves impressiveresults: 47.6% on GAIA, 11.2% on Humanity's Last Exam, and 43.0% on WebWalkerfor Pass@1; 65.0% on GAIA, 26.0% on Humanity's Last Exam, and 70.0% onWebWalker for Pass@5. Further analysis reveals that AEPO improves rolloutsampling diversity while maintaining stable policy entropy, facilitatingscalable web agent training.