One-sentence Summary

Researchers from Shanghai Jiao Tong University, Zhejiang University, National University of Singapore, and other institutions introduce EnvACE, an agentic reinforcement learning method that internalizes environment dynamics through world rehearsal, where the policy alternates between generating tool calls and role-playing environment responses, both jointly optimized by task-success rewards to replace external environment interaction, achieving strong and transferable performance across long-horizon tool-use benchmarks including BFCL-v4, τ2-Bench, VitaBench, and FinMCP-Bench while outperforming environment-scaling baselines, and further enabling private test-time rehearsal that yields additional gains under a moderate budget.

Key Contributions

• The work introduces world rehearsal, a training paradigm where a single policy alternately generates tool calls and simulates the corresponding environment responses, internalizing environment dynamics without requiring external executable environments.
• EnvACE is proposed as an agentic reinforcement learning method that jointly optimizes acting and rehearsal in self-unfolded trajectories by sharing model parameters and applying role-wise GRPO with end-to-end task-success rewards.
• Experiments across BFCL-v4, τ²-Bench, VitaBench, and FinMCP-Bench show that EnvACE consistently outperforms environment-scaling baselines, controlled analyses confirm the benefits of world rehearsal across model scales, and test-time rehearsal yields additional improvements under a moderate budget.

Introduction

Large language models are increasingly deployed as agents that must combine dialogue with tool use across long interaction horizons. Prior training approaches either rely on costly, hard-to-scale executable environments or on LLM-based simulators that can produce inaccurate feedback and still require real-environment grounding. The authors propose EnvACE, a method that trains a single policy to both act and rehearse the environment’s responses inline, internalizing environment dynamics without any external environment interaction. By jointly optimizing acting and environment modeling with task-success rewards, the policy learns to unfold trajectories autonomously through a process called world rehearsal.

Method

The authors propose EnvACE, a framework that internalizes the agent-environment interaction loop into a single policy. As shown in the figure below:

EnvACE comprises three core components: world rehearsal, role-wise Group Relative Policy Optimization (GRPO), and test-time scaling.

In conventional agentic reinforcement learning, an external environment provides observations after each action. EnvACE revises this boundary by assigning observation generation to a rehearsal role of the policy itself. The task is formulated as a finite-horizon partially observable Markov decision process. At step $t$t, the policy observes the interaction history $h_t$ht​ and generates an action $a_t$at​. Instead of querying an external environment, the policy alternates between acting and rehearsal. The acting role generates an environment-facing action:

$$a_t \sim \pi_\theta(\cdot \mid h_t, \text{ACT}).$$at​∼πθ​(⋅∣ht​,ACT).

Conditioned on the history and the generated action, the rehearsal role generates the corresponding environment response:

$$\hat{o}_t \sim \pi_\theta(\cdot \mid h_t, a_t, \text{REHEARSE}).$$o^t​∼πθ​(⋅∣ht​,at​,REHEARSE).

The generated response is appended to the interaction history as $h_{t+1} = h_t \oplus (a_t, \hat{o}_t)$ht+1​=ht​⊕(at​,o^t​), from which the acting role makes its next decision. This unified act-rehearse process allows the trajectory to unfold without an external environment, enabling the policy to internalize environment dynamics as an agent world model.

To optimize this process, the authors employ role-wise GRPO. For each instruction $x$x, EnvACE samples a group of $K$K rollouts, each receiving a trajectory-level reward $R_i$Ri​. Every policy output in a rollout inherits this reward. The authors collect all policy outputs generated under a specific role $r \in \{\text{ACT}, \text{REHEARSE}\}$r∈{ACT,REHEARSE} across the $K$K rollouts to form a group $\mathcal{G}_{x,r}$Gx,r​. A separate reward baseline is computed for each role:

$$\mu_{x,r} = \frac{1}{|\mathcal{G}_{x,r}|} \sum_{y_{j,n} \in \mathcal{G}_{x,r}} R_j.$$μx,r​=∣Gx,r​∣1​yj,n​∈Gx,r​∑​Rj​.

The role-wise advantage $A_{i,m}$Ai,m​ of an output $y_{i,m}$yi,m​ is defined relative to this baseline:

$$A_{i,m} = R_i - \mu_{x,r_{i,m}}.$$Ai,m​=Ri​−μx,ri,m​​.

The shared policy is optimized using the clipped GRPO objective:

$$\max_\theta \mathcal{J}(\theta) = \mathbb{E}_{x,i,m,\ell} \left[ \min \left( \rho_{i,m,\ell}(\theta) A_{i,m}, \text{clip}(\rho_{i,m,\ell}(\theta), 1-\epsilon, 1+\epsilon) A_{i,m} \right) \right],$$θmax​J(θ)=Ex,i,m,ℓ​[min(ρi,m,ℓ​(θ)Ai,m​,clip(ρi,m,ℓ​(θ),1−ϵ,1+ϵ)Ai,m​)],

where $\rho_{i,m,\ell}(\theta)$ρi,m,ℓ​(θ) is the standard GRPO likelihood ratio. Although baselines are computed separately, outputs from both roles jointly update the shared policy parameters $\theta$θ.

At test time, EnvACE leverages world rehearsal to scale inference-time computation before interacting with the external environment. Given a new instruction $x$x, the policy performs $N$N private rehearsal attempts. These attempts can be conducted in two modes. In parallel mode, all attempts are generated independently from the same context. In sequential mode, each new attempt observes previous rehearsal trajectories along with their assessments and revision suggestions, allowing iterative refinement. After completing the attempts, EnvACE summarizes all trajectories and self-evaluations into a compact rehearsal memory $m_x$mx​. The acting role then conditions on $m_x$mx​ during a single committed execution in the external environment, while the rehearsals remain private and do not alter the external state.

Experiment

EnvACE is evaluated on four agentic benchmarks, BFCL-v4, τ²-Bench, VitaBench, and FinMCP-Bench, covering function calling, stateful service interactions, and financial tool use, and is compared against Qwen3 models and environment-scaling baselines. The world rehearsal paradigm, where a single policy jointly acts and generates environment responses, consistently outperforms standard GRPO and representative methods, with particularly strong gains on stateful, multi-turn tasks. Parameter sharing between the acting and rehearsal roles helps the policy internalize environment dynamics, and the benefits scale with model capacity. Test-time rehearsal further improves performance without external interaction, confirming world rehearsal as an effective and scalable training framework for tool-use agents.

EnvACE achieves the strongest overall result among environment-scaling baselines, with consistent advantages across heterogeneous benchmarks. It outperforms a larger Qwen3-8B foundation model on BFCL V4 and leads all 7B–8B models on VitaBench, while scoring second-highest on τ²-Bench. The approach scales well with model size, and combining it with test-time world rehearsal yields further gains over standard inference. EnvACE surpasses EnvScaler-8B and AWM-14B on the overall average, reflecting broad robustness. On τ²-Bench, EnvACE achieves the second-best average, substantially ahead of several environment-scaling baselines. Scaling from 1.7B to 8B improves BFCL V4 average by a large margin and widens the advantage over standard reinforcement learning. Using EnvACE for world rehearsal at test time boosts overall scores beyond non-rehearsal baselines, and the trained EnvACE policy outperforms rehearsing with the base model.

On FinMCP-Bench, EnvACE-8B achieves the highest TF1 score by combining the best tool precision with moderate recall, striking a stronger balance than the recall-leading EnvScaler-8B. The external-simulator baseline (Simulator-8B) yields dramatically lower scores, confirming the advantage of world rehearsal over simulation-only approaches. EnvACE-8B attains the top TF1 (46.78%), driven by a precision of 54.04% that is more than 14 percentage points above the next best precision. EnvScaler-8B leads in tool recall (49.35%) but its TF1 lags behind EnvACE-8B by 3.10 points due to a much lower precision. Simulator-8B records the lowest TF1 (15.95%) and recall (11.36%), underscoring the limitations of relying solely on an external simulator.

Test-time scaling via world rehearsal boosts agent performance consistently across domains when the rehearsal policy is the rehearsal-trained model (EnvACE). Parallel execution with EnvACE achieves the best overall score, while substituting the base model for rehearsal yields negligible gains or even regresses below the no-rehearsal baseline. The benefits come from internalized environment–response knowledge rather than from added inference compute alone. Parallel world rehearsal with EnvACE lifts the overall score from 36.7% (no rehearsal) to 40.9%, setting the highest result. When the base model is used for rehearsal in sequential mode, overall performance drops to 34.9%, underperforming the no-rehearsal baseline, whereas EnvACE rehearsal improves it to 38.5%.

EnvACE is evaluated across a range of heterogeneous agent benchmarks and reliably surpasses environment-scaling baselines and even larger foundation models. The method benefits from model scaling and from test-time world rehearsal, which substantially lifts performance only when using the rehearsal-trained EnvACE model rather than the base model or an external simulator. The results confirm that internalized environment–response knowledge is key to agent robustness, with EnvACE striking an effective balance between precision and recall where other approaches over- or under-prioritize one side.