One-sentence Summary

Researchers from Renmin University of China and IQuest Research propose a unified black-box RL framework that combines sandbox-based concurrent rollouts, a serving proxy, and prefix-tree reconstruction to adapt critic-based PPO and critic-free GRPO for complex agent harnesses; using Qwen3-30A3B, the framework improves $\text{Pass@1}$Pass@1 on ClawGym-Bench by 9.98 and 14.81 points through OpenClaw and Claude Code, respectively, while remaining stable over 200-400 optimization steps and yielding further gains on JobBench and OfficeQA.

Key Contributions

• The paper introduces a unified black-box reinforcement learning framework that treats complex deployment harnesses as opaque rollout engines and decouples policy optimization from harness execution through sandbox-based execution isolation and a serving proxy at the model boundary.
• The framework reconstructs captured model calls into prefix trees, enabling recovery of multi-turn and forked trajectories, and adapts both critic-based PPO and critic-free GRPO over this structure while maintaining training-inference consistency.
• The framework supports mix-harness training for joint optimization of a single model across heterogeneous harnesses; with Qwen3-30A3B, black-box RL improves Pass@1 by 9.98 and 14.81 points on ClawGym-Bench via OpenClaw and Claude Code, by 11.71 and 17.28 points on PinchBench, and yields consistent gains on JobBench and OfficeQA.

Introduction

Agent harnesses have become the central runtime layer for autonomous agents, coordinating how large language models interact with tools, environments, and long-horizon tasks in systems such as Claude Code and OpenClaw. While these harnesses improve agent performance, their value depends on whether the underlying model has been trained to use them effectively, and their opaque internal control flow makes standard reinforcement learning difficult. Prior work lacks stable, scalable black-box RL methods for optimizing general agents through such complex harnesses. The authors introduce a unified black-box RL framework that treats each harness as an unmodified rollout engine, isolates task executions in sandboxes, recovers multi-turn trajectories through a serving proxy and prefix-tree structure, and supports PPO and GRPO optimization. The framework also enables mix-harness training across heterogeneous harnesses, and the authors validate it on OpenClaw and Claude Code with consistent benchmark improvements.

Method

The authors propose a unified black-box reinforcement learning framework that optimizes a trainable model through an unmodified and opaque agent harness, decoupling policy optimization from harness execution. Modern general-purpose agents are deployed through mature harnesses that provide rich execution abstractions, integrating tool orchestration, context management, and failure recovery. Under this paradigm, the interaction trajectory is determined jointly by the model and the harness.

To enable scalable black-box rollouts, the authors establish a dedicated infrastructure. For each task, a task-specific environment is initialized and the selected harness is launched inside a temporary sandbox, providing an isolated workspace and runtime dependencies. Policy optimization is decoupled from harness execution: the training engine handles optimization while the inference engine serves the current policy. A serving proxy is placed at the model-serving boundary to intercept every model request. This proxy invokes the rollout policy, returns the response, and records exact input tokens, generated tokens, rollout log-probabilities, and task metadata without instrumenting the internal harness logic.

As shown in the figure below:

Upon rollout completion, a verifier evaluates the final workspace state to produce a rollout-level reward. The captured model-call records and reward are passed to the training pipeline to reconstruct trainable multi-turn trajectories.

Because model calls captured during a black-box rollout are fragmented, forked, and potentially redundant, the authors organize them into a rollout-level prefix tree. This tree is rooted at the initial task prompt, and each call is attached to the existing node whose accumulated history forms the longest prefix of the input context. This structure reconstructs shared interaction histories and recovers intervening non-model content introduced by the harness.

Not all leaves in the prefix tree correspond to the completion of the main task. The authors apply filtering to retain suitable trajectories. Dead leaves resulting from retries or regenerated responses are discarded by retaining only the leaf with the longest valid continuation within each interaction segment. Rollouts that over-branch into an excessive number of leaves, indicating repeated or failed generation, are discarded entirely. Additionally, auxiliary trajectories from subagents or context compaction are excluded to prevent ambiguous credit assignment and noisy optimization signals.

Once the prefix tree is constructed and filtered, the authors optimize the policy over the recovered multi-trajectory structure. All retained trajectories from a single rollout share the same terminal reward. For Group Relative Policy Optimization, the advantage is computed once per rollout and assigned to all trainable token nodes, with shared prefixes counted only once. For Proximal Policy Optimization, a simplified variant is adopted where trajectories within the same rollout are treated independently with no temporal discounting, degenerating the Generalized Advantage Estimation to $\hat{A}_t = R_i - V_\phi(s_t)$A^t​=Ri​−Vϕ​(st​).

To maintain training-inference consistency, the authors adopt a black-box token-in-token-out discipline. The tokens generated by the inference engine are grafted directly onto the prefix tree as the sole source of training data. The structured text decoded for the harness is never encoded back into the trained trajectory, ensuring the sequence handed to the training engine is identical to the one sampled by the policy. Furthermore, to mitigate off-policy bias caused by numerical differences between the inference and training engines, a token-level importance-sampling ratio is applied to scale the loss of each training token:

Finally, to prevent the policy from specializing to a single harness, the authors introduce mix-harness training. This approach jointly optimizes a shared policy using rollouts from multiple heterogeneous harnesses within the same training run. Rollouts of the same task under different harnesses are randomly mixed in each batch, but their advantages are normalized within separate task-harness groups to prevent harness-dependent interaction patterns from distorting relative advantage estimation.

Experiment

The experiments evaluate a black-box reinforcement learning framework by training Qwen3 models under OpenClaw and Claude Code harnesses and measuring Pass@1 on ClawGym-Bench and PinchBench, with additional tests covering training dynamics, mixed-harness optimization, more challenging JobBench and OfficeQA tasks, cold-start initialization, and comparison with white-box AgentLoop RL. Black-box RL consistently improves over initial policies across heterogeneous harnesses and model scales, and both PPO and GRPO remain stable with broadly comparable results. Mixing rollouts from multiple harnesses introduces no evident instability or performance degradation, and the same unified pipeline extends to more demanding task distributions without task-specific modifications. Cold-start initialization improves stability and final performance but is not required, while white-box AgentLoop RL transfers only partially to an unseen harness and underperforms harness-matched black-box training.

The results compare Pass@1 performance on PinchBench and ClawGym-Bench across several Qwen3 variants and ClawGym-8B. Larger Qwen3 variants generally achieve higher average scores, with Qwen3-235A23B leading most ClawGym-Bench categories and reporting the strongest average. ClawGym-8B excels on PinchBench and software development, while white-box agentloop training shows large in-loop gains but only partial transfer to the OpenClaw harness. ClawGym-8B achieves the top PinchBench score by a clear margin and leads the software development category, but it trails Qwen3-235A23B in five of six ClawGym-Bench categories. Qwen3-235A23B reports the highest average score among the listed models and outperforms ClawGym-8B across most ClawGym-Bench categories. White-box agentloop RL improves over its Qwen3-30A3B initialization in-loop, yet under OpenClaw it still falls short of direct black-box training under the same evaluation harness.

White-box AgentLoop RL provides strong gains when measured in its own training loop, outperforming the starting model and the black-box RL model across all task categories. The white-box trained model also generalizes partially to the external OpenClaw harness, improving over its initialization but not reaching the performance of a model trained directly for OpenClaw. Under the white-box AgentLoop harness, WhiteBox-30A3B achieves the highest average and task-level performance among the evaluated models. WhiteBox-30A3B outperforms both the Qwen3-30A3B initialization and the black-box-trained ClawII-OC-30A3B under the white-box evaluation harness. Under the OpenClaw harness, WhiteBox-30A3B improves over its initialization in most task categories but still trails ClawII-OC-30A3B, indicating only partial white-to-black transfer.

The experiments evaluate Pass@1 performance on PinchBench and ClawGym-Bench across several Qwen3 variants and ClawGym-8B, and separately assess white-box AgentLoop RL under its in-loop harness and the external OpenClaw harness. Larger Qwen3 models generally score higher on average, with Qwen3-235A23B leading most ClawGym-Bench categories, while ClawGym-8B achieves the best PinchBench result and leads software development but trails in most other categories. White-box agentloop training produces strong in-loop gains over its Qwen3-30A3B initialization and the black-box model, but this advantage only partially transfers to OpenClaw, where it improves over initialization yet remains below the model trained directly for that harness.