One-sentence Summary

Columbia University, Dartmouth College, and Microsoft Research present OPENFORGE RL, an open-source framework that trains harness-based agents end-to-end by decoupling training and inference via a proxy recording model calls for standard RL and a Kubernetes orchestrator for remote rollouts, enabling scalable training in any environment, and validate it on diverse agents including tool/claw-based and multimodal GUI browser/computer-use agents, achieving 31.7 ($\text{pass}^3$pass3) and 55.9 ($\text{pass}@3$pass@3) on ClawEval, 33.7 on Qwen-ClawBench, 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager, outperforming open baselines of similar size and matching or surpassing larger models in GUI settings, while revealing that harness choice substantially affects learning difficulty and that RL improves agentic reliability in self-verification, tool coverage, and multi-step planning, though error recovery remains weak.

Key Contributions

• The paper introduces OPENFORGE RL, an open-source framework that enables end-to-end training of LLM- and VLM-based agents directly inside their real inference harnesses by using a lightweight proxy to record model calls as training data and a Kubernetes orchestrator to run rollouts in remote containers, decoupling training from harness inference and making any harness compatible with standard RL codebases.
• Using only hundreds to a few thousand tasks, the framework produces OpenForge-Claw and OpenForge-GUI models that surpass open models of similar size on nearly all tool-use and GUI benchmarks, with OpenForge-Claw reaching 31.7 pass3 on ClawEval and OpenForge-GUI reaching 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager, and in GUI settings matching or exceeding models several times larger.
• The paper analyzes how harness choice and RL shape agent behavior, finding that simpler, better-aligned harnesses are easier to learn, training gains transfer to unseen harnesses, and RL improves agentic reliability such as self-verification and tool coverage, while error recovery remains challenging even after RL.

Introduction

Modern AI agents rely heavily on inference harnesses—orchestration scaffolds that manage multi-turn interaction, tool calls, and external system connections—to perform effectively in complex environments such as software engineering, web browsing, and desktop control. However, training these harness-equipped agents end-to-end remains difficult for the open research community because existing RL frameworks cannot natively express the stateful, multi-process inference that harnesses require, and running harness rollouts at scale demands containerized environments that cannot be co-located on training nodes. These limitations force researchers to use simplified harness reimplementations for training, creating a mismatch between training and deployment. The authors introduce OPENFORGE RL, an open framework that decouples harness inference from training via a lightweight proxy and uses a Kubernetes orchestrator to launch remote rollout containers elastically on cloud providers, enabling any harness to be trained with standard RL codebases in its real deployment setting.

Dataset

The authors describe a synthetic data pipeline designed to address the scarcity of training tasks for non-coding environments, such as browser use, computer use, and daily tool use. This pipeline produces both supervised fine-tuning (SFT) and reinforcement learning (RL) tasks for harness-based agents.

Dataset composition and sources

• The pipeline generates tasks for data-scarce domains, including claw/daily tool use and computer use.
• Candidate instructions are grounded in realistic scenarios, drawn from the web/X API or a pool of reference assets and instructions.

Key details for each subset

• The pipeline does not produce predefined subsets; instead, it creates tasks on demand for a given target domain and a specified number of tasks.
• Each generated task includes an executable environment (defined by a custom Dockerfile) and a verifier script.
• The pipeline supports both Linux/CLI tasks and GUI/computer-use tasks, using tools like Xvfb for virtual displays and pre-installed harnesses such as OpenClaw and Codex.

How the paper uses the data

• The synthesized tasks support both SFT, via distillation from a stronger model’s rollouts, and RL training.
• The authors do not specify training split ratios in this section, but the pipeline is used to experiment with their framework across diverse environments.

Processing details

• The synthesis pipeline mimics human task curation through five parallel stages:
  1. Propose candidate instructions grounded in realistic scenarios.
  2. Prune low-quality and duplicate tasks.
  3. Build an executable environment and a verifier script for each task.
  4. Test the task by rolling out a separate open LLM/VLM.
  5. Refine the task by patching defects until it passes all checks.
• The test and refine stages validate each task end-to-end before it enters the dataset.
• The pipeline extends naturally to different environments because each is defined by a custom Dockerfile.

Method

The authors formulate completing tasks in complex, long-horizon environments as a Markov Decision Process. To address the challenges of training harness-based agents end-to-end, they introduce OPENFORGE RL, a plug-and-play rollout interface that connects popular reinforcement learning frameworks to distributed harness rollouts.

The framework addresses the mismatch between training stacks and deployment harnesses by decoupling the inference process from training. Given an inference server, OPENFORGE RL launches a Kubernetes orchestrator that creates, manages, and deletes rollout container pods on cloud providers. This allows for elastic scaling of concurrent rollouts without overloading training nodes. Simultaneously, a proxy server wraps the inference server and intercepts all generation requests issued by the remote sandboxes. When a rollout finishes, the proxy collects the terminal reward and the prompt-response pairs from the container. These are used to reconstruct training samples as a trajectory:

$$\tau = (s _ {0} ^ {\mathcal {H}}, a _ {0}, r _ {0}), (s _ {1} ^ {\mathcal {H}}, a _ {1}, r _ {1}), \ldots, (s _ {T} ^ {\mathcal {H}}, a _ {T}, r _ {T}); \quad r _ {t} = \gamma^ {T - t} \cdot r _ {T}$$τ=(s0H​,a0​,r0​),(s1H​,a1​,r1​),…,(sTH​,aT​,rT​);rt​=γT−t⋅rT​

typically with $\gamma = 1.0$γ=1.0. When optimizing with group-based algorithms, the advantage is computed by comparing average sample rewards across different trajectories in the same group.

To ensure robust training, the system implements specific strategies for rollout orchestration, asynchronous execution, and error handling. A wall-clock timeout is imposed on each rollout job to prevent unresponsive rollouts from blocking the collection of an entire training batch. If a job exceeds its timeout, it is terminated, and an error signal is returned so training continues with remaining rollouts. Furthermore, samples from trajectories that end in errors unrelated to the policy model are discarded to avoid injecting misleading training signals.

To support training across diverse environments beyond coding, such as browser and computer use, the authors also built a data synthesis pipeline.

This pipeline mimics human task curation. Given a target domain and a desired number of tasks, it spawns agents in parallel to propose candidate instructions grounded in realistic scenarios. It then prunes low-quality and duplicate tasks, builds an executable environment and a verifier script for each task, and tests the task by rolling out a separate model. Finally, it refines the task by patching defects until it passes all checks. This test and refine stage validates each task end-to-end before it enters the dataset. Because each environment is defined by a custom Dockerfile, the pipeline extends naturally from Linux/CLI tasks to GUI and computer-use tasks, supporting both supervised fine-tuning and reinforcement learning.

Experiment

Experiments demonstrate that OPENFORGE RL training across diverse harnesses and environments consistently improves agent performance, with reinforcement learning on top of supervised fine-tuning yielding substantial gains on both text-based tool-use and multimodal GUI benchmarks. Simpler harnesses prove easier to master, and training on multiple harnesses enhances generalization to unseen ones, while RL teaches agents to prefer specialized tools over generic ones and strengthens error recovery and self-verification capabilities.

The table reports the number of SFT trajectories and RL tasks used for training Claw and GUI agents. Browser-use GUI training data is the largest, with over 1,400 SFT trajectories and 900 RL tasks, while computer-use GUI data is the smallest. Claw agents are trained on multiple text-based harnesses, whereas each GUI agent uses a single harness. The browser-use GUI training set is substantially larger than the computer-use GUI set, with roughly twice as many SFT trajectories and over three times as many RL tasks. Claw agents use four different harnesses (ReACT, ZeroClaw, OpenClaw, Codex) for training, while GUI agents each rely on a single harness (Kimi-Agent for computer-use, MolmoWeb for browser-use).

State-of-the-art large language models set strong baselines on agentic benchmarks, with Claude Opus 4.6 leading on ClawEval pass³ and MCPAtlas, and tying with Gemini 3.1 Pro on ClawEval pass@3. The paper further shows that supervised fine-tuning followed by reinforcement learning (SFT+RL) on diverse harnesses yields substantial gains over these baselines, except on OpenClaw where improvements are moderate. Training on multiple harnesses improves generalization, and RL refines tool use by shifting from generic shell commands to specialized tools while strengthening error recovery and self-verification. Claude Opus 4.6 achieves the highest ClawEval pass³ and MCPAtlas pass@1 among evaluated models. Gemini 3.1 Pro matches Claude Opus 4.6 on ClawEval pass@3, and QwenClawBench scores are tightly grouped with Claude slightly ahead. SFT+RL training brings large gains over the base model on every harness except OpenClaw, where gains are only moderate. Training on multiple harnesses (ZeroClaw, OpenClaw, Codex) yields better performance across all evaluated harnesses than training on a single harness, with the largest gains on more complex harnesses. RL reduces reliance on generic shell calls and increases use of specialized tools, while improving error recovery and self-verification capabilities.

Leading vision-language models show a wide performance range on GUI-agent benchmarks, with proprietary systems dominating. Gemini 3.1 Pro achieves the highest OSWorld accuracy, while GPT 5.4 leads on OnlineMind2Web by a large margin, and Kimi K2.5 sets the top WebVoyager score. Open-weight alternatives like Qwen3-VL and OpenCUA-32B lag significantly, especially on the challenging OSWorld tasks. GPT 5.4 reaches 92.8 on OnlineMind2Web, far ahead of the next best model Claude Opus 4.6 at 84.0. On OSWorld, Gemini 3.1 Pro (76.2) narrowly outperforms GPT 5.4 (75.0) and Claude Opus 4.6 (72.7). The open-weight Qwen3-VL model scores only 38.1 on OSWorld, less than half the accuracy of the top proprietary models. Kimi K2.5 is the only model with a reported WebVoyager result above 70, achieving 74.3, while Qwen3-VL trails at 66.4.

Models achieve the highest performance on harnesses that allow direct custom tool integration, with ZeroClaw and ReACT* leading across all training stages. Fine-tuning via SFT and RL yields substantial improvements on most harnesses, but the gain on OpenClaw is modest while it incurs longer prompts and contexts. The best results come from combining SFT and RL, especially on harnesses with simpler, well-engineered tool interfaces. ZeroClaw and ReACT*, which support direct custom tool registration, enable the highest pass@1 scores after SFT+RL (48.5% and 45.1%, respectively), far ahead of OpenClaw (20.9%) and Codex (32.5%). Adding RL to SFT improves pass@1 on most harnesses substantially, but OpenClaw sees only a moderate increase from 16.7% to 20.9%, while requiring much longer contexts than other harnesses.

Training on a single harness (ZeroClaw) already transfers to unseen harnesses, yielding moderate gains over the base model on OpenClaw and Codex. Training on all three harnesses together produces the strongest results across every harness, with the largest improvements on the more complex OpenClaw and Codex evaluations, and even surpasses ZeroClaw-only training on ZeroClaw itself. A model trained only on ZeroClaw improves over the untrained base on unseen OpenClaw by 3.3 points and unseen Codex by 4.6 points, demonstrating cross-harness generalization. Joint training on ZeroClaw, OpenClaw, and Codex delivers the highest pass@1 scores across all harnesses, with gains of 16.0 on ZeroClaw, 9.5 on OpenClaw, and 20.3 on Codex relative to the base model. Multi-harness training lifts ZeroClaw performance to 48.5, exceeding the 46.0 achieved by training on ZeroClaw alone. The largest absolute improvements from multi-harness training occur on the more complex harnesses, with Codex gaining 20.3 points and OpenClaw gaining 9.5 points over the base.

This evaluation examines both text-based Claw agents and GUI agents, comparing state-of-the-art proprietary models against fine-tuned variants. Proprietary models like Claude Opus 4.6, Gemini 3.1 Pro, and GPT 5.4 set strong baselines, but supervised fine-tuning followed by reinforcement learning on diverse training harnesses yields substantial gains, particularly for harnesses with direct custom tool integration. Multi-harness training improves generalization, with joint training outperforming single-harness setups across all evaluations, and reinforcement learning refines tool use by shifting from generic shell commands to specialized tools while strengthening error recovery and self-verification. For GUI agents, proprietary vision-language models dominate benchmarks, with open-weight alternatives lagging significantly on challenging tasks like OSWorld.