One-sentence Summary

Researchers from Shanghai Jiao Tong University, Xiaohongshu Inc., and Southeast University propose Harness-R1, the first method to learn failure-conditioned editing of executable runtime harnesses via online reinforcement learning, where a $9\text{B}$9B harness engineer converts agent failures into validated patches and is trained with group-relative policy optimization after cold-start supervised fine-tuning, raising vanilla Qwen3.5-9B success from $44.3\%$44.3% to $53.6\%$53.6% across WebShop, ALFWorld, and DBBench and further boosting fine-tuned targets by $5.0$5.0 points, thereby enabling co-evolution of the harness engineer and the target agent.

Key Contributions

• Harness-R1 formulates failure-conditioned, lifecycle-wide editing of an executable agent harness as an online reinforcement learning problem for a dedicated harness engineer while keeping the target agent frozen.
• A separate 9B harness engineer is post-trained with group-relative policy optimization using outcome rewards from same-batch reruns of the frozen target agent, so that edits are optimized for realized task success rather than proposed by a fixed editor.
• Across WebShop, ALFWorld, and DBBench, Harness-R1 raises the vanilla Qwen3.5-9B success rate from 44.3% to 53.6%, further improves a fine-tuned target from 59.2% to 64.2%, and transfers positively to twenty unseen target models and 1,270 held-out tasks.

Introduction

Large language model agents rely on an executable runtime harness that includes prompts, memory, tools, and control logic to interact with environments, yet these agents often produce systematic failures after deployment. Prior work attempts to optimize harness components through fixed prompt-based editors or search-based pipelines, but these approaches do not train the editor itself, limiting their reliability and adaptability. The authors introduce Harness-R1, a training paradigm that post-trains a dedicated harness engineer with online reinforcement learning while keeping the target agent frozen. Conditioned on batches of target-agent failures, the engineer generates validated executable runtime patches, and the realized performance gain from rerunning tasks rewards only the engineer, making harness editing a learned capability that complements direct model fine-tuning.

Method

The authors introduce Harness-R1, an online, outcome-grounded framework designed to post-train a dedicated engineer model that improves the executable runtime surrounding a frozen target agent. Rather than updating the agent's weights, the framework formulates harness editing as a batch-conditioned learning problem where each modification is evaluated by rerunning the same tasks.

Let $A$A denote a frozen target agent and $B = \{ x _ { i } \} _ { i = 1 } ^ { n }$B={xi​}i=1n​ be a batch of $n$n tasks in environment $E$E. The component adapted is the base runtime, which assembles context, forwards actions, and relays feedback. Running the unmodified agent yields baseline trajectories and rewards. A deterministic extractor retains failed episodes and compacts their constraints, excerpts, outcomes, and state into a failure packet $s _ { B }$sB​. The engineer $H _ { \theta }$Hθ​ reads this packet and generates a batch-conditioned executable overlay $P$P.

As shown in the figure below:

The overlay $P$P wraps the execution loop as executable hooks at four lifecycle points, leaving the agent's weights untouched:

1. Episode initialization sets up the starting context and episode state.
2. Pre-decision augments the context with retrieved guidance and interface constraints before the agent decides.
3. Pre-action serves as a runtime guardrail that may canonicalize, rewrite, or veto the proposed action before it reaches the environment.
4. Post-feedback inspects the returned observation and triggers recovery when the trajectory stalls.

These hooks touch only the inputs and outputs surrounding the frozen policy. After validation, the overlay is installed and the frozen target reruns every task in $B$B. The full-batch performance difference and engineer reward are defined as:

$$\begin{array}{l} \Delta_ {B} (P) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(R _ {i} ^ {P} - R _ {i} ^ {0}\right), \\ r (B, P) = \left\{ \begin{array}{l l} \Delta_ {B} (P), & \text {if valid and complete}, \\ 0, & \text {otherwise}. \end{array} \right. \end{array}$$ΔB​(P)=n1​∑i=1n​(RiP​−Ri0​),r(B,P)={ΔB​(P),0,​if valid and complete,otherwise.​​

Harness-R1 learns the editing policy in two stages. First, cold-start supervised fine-tuning initializes a prior over valid, executable edits. A strong teacher proposes serialized editing responses $y _ { j } ^ { T }$yjT​ from the compact failure packets $s _ { j }$sj​. The resulting dataset $\mathcal { D } _ { \mathrm { S F T } } = \{ ( s _ { j }, y _ { j } ^ { T } ) \} _ { j = 1 } ^ { M }$DSFT​={(sj​,yjT​)}j=1M​ initializes the engineer by teacher-forced next-token prediction:

$$\begin{array}{c} \mathcal {L} _ {\mathrm{SFT}} (\theta) = - \frac {1}{\sum_ {j = 1} ^ {M} | y _ {j} ^ {T} |} \sum_ {j = 1} ^ {M} \sum_ {t = 1} ^ {| y _ {j} ^ {T} |} \\ \log H _ {\theta} \big (y _ {j, t} ^ {T} \mid s _ {j}, y _ {j, < t} ^ {T} \big). \end{array}$$LSFT​(θ)=−∑j=1M​∣yjT​∣1​∑j=1M​∑t=1∣yjT​∣​logHθ​(yj,tT​∣sj​,yj,<tT​).​

Second, outcome-grounded GRPO optimizes the realized task utility of patches applied to the frozen target. Starting from the supervised policy, the authors perform online GRPO and sample $K = 8$K=8 candidate patches from the current policy for each failure packet. Each candidate is parsed and validated into a patch, installed independently, and evaluated by rerunning the frozen target on the same full task batch. Invalid, no-op, or incomplete evaluations receive zero reward. For rewards $r _ { k } = r ( B, P _ { k } )$rk​=r(B,Pk​), let $\mu _ { B }$μB​ and $\sigma _ { B }$σB​ be the empirical mean and standard deviation within the eight candidates generated from the same packet. The rewards are normalized into advantages:

$$\widehat {A} _ {k} = \frac {r _ {k} - \mu_ {B}}{\sigma_ {B}}.$$Ak​=σB​rk​−μB​​.

Let $y _ { k } = ( y _ { k, 1 }, \ldots, y _ { k, T _ { k } } )$yk​=(yk,1​,…,yk,Tk​​) be the engineer response parsed into $P _ { k }$Pk​, and let $\rho _ { k, t } ( \theta ) = H _ { \theta } ( y _ { k, t } \mid s _ { B }, y _ { k, < t } ) / H _ { \theta _ { \mathrm { o l d } } } ( y _ { k, t } \mid s _ { B }, y _ { k, < t } )$ρk,t​(θ)=Hθ​(yk,t​∣sB​,yk,<t​)/Hθold​​(yk,t​∣sB​,yk,<t​). The sequence-level advantage is shared by all response tokens, and the engineer maximizes the token averaged clipped surrogate:

$$\begin{array}{c} g _ {k, t} (\theta) = \min \Bigl \{\rho_ {k, t} \widehat {A} _ {k}, \\ \operatorname{clip} (\rho_ {k, t}, 1 - \epsilon_ {\ell}, 1 + \epsilon_ {h}) \widehat {A} _ {k} \Bigr \}, \\ \mathcal {J} (\theta) = \mathbb {E} \left[ \frac {1}{K} \sum_ {k = 1} ^ {K} \frac {1}{T _ {k}} \sum_ {t = 1} ^ {T _ {k}} w _ {k, t} g _ {k, t} (\theta) \right]. \end{array}$$gk,t​(θ)=min{ρk,t​Ak​,clip(ρk,t​,1−ϵℓ​,1+ϵh​)Ak​},J(θ)=E[K1​∑k=1K​Tk​1​∑t=1Tk​​wk,t​gk,t​(θ)].​

Here $w _ { k, t } = \mathrm { c l i p } ( \ell _ { k, t } ^ { \mathrm { t r } } - \ell _ { k, t } ^ { \mathrm { r o } }, 0, 2 )$wk,t​=clip(ℓk,ttr​−ℓk,tro​,0,2) is the truncated importance weight. Only the engineer parameters $\theta$θ are updated, and the outer loop iterates over update bundles until the training budget is exhausted. Base trajectories, rewards, and failure packets are cached before optimization, while online evaluation reruns only the patched target for candidates sampled from the current engineer.

Experiment

The experiments evaluate Harness-R1, an outcome-trained harness editor, across three interactive environments that stress complementary agent failures in web navigation, embodied tasks, and database interaction. The outcome-trained editor consistently improves the frozen target agent, outperforms fixed prompt-based strategies and strong frontier models, and further boosts a fine-tuned target, demonstrating co-evolution with the agent. The learned editing policy generalizes to unseen target agents and held-out tasks where frontier engineers fail, and ablation reveals that pre-action mediation and post-feedback recovery are the most impactful intervention points.

Outcome-trained harness editing (Harness-R1) consistently improves the target agent's success across WebShop, ALFWorld, and DBBench, outperforming prompt-based strategies and frontier models. The engineer co-evolves with the target, further boosting a supervised fine-tuned agent, and ablation shows that pre-action mediation and post-feedback recovery are the most influential intervention points, with environment-specific importance. Harness-R1 raises the equal-weight average success from the baseline of 44.3% to 53.6%, a gain of 9.3 percentage points. Among prompt-based methods, ReAct improves the average by 3.2 points, while Self-Refine degrades it by 2.5 points relative to the default harness. The strongest frontier model, GLM-5.2, achieves a 48.8% average, falling short of Harness-R1 by 4.8 points. After the target agent is fine-tuned via supervised learning, Harness-R1 further lifts success by 5.0 points, showing the engineer can co-evolve with a stronger actor. Disabling pre-action mediation or post-feedback recovery reduces success by roughly 3 to 4 points each, whereas removing episode-start or pre-decision changes has negligible impact.

Outcome-trained harness editing (Harness-R1) consistently improves agent success across WebShop, ALFWorld, and DBBench, surpassing prompt-based methods and the strongest frontier model. The harness co-evolves with a supervised fine-tuned agent to yield further gains, and ablation shows that pre-action mediation and post-feedback recovery are the most influential intervention points, while episode-start and pre-decision changes have negligible impact.