One-sentence Summary

POSTECH, KAIST, et al. propose AutoSaddler, an automatic harness optimization framework that treats harness improvement as offline learning from agent execution traces, combining failure-trace diagnosis, structured code-based patch generation, and validation-based selection to substantially boost LLM agent robustness, achieving gains of 9.0, 9.6, and 10.0 percentage points on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0, respectively.

Key Contributions

• AutoSaddler is an automatic harness optimization framework that formulates harness improvement for LLM agents as an offline learning problem, combining in-depth failure diagnosis, structured patch generation that treats the harness as code, and generalization-aware update selection to produce durable harness improvements.
• On GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0, AutoSaddler improves agent performance by 9.0, 9.6, and 10.0 percentage points over base harnesses, respectively, and outperforms the strongest automated baselines by 7.4, 4.4, and 6.7 points.
• Ablation studies suggest that effective harness optimization benefits from deep debugging rather than shallow reflection, targeted modifications rather than unconstrained editing, and generalization-aware selection rather than trajectory-specific repair.

Introduction

Large language models exhibit jagged intelligence, performing well on some tasks while failing on closely related ones, which undermines their reliability in autonomous, multi-step agent applications. To mitigate this, developers build external harness layers (prompts, tools, middleware) around the model, but manually tuning these harnesses is slow, expensive, and hard to scale because the design space is vast and evaluating long-horizon trajectories is costly. The authors formulate automatic harness optimization as an offline learning problem and introduce AutoSaddler, a framework that iteratively refines the harness using failure signals from batches of training tasks. AutoSaddler combines in-depth diagnosis of failed trajectories, structured patching of the harness code, and generalization-aware selection to produce durable updates that improve agent performance across environments.

Method

The authors propose AutoSaddler, an iterative framework designed for the automatic optimization of LLM agent harnesses. They formulate this harness optimization as an offline learning problem, adopting a mini-batch training paradigm to manage the high cost of rollout-based evaluation. The optimization space is defined over three distinct classes of harness parameters, namely prompts, tools, and middleware, denoted as $\theta = (\theta_{\text{prompt}}, \theta_{\text{tool}}, \theta_{\text{middleware}})$θ=(θprompt​,θtool​,θmiddleware​). The primary objective is to maximize the expected task performance over a target task distribution within a predefined rollout budget $K$K.

Refer to the framework diagram for a comprehensive view of the iterative optimization loop.

As shown in the figure below, each iteration $n$n begins by evaluating the current harness $H_n$Hn​, parameterized by $\theta_n$θn​, on a mini-batch $B_n$Bn​ sampled from the training set. The workflow then transitions into the Diagnosis-Patch Session. During this phase, the execution traces from the mini-batch, encompassing both successful and failed runs, are analyzed by a Diagnosis-Patch Agent. To mitigate long-context challenges, the agent is provided with the harness codebase and structured guidance to progressively retrieve relevant trace details. Based on this evidence, the agent identifies suspected root causes and proposes a structured patch $\Delta \theta_n$Δθn​. Rather than allowing unconstrained edits, the patch space is strictly organized into three categories corresponding to the harness layers: Prompt, Tool, and Middleware. Furthermore, the authors divide these patch types into two higher-level groups: Capability Patches, which modify executable code or orchestration logic, and Steering Patches, which consist of textual edits. To optimize these effectively, AutoSaddler employs a Phased Patch Scheduling strategy, analogous to learning-rate scheduling, where optimization initiates with a Capability Patch phase before transitioning to a Steering Patch phase.

Following the generation of the updated harness $H_n' = H_n + \Delta \theta_n$Hn′​=Hn​+Δθn​, the system verifies the patch on the same mini-batch. If the patch yields a mini-batch improvement, it is further evaluated on the development set to estimate generalization. Regardless of the verification outcome, the workflow proceeds to the Reflection Session. Here, a Reflection Agent compares the pre-patch and post-patch traces, categorizing the outcomes into fixed, regressed, still-failing, and still-passing cases. Targeted self-reflection questions are utilized to elicit insights into the effectiveness of the patch, the addressed failure patterns, and any observed regressions. The extracted lessons, along with the patch description and evaluation metrics, are stored as node-level attributes in the EvoDAG.

The Evolution Session leverages the EvoDAG, a directed acyclic graph $\mathcal{G} = (V, E)$G=(V,E) that serves as the cumulative memory of the optimization process. Each node $v_n \in V$vn​∈V represents a previously explored harness annotated with its associated lessons and performance signals, while each directed edge $e \in E$e∈E represents the diff between a parent harness and its descendant. Instead of merely continuing from the most recent harness, the Evolution Agent consults the full EvoDAG to synthesize the next candidate harness $H_{n+1}$Hn+1​. By composing elements from any subset of previously explored harnesses guided by accumulated lessons, this merge operation functions similarly to evolutionary search, enabling the framework to escape local optima by recombining successful components across different lineages. Once the rollout budget is exhausted, AutoSaddler returns the candidate harness that achieved the highest empirical development-set score.

Experiment

AutoSaddler is evaluated on three diverse benchmarks: GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0, using base harnesses and compared against prompt-centric (GEPA) and harness-optimization (Meta-Harness) baselines. It consistently outperforms both the base harnesses and the strongest automated baselines, and on Terminal-Bench even surpasses a manually expert-tuned harness. Ablation studies demonstrate that its three core design principles (in-depth diagnosis, structured intervention, and generalization-aware selection) are essential: removing any of them substantially degrades performance, with generalization-aware selection proving most critical by preventing overfitting and reducing regressions on unseen scenarios.

The patch taxonomy organizes harness modifications into prompt, tool, and middleware categories. Each subtype is labeled as either Capability or Steering, distinguishing patches that extend or repair functional tooling from patches that guide agent behavior. This separation clarifies whether a patch changes what the agent can do or how it uses existing capabilities. Capability patches include adding new tools, modifying tool parameters, and fixing internal tool implementation, all of which extend or correct functional behavior. Steering patches include prompt rule additions or modifications, tool description fixes, and pre-tool-use hooks, which guide behavior without adding new functional capabilities.

AutoSaddler achieves the highest test-set Pass@1 on GAIA2, outperforming the default agent and all automated baselines. Ablation studies show that each of its three core design principles—in-depth diagnosis, structured intervention, and generalization-aware selection—contributes substantially to overall performance, with generalization-aware selection preventing the largest degradation when removed. AutoSaddler improves Pass@1 by 9.0 percentage points over the default agent (53.0% to 62.0%). It surpasses the strongest automated baseline, GEPA, by 7.4 percentage points (54.6% vs. 62.0%). Removing in-depth diagnosis drops Pass@1 from 62.0% to 57.8%. Removing structured intervention reduces Pass@1 from 62.0% to 56.9%. Ablating generalization-aware selection causes the largest performance drop, from 62.0% to 50.6%.

AutoSaddler discovers harnesses that consistently outperform both manual and automated baselines on SWE-Bench Pro and Terminal-Bench 2.0. On SWE-Bench Pro, it raises the average Pass@1 by 8.4 points over the SWE-agent manual harness and surpasses the best automated baseline GEPA by 6.2 points. On Terminal-Bench 2.0, it improves over the base Terminus 2 harness by 10.0 points and even beats the expert-tuned KIRA harness by 2.5 points. AutoSaddler achieves the highest average Pass@1 on SWE-Bench Pro, outperforming both the manual SWE-agent harness and the automated GEPA and Meta-Harness baselines. On Terminal-Bench 2.0, AutoSaddler surpasses the manually expert-tuned Terminus KIRA harness, demonstrating the effectiveness of automated harness optimization.

The patch taxonomy categorizes harness modifications into capability and steering patches, clarifying whether they extend tool functionality or guide agent behavior. AutoSaddler, an automated harness optimization method, consistently surpasses manual and automated baselines on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0. Ablation studies demonstrate that its three design principles of in-depth diagnosis, structured intervention, and generalization-aware selection are all essential, with generalization-aware selection being the most critical for maintaining performance.