One-sentence Summary

Researchers from the University of Notre Dame, LMU Munich, and the University of Southern California propose MemoHarness, an adaptive agent harness optimization framework that decomposes the harness into six editable control dimensions, learns from its own executions via a dual-layer experience bank of per-case diagnoses and distilled global patterns, and adapts the harness to each test case using retrieved experience without test-time labels or search, demonstrating improvements across shell-agent, code-generation, and analytical-reasoning benchmarks.

Key Contributions

• The framework decomposes the agent harness into six editable control surfaces and uses a dual-layer experience bank so that search accumulates reusable diagnostic knowledge instead of only scalar scores.
• A test-time adaptation mechanism adapts the search-derived global harness to each new case by retrieving past successes, failures, and global patterns, without test-time feedback, gradient updates, or extra search rounds.
• Across shell-agent, code-generation, and analytical-reasoning benchmarks, MemoHarness improves task success over fixed-harness baselines, shows selective positive transfer to unseen suites and held-out base models, and remains cost-competitive when retrieved context is cacheable.

Introduction

The performance of LLM-based agents depends heavily on the surrounding control layer, or "agent harness," which governs context construction, tool access, inference orchestration, memory, and output handling. Harness design can dramatically shift task success rates even when the base model and tools remain fixed. Prior work largely focuses on optimizing isolated components such as prompts or static workflows, and existing harness-level approaches produce only a single, fixed configuration that cannot adapt at test time without additional labels or search runs. The authors address these limitations by introducing MemoHarness, a framework that learns from past executions to jointly optimize six harness dimensions and adapt the harness to each new case using retrieved experience, all without test-time feedback.

Method

The authors propose MemoHarness, a framework for adaptive harness optimization that learns a global base harness during a training-time search phase and applies case-specific adaptations at test time. As shown in the figure below, the overall pipeline decomposes into these two distinct phases.

To enable structured diagnosis and repair, the authors decompose the harness into a six-dimensional space rather than treating it as a monolithic prompt. A harness configuration is defined as an element of the product space $W \in \mathcal{W} = \mathcal{W}^{(1)} \times \dots \times \mathcal{W}^{(6)}$W∈W=W(1)×⋯×W(6), where each component controls a distinct functional stage of the inference pipeline, such as context assembly, tool interfaces, generation policy, orchestration, memory management, and output handling. This decomposition turns harness search into structured editing over separable control surfaces.

During the training-time search phase, the system operates over a labeled search set $\mathcal{D}_{\text{search}} = \{(u_i, \phi_i, y_i^\star)\}_{i=1}^n$Dsearch​={(ui​,ϕi​,yi⋆​)}i=1n​. The process begins with a minimal harness $W_0$W0​ and proceeds iteratively. At each iteration $t$t, a controller forms a query based on the current harness and the accumulated experience bank, retrieving a bounded evidence slice to propose the next configuration $W_t$Wt​. The proposed harness is executed on every search case to collect execution trajectories $\tau_i(W_t)$τi​(Wt​), compute task rewards $r_i(W_t)$ri​(Wt​), and measure execution costs $c_i(W_t) = n_i^{\text{tok}}(W_t)$ci​(Wt​)=nitok​(Wt​).

To guide the search effectively, the authors maintain a dual-layer experience bank $\mathcal{B}_t = (\mathcal{E}_t, \mathcal{G}_t)$Bt​=(Et​,Gt​). The first layer, $\mathcal{E}_t$Et​, stores per-case execution entries that record configuration deltas, trajectories, rewards, costs, and diagnostic signals indicating success or failure dimensions. The second layer, $\mathcal{G}_t$Gt​, contains distilled global patterns extracted periodically from failure clusters to summarize recurring phenomena and the expected effects of targeted harness changes.

The selection of the final global harness $W^\star$W⋆ follows a correctness-first principle. The authors compute the mean task reward $\bar{r}_t$rˉt​ and mean cost $\bar{c}_t$cˉt​ for each candidate. The optimal harness is selected via lexicographic ordering:

This ensures that primary correctness is optimized first, while lower token usage serves strictly as a secondary tiebreaker.

In the test-time case adaptation phase, the system operates on unlabeled evaluation cases without iterative feedback. For a given test case $x = (u, \phi)$x=(u,ϕ), the controller retrieves relevant global patterns and feature-matched historical slices from the frozen experience bank $\mathcal{B}_T$BT​. It specifically identifies the most similar successful and failed training examples by computing cosine similarity between instruction representations:

Using the top-$K$K similar successful and failed entries alongside the retrieved global patterns, the controller constructs a test-time evidence slice. Conditioned on this evidence, the controller emits a case-specific harness $W(x) = \Pi_{\text{test}}(W^\star, x, \mathcal{S}_{\text{test}}(x))$W(x)=Πtest​(W⋆,x,Stest​(x)), which is then executed to produce the final prediction. This design allows simple cases to remain lightweight while enabling complex cases to invoke richer orchestration only when warranted by the retrieved experience.

Experiment

The evaluation spans three task families (Terminal-Bench, LiveCodeBench, FinanceAgent) and compares MemoHarness against strong baselines, showing that adaptive harness optimization yields consistent gains, particularly on longer-horizon agentic workloads where search discovers beneficial edits over iterations. Learned harnesses transfer positively to unseen evaluation suites and across diverse base models, with the broadest lift coming from the terminal-oriented harness, indicating that some control decisions generalize beyond the source task. The approach remains cost-effective under caching assumptions, as the majority of retrieved context is reusable.

The MemoHarness harness space decomposes inference into six functional stages, from context assembly to output processing, enabling structured editing over separable control surfaces. Learned harnesses transfer across tasks and models, with the strongest gains when the source search task is long-horizon and toolcentric, and the approach remains cost-effective by caching most retrieved context. Decomposing the harness into six dimensions allows targeted diagnosis and repair at each stage of inference. A harness learned on Terminal-Bench improves MMMLU, StrongReject, and SWE-Bench Pro, while saturated benchmarks show no movement. Transferring a harness from GPT-5.3-Codex to six other models yields a mean improvement of +0.098, with every model gaining over its base setting. Strongest transfer occurs when the source search task is long-horizon and toolcentric, suggesting some control decisions generalize across evaluation suites. MemoHarness uses more input tokens due to experience retrieval, but most are cached, resulting in lower reported cost than Codex and Claude Code while achieving higher task success.

MemoHarness harnesses learned from tool-centric search tasks improve cross-dataset generalization over a shared Codex baseline, with the Terminal-Bench-derived harness delivering the broadest gains. Gains appear primarily on unsaturated benchmarks such as MMMLU, StrongReject, and SWE-Bench Pro, while already-saturated suites remain unchanged. Transfer strength varies by search source, with long-horizon tool-centric tasks yielding more portable control decisions. The Terminal-Bench-derived MemoHarness provides the widest lift, improving MMMLU, StrongReject, and SWE-Bench Pro over the shared Codex baseline. The LiveCodeBench-derived harness improves MMMLU and StrongReject, while the FinanceAgent-derived harness mainly improves StrongReject and LawBench. Saturated benchmarks such as HumanEvalFix and Reasoning-Gym-Easy show no movement across all MemoHarness variants. LawBench results remain mixed, with only the FinanceAgent-derived harness showing a modest gain and the LiveCodeBench-derived harness slightly decreasing performance. The strongest cross-dataset transfer occurs when the search source is long-horizon and tool-centric, suggesting that learned control decisions can survive a change in evaluation suite.

A harness optimized via search with one model transfers positively to diverse unseen models on terminal tasks, with every tested model improving over its base configuration. The mean gain across six transfer models is +0.098, ranging from +0.038 to +0.233, while the source model retains its original improvement. All six transfer models improve over their base setting, with gains from +0.038 on GPT-4.1 to +0.233 on GLM-5. The source model GPT-5.3-Codex retains its +0.084 improvement, and the mean transfer gain across models is +0.098. The harness transfers without re-optimization, suggesting a portable execution policy for context gathering, tool invocation, and output finalization on terminal tasks. Smaller gains on stronger base models like GPT-4.1 indicate well-calibrated defaults may leave less room for harness-driven improvement.

MemoHarness achieves higher task success than Codex while incurring lower total cost under a caching-aware accounting protocol, primarily because most of its additional input tokens are served from cache. Terminus and OpenCode remain the cheapest options but deliver substantially lower accuracy. MemoHarness uses more raw input tokens than any other framework, but 13.32M of its 14.18M input tokens are cached, keeping non-cached input at only 0.86M. The reported cost for MemoHarness is $6.89, which is lower than Codex ($6.89,whichislowerthanCodex(10.28) and Claude Code ($9.51) despite MemoHarness achieving higher task success. OpenCode reports the lowest cost at$9.51)despiteMemoHarnessachievinghighertasksuccess.OpenCodereportsthelowestcostat2.34, driven by a high cache-hit ratio (5.07M cached of 5.48M input) and minimal non-cached tokens, but its accuracy is substantially lower. Terminus achieves the second-lowest cost at $6.68 with fewer total input tokens than MemoHarness, yet its accuracy remains substantially below MemoHarness.

MemoHarness decomposes inference into six functional stages, enabling structured editing over separable control surfaces. Experiments show that harnesses learned on long-horizon, tool-centric search tasks transfer effectively across diverse benchmarks and models, improving performance on unsaturated evaluations while leaving saturated ones unchanged. The approach also transfers across models without re-optimization, yielding consistent gains, and remains cost-effective by caching most of its retrieved context, achieving higher task success than Codex and Claude Code at lower reported cost.