One-sentence Summary

Researchers from the Allen Institute for AI and the University of Washington find that automatic harness evolution for LLM agents does not consistently outperform simple test-time scaling methods under matched feedback and inference budgets on Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6, and exhibits limited generalization to held-out tasks, calling for fairer evaluation protocols.

Key Contributions

• A controlled comparison of automatic harness evolution against test-time scaling baselines under matched feedback and inference budgets shows that evolution methods do not consistently outperform simpler scaling approaches on Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6.
• Evaluating evolved harnesses on held-out tasks from the same benchmark reveals limited generalization, questioning the practical utility of the discovered improvements.
• The study identifies a flawed evaluation protocol where search and final evaluation share the same benchmark, risking overfitting, and advocates for fairer evaluation protocols and benchmarks for automatic harness design.

Introduction

Large language model (LLM) agents rely on external harnesses (prompts, tools, memory, verification, and control logic) to interact with complex environments, and prior work shows that harness design can dramatically affect agent performance even when the underlying model is fixed. However, harness engineering remains mostly manual, motivating automatic harness evolution methods that search over harness configurations using benchmark feedback. The existing evaluation of these methods has two key limitations: it often uses the same public benchmark for both search and final measurement, which risks overfitting, and it lacks comparisons to test-time scaling baselines that allocate additional computation directly to evaluation tasks. The authors revisit the evaluation of automatic harness evolution, comparing it against parallel sampling and sequential refinement under a controlled budget that matches feedback and inference compute. Their findings show that harness evolution does not consistently outperform simple test-time scaling, and that gains observed on overlapping tasks may reflect adaptation to specific instances rather than generalizable harness improvements, underscoring the need for stricter evaluation protocols.

Experiment

The experiments compare automatic harness evolution with test-time scaling methods (parallel sampling, sequential refinement, harness scaling) on Terminal-Bench using three frontier models, under settings with and without unit test feedback, and on a held-out task split. Harness evolution does not consistently outperform simpler test-time discovery; its gains largely come from making multiple attempts rather than from genuinely improved harness design, and it fails to generalize to unseen tasks, instead overfitting to task-specific shortcuts. Qualitative analysis shows that while the meta agent produces rational edits, these rarely convert hard failures into successes, suggesting that harness evolution may only be beneficial when tasks are both difficult and strongly harness-dependent.

When unit tests are unavailable, test-time scaling via parallel sampling provides consistent gains, while automatic harness evolution underperforms the baseline and can harm stronger models. Self-generated feedback alone is too noisy to reliably guide harness revision, making parallel sampling a safer and more effective strategy in this setting. Parallel sampling improved average pass@1 across all three models and achieved the highest overall average, while harness evolution fell below the direct sampling baseline. Harness evolution caused a sharp drop of over 5 points on GPT-5.4, showing that iterative self-revision can degrade performance when correctness signals are absent.

When unit tests are available, test-time scaling methods substantially outperform direct sampling and automatic harness evolution. Parallel sampling and sequential refinement both achieve large gains, with sequential refinement reaching the highest pass@5 scores. Harness evolution improves only modestly over the baseline and remains well below the test-time scaling approaches. Parallel sampling delivers the highest pass@1 performance, raising average scores from 72.9 to 86.0 and matching its pass@5 accuracy. Sequential refinement achieves the best pass@5 results, reaching 93.3 on GPT-5.4, far exceeding harness evolution’s best pass@5 of 89.3.

When a harness is evolved on a set of training tasks and evaluated on unseen held-out tasks, it provides almost no benefit over the initial harness. The evolved harness yields a small improvement on one model and none on the other, averaging less than one point, indicating that the process overfits the training data and fails to generalize. The discovered modifications appear to be task-specific shortcuts rather than reusable design principles. On held-out tasks, Harness Evolution improved Claude Opus 4.6 by only 1.2 points and left GPT-5.4 unchanged, for an average gain of 0.6 points. The near-zero transfer performance suggests that the algorithm overfits training tasks, memorizing fixes instead of learning general scaffolding strategies.

The evaluation compares test-time scaling methods (parallel sampling and sequential refinement) to automatic harness evolution first without and then with unit tests, and finally on held-out tasks. In both settings, test-time scaling reliably improves performance, while harness evolution is harmful without tests, provides only modest gains with tests, and fails to generalize to unseen tasks due to overfitting.