One-sentence Summary

After systematically decomposing and evaluating 30 discovery harnesses across 12 model–problem pairs and finding no universally superior harness, researchers from UC Berkeley, MIT, and the MIT-IBM Watson AI Lab propose an adaptive-allocation method that leverages early progress to prune weak runs and reallocate compute, outperforming both fixed harnesses and ensembles.

Key Contributions

• The paper systematically decomposes the OpenEvolve and TTT-Discover discovery harnesses into their constituent components and evaluates 30 budget-matched harness variants across 12 model–problem pairs using over 3.1 million LLM rollouts and repeated-trial statistical analysis.
• The study demonstrates that no fixed discovery harness generalizes reliably across models and problems; harness choice is better treated as a model- and problem-dependent hyperparameter, with simpler alternatives often outperforming OpenEvolve-style configurations.
• The work presents an adaptive harness ensemble that uses early-run progress to prune underperforming harnesses and reallocate compute to stronger survivors, achieving average final performance gains over both a single fixed harness and a non-adaptive ensemble.

Introduction

The authors investigate LLM-guided autonomous discovery systems that iteratively generate, evaluate, and improve candidate solutions. In these systems, a harness controls archive construction, parent selection, exploration, and budget allocation, but prior work evaluates composite harnesses with only a few trials, making it impossible to separate meaningful design choices from run-to-run variance. The authors conduct a large-scale, statistically controlled evaluation of 30 budget-matched harnesses across 3.1 million rollouts and 12 model–problem pairs, finding that no single fixed harness reliably transfers across settings. They then show that treating harness choice as an online, problem-dependent hyperparameter—with partial-run feedback used to prune and reallocate compute—improves final performance over committing to a single harness upfront.

Method

The authors develop a framework for program discovery that unifies two popular search harnesses and extends them with an online budget allocation strategy. The method begins with a greedy sequential best-of-N baseline, which at each iteration selects the single highest-scoring program from the history $\mathcal{H}_t$Ht​ as the parent:

This deterministic top-1 selection, $p_t \sim \mathcal{E}_t^1$pt​∼Et1​, generates N candidate children that are evaluated and appended to the history. The total rollout budget is fixed at $B = N T$B=NT across iterations.

Moving from this baseline to the OpenEvolve harness involves four progressive relaxations: replacing the top-1 archive with a top-K archive $\mathcal{E}_t^K$EtK​, introducing epsilon-greedy exploration by occasionally sampling from the full history $\mathcal{H}_t$Ht​, shifting the budget from breadth to depth by decreasing N and increasing T, and finally adding inspiration sampling, MAP-Elites diversity maintenance, and multi-island evolution with crossovers. To derive the TTT-Discover harness, the scoring function is transformed into a value estimate of the subtree rooted at a program, an Upper Confidence Bound (UCT) exploration bonus is added based on visitation counts, and the bonus is modified to a PUCT rule with a prior estimate. TTT-Discover also samples multiple parents per time step rather than a single parent.

Because the optimal fixed harness varies across model–problem pairs, the authors introduce an online allocation policy that dynamically selects among harnesses using intermediate feedback. The adaptive harness ensemble starts multiple harness configurations, advances them to one or more checkpoints (e.g., 25%, 50%, 75% of a full run), ranks the partial runs by the best evaluator score observed so far, prunes weaker runs, and spends the remaining compute on the survivors. The total budget is fixed to $B_e = 5$Be​=5 full-run equivalents, matching the standard best-of-five evaluation. A single-stage policy satisfies the budget constraint

where $m$m partial runs are advanced to checkpoint $q$q, and $s$s survivors are completed. Multi-stage pruning uses several checkpoints $0 = q_0 < q_1 < \dots < q_L = 1$0=q0​<q1​<⋯<qL​=1, with $m_\ell$mℓ​ active configurations at stage $\ell$ℓ, leading to

The strongest policies begin with a broad portfolio at early checkpoints and progressively concentrate the budget as more informative feedback becomes available. For example, a three-stage 12→5→2→1 schedule prunes at 25%, 50%, and 75% of a full run, outperforming both fixed-harness commitment and an unpruned harness ensemble under the same compute budget.

Experiment

The evaluation compares search harnesses across four LLMs and three mathematical discovery tasks under a fixed rollout budget. Pair-level and cross-pair significance tests reveal that no single fixed harness consistently outperforms the simple Sequential BoN baseline, and the strongest observed harness varies across model–problem pairs. Early-run performance is shown to be predictive of final outcomes, and an adaptive online allocation strategy that starts many harnesses and prunes based on intermediate feedback yields higher average performance than both single-harness commitment and an unpruned harness portfolio.

Under a fixed compute budget of five full-run equivalents, adaptive online harness allocation policies that start multiple configurations, evaluate partial progress, and prune weaker runs consistently outperform fixed strategies. The strongest adaptive schedule, a three-stage pruning approach, raised the average final score from 84.35% to 85.75% by beginning with a broad portfolio and progressively concentrating resources on the most promising candidates. This demonstrates that early partial-run feedback is effective for directing compute toward better solutions. Every adaptive pruning schedule surpassed the single-harness baseline, the unpruned harness portfolio, and the Sequential BoN reference in average final score. The top-performing policy, which prunes at 25%, 50%, and 75% of a full run, achieved the highest average score of 85.75% and outperformed the unpruned portfolio on 11 of 12 model–problem pairs.

Under a fixed compute budget of five full-run equivalents, adaptive online allocation policies that launch multiple configurations, assess partial progress, and prune weaker runs consistently outperform static strategies. A three-stage pruning schedule that starts with a broad portfolio and progressively concentrates resources on the most promising candidates raised the average final score, demonstrating that early partial-run feedback effectively directs compute toward better solutions. All adaptive pruning schedules surpassed the single-harness baseline, the unpruned harness portfolio, and the Sequential BoN reference, with the best policy (pruning at 25%, 50%, and 75% of a full run) outperforming the unpruned portfolio on 11 of 12 model–problem pairs.