Command Palette
Search for a command to run...
La découverte automatisée ne possède pas de harnais universellement supérieur
La découverte automatisée ne possède pas de harnais universellement supérieur
Akshat Gupta Jermaine Lei Alexander Lu Gopala Anumanchipalli Leshem Choshen
Résumé
Les systèmes de découverte autonomes tels qu'OpenEvolve et TTT-Discover sont souvent utilisés comme des harnais polyvalents. En pratique, il s'agit cependant de systèmes composites combinant plusieurs choix de conception relatifs aux archives, à la sélection des parents, à l'exploration et à l'allocation du budget en une recette unique. Les exécutions de découverte étant coûteuses et intrinsèquement stochastiques, les harnais existants sont souvent comparés à l'aide d'un nombre trop faible d'essais indépendants pour distinguer les améliorations méthodologiques clés de la variance inter-exécutions. Nous décomposons systématiquement la recherche évolutionnaire de type OpenEvolve et le harnais de recherche TTT-Discover en leurs composants constitutifs et évaluons systématiquement 30 harnais à budget équilibré sur 12 paires modèle–problème, en utilisant plus de 3,1 millions de déploiements de grands modèles de langage et une analyse statistique à essais répétés. Nos résultats montrent que les harnais de découverte souffrent d'un problème de généralisation : aucun harnais fixe n'est fiablement supérieur sur l'ensemble des paires modèle–problème évaluées, et les variantes d'OpenEvolve sous-performent généralement par rapport à des alternatives plus simples. Ainsi, le choix du harnais est mieux appréhendé comme un hyperparamètre que comme une recette universelle, et devrait être adapté au problème spécifique et au modèle sous-jacent. Nous constatons également que les progrès précoces de la découverte prédisent la performance finale, et utilisons cette propriété pour présenter une expérience d'allocation adaptative à budget équilibré qui lance plusieurs harnais, élague les exécutions partielles faibles et réalloue le calcul aux survivants les plus performants, surpassant à la fois l'engagement envers un harnais fixe échantillonné aléatoirement et un ensemble de harnais non adaptatif. Ensemble, ces résultats motivent le passage d'une sélection de harnais fixe à une adaptation en ligne guidée par les performances précoces. Nous publions l'ensemble des réservoirs d'exécutions, y compris les distributions nulles de référence pour chaque paire modèle–problème, en tant qu'infrastructure statistique réutilisable pour les futures propositions de harnais.
One-sentence Summary
After systematically decomposing and evaluating 30 discovery harnesses across 12 model–problem pairs and finding no universally superior harness, researchers from UC Berkeley, MIT, and the MIT-IBM Watson AI Lab propose an adaptive-allocation method that leverages early progress to prune weak runs and reallocate compute, outperforming both fixed harnesses and ensembles.
Key Contributions
- The paper systematically decomposes the OpenEvolve and TTT-Discover discovery harnesses into their constituent components and evaluates 30 budget-matched harness variants across 12 model–problem pairs using over 3.1 million LLM rollouts and repeated-trial statistical analysis.
- The study demonstrates that no fixed discovery harness generalizes reliably across models and problems; harness choice is better treated as a model- and problem-dependent hyperparameter, with simpler alternatives often outperforming OpenEvolve-style configurations.
- The work presents an adaptive harness ensemble that uses early-run progress to prune underperforming harnesses and reallocate compute to stronger survivors, achieving average final performance gains over both a single fixed harness and a non-adaptive ensemble.
Introduction
The authors investigate LLM-guided autonomous discovery systems that iteratively generate, evaluate, and improve candidate solutions. In these systems, a harness controls archive construction, parent selection, exploration, and budget allocation, but prior work evaluates composite harnesses with only a few trials, making it impossible to separate meaningful design choices from run-to-run variance. The authors conduct a large-scale, statistically controlled evaluation of 30 budget-matched harnesses across 3.1 million rollouts and 12 model–problem pairs, finding that no single fixed harness reliably transfers across settings. They then show that treating harness choice as an online, problem-dependent hyperparameter—with partial-run feedback used to prune and reallocate compute—improves final performance over committing to a single harness upfront.
Method
The authors develop a framework for program discovery that unifies two popular search harnesses and extends them with an online budget allocation strategy. The method begins with a greedy sequential best-of-N baseline, which at each iteration selects the single highest-scoring program from the history Ht as the parent:
pt=argx∈HtmaxS(x).This deterministic top-1 selection, pt∼Et1, generates N candidate children that are evaluated and appended to the history. The total rollout budget is fixed at B=NT across iterations.
Moving from this baseline to the OpenEvolve harness involves four progressive relaxations: replacing the top-1 archive with a top-K archive EtK, introducing epsilon-greedy exploration by occasionally sampling from the full history Ht, shifting the budget from breadth to depth by decreasing N and increasing T, and finally adding inspiration sampling, MAP-Elites diversity maintenance, and multi-island evolution with crossovers. To derive the TTT-Discover harness, the scoring function is transformed into a value estimate of the subtree rooted at a program, an Upper Confidence Bound (UCT) exploration bonus is added based on visitation counts, and the bonus is modified to a PUCT rule with a prior estimate. TTT-Discover also samples multiple parents per time step rather than a single parent.
Because the optimal fixed harness varies across model–problem pairs, the authors introduce an online allocation policy that dynamically selects among harnesses using intermediate feedback. The adaptive harness ensemble starts multiple harness configurations, advances them to one or more checkpoints (e.g., 25%, 50%, 75% of a full run), ranks the partial runs by the best evaluator score observed so far, prunes weaker runs, and spends the remaining compute on the survivors. The total budget is fixed to Be=5 full-run equivalents, matching the standard best-of-five evaluation. A single-stage policy satisfies the budget constraint
mq+s(1−q)≤Be,where m partial runs are advanced to checkpoint q, and s survivors are completed. Multi-stage pruning uses several checkpoints 0=q0<q1<⋯<qL=1, with mℓ active configurations at stage ℓ, leading to
ℓ=1∑Lmℓ(qℓ−qℓ−1)≤Be.The strongest policies begin with a broad portfolio at early checkpoints and progressively concentrate the budget as more informative feedback becomes available. For example, a three-stage 12→5→2→1 schedule prunes at 25%, 50%, and 75% of a full run, outperforming both fixed-harness commitment and an unpruned harness ensemble under the same compute budget.
Experiment
The evaluation compares search harnesses across four LLMs and three mathematical discovery tasks under a fixed rollout budget. Pair-level and cross-pair significance tests reveal that no single fixed harness consistently outperforms the simple Sequential BoN baseline, and the strongest observed harness varies across model–problem pairs. Early-run performance is shown to be predictive of final outcomes, and an adaptive online allocation strategy that starts many harnesses and prunes based on intermediate feedback yields higher average performance than both single-harness commitment and an unpruned harness portfolio.
Under a fixed compute budget of five full-run equivalents, adaptive online harness allocation policies that start multiple configurations, evaluate partial progress, and prune weaker runs consistently outperform fixed strategies. The strongest adaptive schedule, a three-stage pruning approach, raised the average final score from 84.35% to 85.75% by beginning with a broad portfolio and progressively concentrating resources on the most promising candidates. This demonstrates that early partial-run feedback is effective for directing compute toward better solutions. Every adaptive pruning schedule surpassed the single-harness baseline, the unpruned harness portfolio, and the Sequential BoN reference in average final score. The top-performing policy, which prunes at 25%, 50%, and 75% of a full run, achieved the highest average score of 85.75% and outperformed the unpruned portfolio on 11 of 12 model–problem pairs.
Under a fixed compute budget of five full-run equivalents, adaptive online allocation policies that launch multiple configurations, assess partial progress, and prune weaker runs consistently outperform static strategies. A three-stage pruning schedule that starts with a broad portfolio and progressively concentrates resources on the most promising candidates raised the average final score, demonstrating that early partial-run feedback effectively directs compute toward better solutions. All adaptive pruning schedules surpassed the single-harness baseline, the unpruned harness portfolio, and the Sequential BoN reference, with the best policy (pruning at 25%, 50%, and 75% of a full run) outperforming the unpruned portfolio on 11 of 12 model–problem pairs.