One-sentence Summary

NVIDIA, NTU, and MIT propose SoL-Pi, a recursively scaling auto-research harness that distills four surviving mechanisms spanning action execution, context compaction, observation handling, and delegated reading, achieving performance comparable to Pi across GPT-5.6 Sol and Opus 5 on the 51-task EdgeBench evaluation while reducing recorded token traffic by 44.7–49.0% and API cost by about one third.

Key Contributions

• Introduces SoL-Pi, an automated harness-layer search system that discovers reusable efficiency mechanisms across action execution, context compaction, observation handling, and delegated reading while keeping the underlying model fixed.
• On the 51-task EdgeBench evaluation, SoL-Pi achieves performance comparable to Pi across GPT-5.6 Sol and Opus 5 while reducing recorded token traffic by 44.7 to 49.0 percent and API cost by about one third.
• Best-performing candidates improve model performance by 5.3 to 12.8 percent and token efficiency by 9.8 to 18.2 percent; estimated hourly savings are $8.75 to$8.75to13.50 relative to native Codex and Claude Code harnesses and $4.36 to$4.36to5.71 relative to Pi.

Introduction

Foundation models now support increasingly open-ended, long-horizon agents for autonomous research, software engineering, and early recursive self-improvement, making task-level token efficiency a first-order concern. Prior efficiency work has mostly reduced cost per token through infrastructure, compression, or cheaper models, while automated harness-level optimization remains difficult because tool use, context management, verification, delegation, recovery, and termination are tightly coupled, and evolved harnesses can overfit their search tasks. The authors introduce SoL-Pi, an RSI-inspired auto-research system that searches for harness improvements with a broad-to-deep funnel, freezes candidates before held-out validation, and scales through isolated lineages. SoL-Pi discovers four mechanisms that transfer beyond development tasks, maintaining performance on EdgeBench across GPT-5.6 Sol and Opus 5 while reducing recorded token traffic by 44.7 to 49.0 percent and API cost by about one third.

Dataset

The authors use two types of search environments, totaling 535 executable environments.

• Sources and composition:

  • Repository-derived tasks: 495 environments built from GitHub issue and pull request pairs.
  • Verifier-driven synthetic tasks: 40 environments with executable success verifiers, mostly using a Terminal-Bench-2-style interface.

• Repository-derived processing:

  • Each environment pairs a GitHub issue with the pre-fix repository state and offline dependencies.
  • The accepted patch and change history serve as a reference trajectory.
  • The pull request and regression test are hidden from the agent.
  • Only environments whose test fails before the patch and passes afterward are retained.

• Verifier-driven processing:

  • An executable verifier is generated first to define success, then a task environment is built around it.
  • This supports multiple valid solution paths without requiring a reference trajectory.

• Usage:

  • The search pipeline uses separate environments for mechanism discovery and transfer evaluation.
  • These environments support automatically evaluated search grounded in software changes and open-ended problem solving.
  • The construction paths and the isolation boundary between mechanism search and held-out evaluation are summarized in Figure 3.

Method

The authors introduce SoL-Pi, a system designed to discover transferable harness improvements for token efficiency through an automated, recursive self-improvement inspired search process. Rather than relying on manual inspection of execution traces, the framework employs a research agent to analyze trajectories from a base harness, identify recurring sources of overhead, and propose candidate mechanisms to address them.

The overall workflow operates as a broad-to-deep funnel. The research agent observes execution traces from a higher-cost harness, proposes changes to reduce token cost, and tests these candidates against fixed capability and efficiency metrics. Candidates must improve efficiency while keeping capability metrics within predeclared tolerances. The system strictly isolates these optimization metrics from the agent's control to prevent gaming the acceptance criteria. Successful mechanisms are merged into the final SoL-Pi harness, which is then evaluated on held-out datasets like EdgeBench to ensure generalization.

The search pipeline allocates effort in two stages: an outer stage exploring a broad pool of hypotheses and an inner stage developing selected hypotheses independently. As illustrated in the workflow diagram, the process begins with trajectory rollouts and map-reduce analysis to identify avoidable work. This informs mechanism proposals and candidate implementation. An iterative implementation loop extends the standard autoresearch cycle; an implementer refines the candidate to meet explicit completion criteria, followed by an independent review. Failed reviews trigger revisions. Crucially, the development feedback loop is strictly separated from held-out evaluation. The harness and acceptance rules are frozen before any evaluation on held-out data, ensuring that held-out results never feed back into the search loop to cause overfitting.

To support this search, the authors utilize a diverse set of 535 executable environments, distinct from the held-out EdgeBench. These search environments fall into two categories. Repository-derived environments pair GitHub issues with pre-fix repository states and hidden regression tests, retaining only tasks where tests fail before the patch and pass afterward. Verifier-driven environments use executable success verifiers to define tasks, allowing for multiple valid solution paths without requiring a reference trajectory. This setup grounds the search in real software changes and open-ended problem solving while maintaining a strict isolation boundary for final validation.

The search process yields four reusable mechanisms that pass capability-constrained selection, each targeting different stages of the agent-environment loop to reduce redundant work. Action Fusion combines a file mutation and its follow-up command into a single tool request, eliminating an intermediate model round trip. Online Context Compact uses plan-step completion to reconsider context compaction, invoking it only when projected input savings exceed the estimated cost of rewriting the prompt cache. ObservationPack addresses large tool outputs by archiving results exceeding 10 KiB locally; it sends them in full for the first two requests but substitutes a stable handle and a short excerpt thereafter, allowing on-demand retrieval. Finally, the Evidence-Preserving Reducer compresses build and test logs using a lower-cost model to extract key evidence into a compact receipt. A deterministic verifier checks the receipt's integrity, falling back to the original log if verification fails or if the receipt provides no size reduction. These mechanisms operate complementarily to lower token traffic while preserving the information necessary for the agent to complete its tasks.

Experiment

The evaluation compares SoL-Pi against native and Pi baselines on EdgeBench, Terminal-Bench 4, IMO 2026, and a kernel-optimization swarm, using GPT-5.6 Sol and testing transfer to Opus 5. SoL-Pi consistently lowers token traffic, API cost, and cost per solved task while preserving or improving aggregate task performance, and these efficiency gains transfer to an unseen backend and generalize beyond the original benchmark. In a multi-agent kernel-optimization setup, SoL-Pi workers also yield better results than Pi workers under the same budget, suggesting that harness efficiency can improve collective exploration. Ablation and lineage analyses show that each learned mechanism reduces token usage, that mechanisms interact complementarily, and that the auto-research loop can stabilize and retain mechanisms such as Action Fusion.

Among the listed GPT-5.6 Sol harnesses on EdgeBench, Oh-My-Opencode delivers the highest average score while OpenSquilla delivers the lowest token traffic, lowest token cost, and best token efficiency. Average score and token efficiency are not aligned across harnesses: OpenCode records the highest token cost and weakest token efficiency but a mid-level average score. The official GPT-5.5 checkpoint reports an average score of 31.2 but no token traffic or cost. Oh-My-Opencode achieves the highest average score (38.523) among the listed GPT-5.6 Sol harnesses, exceeding the official GPT-5.5 checkpoint score of 31.2. OpenSquilla has the lowest total token traffic, lowest token cost, and best token efficiency among the listed GPT-5.6 Sol rows, but also the lowest average score.

On public EdgeBench tasks, the efficiency-oriented SoL-Pi configuration cuts total token traffic and token cost substantially relative to Pi while retaining most of Pi's average score. The performance-oriented configuration achieves the highest average score in the GPT-5.6 Sol block and still lowers token traffic and cost relative to Pi. The approach also transfers to an unseen Opus 5 backend without further search or adaptation, preserving most of Pi's score with large token and API cost savings. SoL-Pi Efficiency cuts total token traffic nearly in half and lowers token cost substantially versus Pi while retaining 93.7% of Pi's average score. SoL-Pi Performance attains the highest average score among the GPT-5.6 Sol configurations, improving over Pi by 5.3% while still reducing token traffic and token cost.

Across Terminal-Bench 4 and IMO 2026, no single harness dominates both solved tasks and cost. Codex and Pi solve the most Terminal-Bench 4 tasks, while SoL-Pi has the lowest total and per-task cost despite solving fewer tasks. On IMO 2026, Codex passes the most problems, and SoL-Pi matches Pi's pass count with the lowest total and per-problem cost. SoL-Pi lowers total model cost and cost per solved task on Terminal-Bench 4 relative to Codex and Pi, while solving three fewer tasks. On IMO 2026, SoL-Pi and Pi both pass three problems, but SoL-Pi achieves lower total cost and lower cost per passed problem. Codex reaches the highest IMO 2026 pass count at five problems but also has the highest total model cost among the compared harnesses.

An add-one evaluation on EdgeBench shows that each learned mechanism reduces total token traffic when added to Pi, although score effects vary by mechanism. ObservationPack achieves the highest average score among the single-mechanism variants under GPT-5.6 Sol, and the full stack achieves the lowest token count and cost while retaining most of Pi's score. All single-mechanism configurations also lower cost per score point relative to the baseline. Adding any single mechanism reduces total token traffic, and all single-mechanism configurations improve cost per score point relative to the Pi baseline. Under GPT-5.6 Sol, ObservationPack gives the highest average score among single-mechanism variants and is selected as the performance-oriented configuration. The full stack achieves the lowest total token count and cost across backends while retaining most of Pi's average score. The full stack trades lower cache-read traffic for a modest increase in cache-write traffic, yet total model cost still falls.

The experiments evaluate harness variants across EdgeBench, Terminal-Bench 4, and IMO 2026, focusing on task score, token traffic, and model cost. On EdgeBench, existing harnesses show that average score and token efficiency are not aligned, while the proposed SoL-Pi configurations achieve either near-Pi scores with large token and cost savings or the highest score with still-reduced cost, and the approach transfers to an unseen Opus 5 backend. On Terminal-Bench 4 and IMO 2026, no single harness dominates both solved tasks and cost, but SoL-Pi consistently lowers total and per-task cost with comparable or fewer solves relative to Codex and Pi. Ablations indicate that each learned mechanism reduces token traffic and improves cost per score point, with ObservationPack strongest for single-mechanism score and the full stack minimizing token count and cost.