One-sentence Summary

Researchers from ByteDance Seed introduce EdgeBench, a suite of 134 real-world tasks with ultra-long horizons of at least 12 hours spanning scientific discovery, software engineering, combinatorial optimization, professional knowledge work, formal mathematics, and interactive games, and report, to the best of the knowledge, the first evidence that agent learning performance follows a log-sigmoid scaling law with remarkably high precision ($R^{2}=0.998$R2=0.998) while learning speed roughly doubles every three months.

Key Contributions

• EdgeBench is a benchmark of 134 diverse real-world tasks spanning scientific discovery, software engineering, combinatorial optimization, professional knowledge work, formal mathematics, and interactive games. Each task supports at least 12 hours of continuous agent operation with rich multilevel feedback, and 51 tasks together with the evaluation framework are publicly released.
• Analysis of roughly 38,000 hours of agent interaction reveals the first evidence that overall performance during environment learning follows a log-sigmoid scaling law (R² = 0.998), a relationship that holds across task families and enables forecasting later performance from early trajectories.
• Agent learning speed roughly doubles every three months across recent frontier model generations.

Introduction

The authors introduce EdgeBench, a benchmark of 134 diverse, day-scale tasks spanning scientific research, software engineering, and other domains, each providing multi-level feedback to mirror real-world trial-and-error workflows. Prior benchmarks struggled to measure learning because they offered short horizons and only end-state metrics, and test-time scaling studies had not examined cross-domain learning from environment interaction. By analyzing roughly 38,000 hours of agent–environment interaction, the authors uncover a log-sigmoid scaling law that relates aggregate performance to elapsed interaction time (R² = 0.998), propose a theoretical derivation based on frontier expansion on latent task graphs, and show that agent learning speed doubles roughly every three months.

Dataset

The authors of EdgeBench curate a collection of 134 real-world tasks, organized into six capability families. Each task is selected to be unsolved by current agents and to support continuous, iterative workflows rather than one-shot completion. Tasks that primarily require visual understanding (e.g., GUI operation) are excluded to isolate reasoning and learning ability.

Dataset composition

• Scientific Problems & ML (39 tasks) Uses real research data and experimental settings from working scientists. Agents must hypothesize, select models, validate against noisy observations, and refine iteratively. Many problems are open-ended with no known optimal solution.

• Systems & Software Engineering (36 tasks) Production-grade codebases with changes spanning thousands of lines (up to over 100,000 lines). Agents reason about cross-module coupling while meeting correctness and performance targets.

• Combinatorial Optimization (19 tasks) Open-ended, mostly NP-hard problems where exact methods are intractable. Progress relies on designing, tuning, and iterating on heuristic search strategies.

• Professional Knowledge Work (19 tasks) Real white-collar deliverables across finance, education, healthcare, and legal domains. Tasks match work that would take a human professional with 3+ years of experience roughly three full days. Many include multi-round review rubrics that simulate client feedback, enabling agents to learn from structured critique and revise iteratively.

• Formal Math & Theorem Proving (13 tasks) Frontier mathematical problems requiring large-scale machine-checked proofs in Lean. Agents receive structured intermediate guidance and can extend partial proofs incrementally. Most tasks are newly created for EdgeBench.

• Interactive Games & Simulators (8 tasks) Real games designed for human players, where mastery normally takes tens of hours. Large state spaces and procedural generation create strong out-of-distribution pressure. Agents must develop and refine strategies through many episodes of high-frequency interaction.

How the data is used

The paper employs the full set of 134 tasks as a benchmark to evaluate agents’ iterative reasoning and continuous learning capabilities. No training split is mentioned; the tasks are used for evaluation only, with an emphasis on measuring progress over repeated interactions, guided by the task-specific feedback loops described above.

Method

The authors design a comprehensive evaluation framework to measure how autonomous agents learn from experience in unfamiliar real-world environments. To capture the emergence of learning behaviors such as exploration and strategy revision, the framework utilizes 134 diverse tasks spanning six capability families, each structured as a day-scale challenge. The experimental pipeline evaluates five frontier models across these tasks. For each task-model pair, the authors execute three independent 12-hour trials and record the full submission trajectory, allowing for a detailed analysis of environment interaction over extended time horizons.

As shown in the figure below, the per-task learning curves exhibit highly heterogeneous dynamics across different models and domains.

The trajectories range from smooth incremental gains to long plateaus, abrupt breakthroughs, and irregular regressions. Despite this individual heterogeneity, the cross-task average curves reveal an unexpectedly smooth and common structure. To model this aggregate behavior, the authors fit the averaged environment learning curves using a three-parameter log-sigmoid model:

$S(t) = \frac{S_{\max}}{1 + (t_{\mathrm{mid}} / t)^\beta}$S(t)=1+(tmid​/t)βSmax​​

where $t$t represents the elapsed interaction time and $S(t)$S(t) denotes the best-so-far performance. In this formulation, $S_{\max}$Smax​ serves as the attainable score ceiling, $t_{\mathrm{mid}}$tmid​ indicates the interaction time at which the curve reaches half of that ceiling, and $\beta$β controls the sharpness of the progress concentration in log time. A smaller $t_{\mathrm{mid}}$tmid​ implies that the model reaches the bulk of its attainable score sooner, while a larger $\beta$β corresponds to a steeper learning transition.

To explain the theoretical underpinnings of this log-sigmoid law, the authors propose a model where environment learning is treated as a frontier expansion process on underlying task graphs. Each task is represented by a latent graph of score units. An edge weight measures how much an unlocked source unit helps unlock a target unit. A locked unit receives an influence field from its unlocked neighbors, and it unlocks randomly at an expected rate proportional to this field. Consequently, the expected score-growth rate is exactly the weighted frontier cut from unlocked units to locked units.

Under a mean-field approximation, the macroscopic unlocked-locked cut assumes a product-measure influence, leading to a differential equation where the score growth rate is proportional to $x(1 - x)$x(1−x). Here, the unlocked score mass supplies reusable capability, while the locked score mass measures the remaining opportunity for improvement. Furthermore, the effective time coordinate is logarithmic because self-similar graph structures imply that the search volume needed to traverse the graph grows exponentially with the difficulty scale. Substituting the logarithmic time coordinate into the frontier equation and solving it yields the exact log-sigmoid functional form, demonstrating that learning from environments and learning from pretraining data induce the same mathematical scaling form.

Experiment

EdgeBench’s dual-loop protocol isolates an agent’s local workspace from a hidden-judge evaluation, measuring learning through iterative submissions. Across 134 tasks and five frontier models, environment learning follows a precise log-sigmoid scaling law, with agent learning speed doubling approximately every three months. Agents benefit from accumulated experience beyond repeated sampling, and longer context windows provide a stable advantage, while learning trajectories reveal structured diagnose-edit-evaluate loops that drive improvement through sparse but cumulative breakthroughs.

All three-parameter S-curve families fit the agent's full-window learning trajectory with markedly lower error than the two-parameter log-linear baseline, confirming that performance improves in a saturating sigmoidal pattern rather than a simple logarithmic trend. The log-sigmoid yields the smallest RMSE, but the four S-curves differ by only 0.014 points, indicating that no single S-shape is uniquely required to capture the learning dynamics. Three-parameter S-curves (log-sigmoid, log-probit, log-gompertz, Weibull CDF) achieve RMSE values between 0.390 and 0.404, while the log-linear baseline reaches 0.717, a substantially worse fit. The log-sigmoid gives the lowest error, but the extremely narrow spread among the S-curve families suggests that the learning trajectory is broadly sigmoidal without strongly preferring one specific functional form.

Opus 4.8 leads the leaderboard across all time budgets, reaching the highest 12-hour overall score and outperforming all other models in every category. Submission efficiency analysis shows that better final performance is not simply a matter of more frequent effective submissions; GPT-5.4 had the highest effective-submission rate yet finished third, while Opus 4.8 submitted less often than GPT-5.5 but still achieved the best result. A detailed agent trajectory reveals that structured, feedback-driven loops—making the problem measurable, decomposing failures, focusing on a main bottleneck, and preserving a stable core while repairing residual errors—can convert many failed trials into a few high-impact improvements. Opus 4.8 holds the top overall score at every time budget and leads in all six categories at 12 hours, with GPT-5.5 as the closest competitor across the board. GPT-5.4 had the highest effective-submission rate but still trailed the top two models, indicating that deliberate, high-quality improvements matter more than the sheer number of submissions.

An agent with accumulated experience increasingly outperforms a baseline without experience as the time budget grows, with the gap widening notably at 6 and 12 hours. The underlying run progresses through distinct phases: making the task measurable, decomposing failures, concentrating on a dominant bottleneck, and finally repairing residual errors while preserving the core solution, resulting in uneven but cumulative improvements. The performance gain from experience is minimal at 2 hours but becomes substantial after 12 hours, showing that the advantage compounds with time. Improvement follows a structured diagnose-edit-evaluate loop in which the agent first makes the problem scoreable, then isolates subproblems, targets a main bottleneck, and later patches remaining errors without disrupting the working pipeline.

An agent given a 12-hour budget and periodic auto-evaluation improved its score from 42.8 to 67.0 through a sparse but structured loop of diagnose, edit, and evaluate. The agent first made the task measurable, then decomposed failures, narrowed in on a dominant bottleneck, and finally repaired residual errors while keeping a working core solution. Only 27 of 224 agent submissions improved the best-so-far score, showing that most actions were exploratory probes rather than direct progress. The agent's improvement was not smooth but occurred in uneven jumps corresponding to distinct behavioral phases: making the problem scoreable, localizing the signal, improving source dynamics, and repairing the remaining waveform errors. The largest performance gain came from identifying a velocity/separation bottleneck and persistently searching around source-mass calibration, raising source dynamics from 64.2 to 89.0 in a 4–5 hour window. After finding a stable solution, the agent kept the core model and only attempted targeted residual corrections, which lifted the H1 waveform component score from roughly 47 to 95.

Agent learning trajectories follow a broadly sigmoidal pattern, with three-parameter S-curves fitting significantly better than a log-linear baseline. High-performing agents rely on structured, feedback-driven loops that make problems measurable, decompose failures, target dominant bottlenecks, and repair residual errors, rather than on frequent submissions. Accumulated experience compounds over time, as deliberate diagnose-edit-evaluate cycles turn exploratory probes into uneven but impactful improvements.