One-sentence Summary

Researchers from Shanghai Jiao Tong University, Singapore Management University, East China Normal University, and Shanghai Innovation Institute propose EarlyEval, a lightweight framework that trains LightGBM success and failure classifiers on behavioral, textual, and reference-solution features to halt agent runs early at calibrated confidence thresholds, eliminating 13–26% of steps and up to 44.1% input tokens across SWE-bench Verified, TerminalBench, and Toolathlon with 89–97% accuracy and minimal resolve rate perturbation.

Key Contributions

• Early outcome prediction is introduced as a complementary efficiency axis for LLM agent benchmarking, cutting cost within each task by terminating runs early based on intermediate behavior rather than reducing the number of tasks.
• EarlyEval, a lightweight framework, trains a pair of LightGBM success and failure classifiers on behavioral, textual, and reference-solution features and stops an agent run when either classifier exceeds a calibrated confidence threshold.
• Across SWE-bench Verified, TerminalBench, and Toolathlon, EarlyEval eliminates 13% to 26% of agent steps and up to 44.1% of input tokens at 89% to 97% prediction accuracy, while per-agent resolve rates shift by only one to two percentage points on average.

Introduction

Evaluating LLM agents is essential for guiding development, but the cost of running modern agentic benchmarks has surged, with a single pass on SWE-bench Verified costing hundreds of dollars and longer-rollout benchmarks reaching thousands. This expense makes frequent evaluation impractical for many teams and slows the iteration cycle. Prior efforts to reduce costs have focused on benchmark distillation, which downsizes the task set while leaving the per-task execution cost unchanged. The authors introduce EarlyEval, a complementary approach that terminates an agent’s rollout early when its final outcome can be confidently predicted from intermediate behavior. By training lightweight classifiers on historical trajectories from other agents, EarlyEval cuts execution steps and token consumption while preserving per-agent resolve rates and leaderboard rankings across multiple benchmarks.

Dataset

The authors construct a training dataset from agent trajectories collected across three multi-step benchmarks. Each trajectory is a sequence of steps with a final binary success label, and the data is used to train a model that predicts eventual success from partial execution prefixes.

Dataset sources and composition

• SWE-bench Verified

  • 500 human-validated GitHub issues across 12 Python repositories.
  • Trajectories generated by mini-SWE-agent paired with 16 LLMs (Claude, GPT-5, Gemini, GLM, DeepSeek, Devstral, Kimi, MiniMax families).
  • Total: 7,805 trajectories.
  • Each task includes a gold patch, enabling reference-solution features.

• TerminalBench

  • 89 command-line automation tasks.
  • 37 distinct agent configurations (scaffolds: mini-SWE-agent, Codex CLI, Claude Code, Gemini CLI, OpenHands, Terminus-2; models: GPT-5, GPT-5-mini, Claude-Haiku-4.5, Claude-Opus-4.5, Gemini-2.5-Pro).
  • Total: 6,757 trajectories (multiple rollouts per task).
  • No per-task reference solutions; only behavioral and textual features are used.

• Toolathlon

  • 108 complex API and tool-use tasks.
  • Native Toolathlon scaffold with 22 LLMs, three rollouts per task.
  • Total: 7,116 trajectories.
  • No reference solutions; same feature restrictions as TerminalBench.

Data processing pipeline

• Trajectories shorter than 10 steps are discarded (insufficient signal).
• For each trajectory of length T, all prefixes of lengths 0 through T are extracted and paired with the trajectory’s final outcome label (success/failure).
• Each prefix is converted to a fixed-length feature vector composed of three families:
  • Behavioral features: run progression metrics (volume, pacing, milestone timing, error/test signals, stalling patterns).
  • Textual features: semantic blocks (task prompt, full action history, most recent action, full environment feedback, most recent feedback) are vectorized independently with TF-IDF over word n-grams, then compressed to 64 dimensions (prompt) or 128 dimensions (action/feedback groups) via Truncated SVD, preserving block boundaries while keeping dimensionality low.
  • Reference-solution features (SWE-bench only): measures structural overlap between the current prefix and the provided gold patch (files, symbols, tests). Omitted for TerminalBench and Toolathlon.

Usage in the model The resulting feature vectors and binary labels serve as the training set. The model learns from these prefix-label pairs to predict final success at intermediate steps, enabling early termination decisions. No explicit train/validation split or mixture ratios are detailed in the provided text.

Method

The authors propose EarlyEval, a framework designed to predict an agent’s final outcome on a benchmark task from a partial trajectory and halt execution as soon as the eventual outcome becomes statistically evident. The system operates through a sequential inference workflow comprising two primary stages: offline predictor construction and online step-by-step inference.

As shown in the figure below:

In the offline phase, the authors leverage historical agent runs that have already been evaluated to construct the training data. For a given benchmark, they collect a pool of trajectories $\tau = (e_1, ..., e_T)$τ=(e1​,...,eT​), each associated with a binary evaluation score $y \in \{0, 1\}$y∈{0,1}. Trajectories shorter than 10 steps are discarded to ensure sufficient signal for optimization. Each trajectory is decomposed into a sequence of labeled prefixes $\tau_{:k} = (e_1, \ldots, e_k)$τ:k​=(e1​,…,ek​) for $k = 0, 1, \ldots, T$k=0,1,…,T, where each prefix is paired with the final outcome label $y$y. These prefixes are mapped to a fixed-length multimodal feature vector $\phi(\tau_{:k}) \in \mathbb{R}^d$ϕ(τ:k​)∈Rd. The feature extraction process captures three distinct families of signals. Behavioral features capture run progression invariants, including volume metrics, structural composition, milestone timing, and environment feedback signals. Textual features encode the natural-language context by isolating the task prompt, action history, and environment feedback into distinct semantic blocks. Each block is vectorized using TF-IDF over word n-grams and compressed via Truncated Singular Value Decomposition to maintain computational efficiency. Reference-solution features are optionally leveraged when ground-truth human patches are available, measuring the structural overlap between the current prefix and the gold solution.

To judge these partial trajectories, the authors train a pair of agent-agnostic predictors using gradient-boosted decision tree ensembles via LightGBM. This architecture is selected for its ability to evaluate high-dimensional feature vectors in under a millisecond on a single CPU core, ensuring negligible computational overhead during step-by-step inference. EarlyEval optimizes two separate ensembles over the feature representation $\phi$ϕ: a success predictor $h_+$h+​ and a failure predictor $h_-$h−​. Both models ingest the same feature vector but are optimized against inverted target sets. The success predictor targets $y = 1$y=1, while the failure predictor targets $1 - y = 1$1−y=1 (i.e., $y = 0$y=0). Training two distinct predictors allows positive and negative evidence to accumulate independently, reflecting the asymmetric behaviors signaling success and failure. This design also creates an explicit unconfident region where both predictors output low probabilities, allowing the agent to continue execution when the outcome remains ambiguous. To prevent data leakage, the trajectory pool is partitioned by task into training and validation folds. Furthermore, each prefix instance is weighted by $1 / (T + 1)$1/(T+1) to ensure that every trajectory contributes identical total mass to the objective function, preventing prolonged trajectories from dominating the optimization loss.

During online inference, the agent interacts with the environment step by step. At each step, EarlyEval extracts the feature vector $\phi$ϕ from the accumulated partial trajectory and inputs it into both ensembles. Because regularized tree ensembles can distort output probability scales, the authors recalibrate the raw scores using Platt scaling. A one-dimensional logistic regression maps the raw ensemble score $\hat{s}$s^ to a calibrated probability:

$$p = \sigma \big(a \logit(\hat{s}) + b \big)$$p=σ(a\logit(s^)+b)

where the scalar parameters $a$a and $b$b are fitted on the held-out validation split. This monotonic transformation rescales the outputs so that confidence thresholds carry a consistent meaning across both predictors. The calibrated probabilities $p_+$p+​ and $p_-$p−​ parameterize a dual-threshold decision mechanism. The system compares $p_+$p+​ against a success threshold $s$s and $p_-$p−​ against a failure threshold $f$f. A run is intercepted and marked with a predicted outcome at the first step where $p_+ \ge s$p+​≥s or $p_- \ge f$p−​≥f. If both probabilities remain below their respective thresholds, the system defers commitment and allows the agent to proceed to subsequent steps. This threshold-based logic dictates the stringency of evidence required before intervention, enabling a tunable trade-off between prediction accuracy and compute savings.

Experiment

EarlyEval is evaluated on three multi-step agentic benchmarks (SWE-bench Verified, TerminalBench, Toolathlon) using a leave-one-agent-out protocol to predict task outcomes and halt unpromising trajectories early. The system achieves substantial reductions in execution steps and token usage while maintaining high fidelity to full-run resolve rates and preserving agent rankings with strong rank correlations. Ablation studies show that EarlyEval's predictions are robust to missing feature families, with behavioral features being most critical, and that a LightGBM backbone outperforms neural and linear alternatives in the cost-fidelity trade-off.

Evaluation costs vary substantially across benchmarks and models, with no single model consistently cheapest. GPT-5.5 offers the lowest cost on three of five benchmarks, while Gemini 3.1 Pro is most economical on the multimodal suite and Claude 5 leads on SWE-bench Verified. GPT-5.5 achieves the lowest cost on SWT-bench, Commit0, and GAIA, where its GAIA run costs only $122 compared to$122comparedto1,305 for Claude 5. Gemini 3.1 Pro evaluates SWE-bench Multimodal for $641, significantly undercutting GPT-5.5 ($641,significantlyundercuttingGPT−5.5(1,453) and Claude 5 ($2,270).

The LightGBM backbone uniquely combines high early-stopping accuracy with substantial coverage, achieving the largest step reduction and minimal metric distortion. In contrast, direct neural and linear models either suffer from low accuracy or remain too passive to save meaningful compute, while a fine-tuned LLM judge offers competitive fidelity but introduces prohibitive inference overhead that cancels out its own savings. LightGBM reaches 95.0% accuracy and 34.8% coverage, cutting execution steps by 26.0% while keeping metric distortion to just 1.1 points. The fine-tuned Qwen judge attains 90.7% accuracy but saves only 17.9% of steps and requires a costly model forward pass at every trajectory step, offsetting the computational savings early stopping aims to provide.

Two experiments examine efficiency in model evaluation. The first compares evaluation costs across five benchmarks, revealing that no single model is universally cheapest and that cost efficiency depends on the benchmark. The second evaluates early-stopping strategies, where a LightGBM backbone achieves the best balance of high accuracy, substantial step reduction, and minimal metric distortion, while fine-tuned LLM judges incur prohibitive inference overhead that negates their savings.