One-sentence Summary

Tencent HY LLM Frontier, University of Maryland, College Park, and collaborators propose Long-Horizon-Terminal-Bench, a benchmark of 46 long-horizon terminal tasks with dense reward-based grading via fine-grained subtask decomposition that evaluates agents' planning, long-context management, and debugging, and even the strongest model achieves only $15.2\%$15.2% $\text{pass}@1$pass@1 at a partial-reward threshold of $0.95$0.95.

Key Contributions

• Long-Horizon-Terminal-Bench introduces 46 containerized terminal tasks across nine domains, each decomposed into fine-grained subtasks with deterministic, environment-grounded graders to provide dense partial-credit signals.
• Evaluation of 15 frontier models under a shared terminus-2 agent shows tasks average 231 episodes, 9.9M tokens, and 85.3 minutes per run; the strongest model, GPT-5.5, reaches only 15.2% pass@1 at a 0.95 partial-reward threshold, and the mean across models is 4.3%.
• Analysis of failure modes reveals that reliable long-horizon completion, including timeouts and weak self-verification, is the primary bottleneck over local reasoning, highlighting the need for better planning, memory, and calibrated stopping decisions.

Introduction

The authors introduce Long-Horizon-Terminal-Bench, a benchmark of 46 containerized terminal tasks across nine domains, targeting long-horizon workflows where AI agents must execute hundreds of steps, maintain plans, and recover from mistakes over tens of minutes. Real-world tasks like reproducing research results or debugging compiler toolchains require sustained, coherent progress, but existing benchmarks focus on short tasks and grade only final outcomes, yielding sparse rewards and no visibility into where agents fail. The key contribution is a dense subtask-based grading system that assigns partial credit for intermediate milestones, revealing that even the strongest frontier agent (GPT-5.5) achieves only 15.2% success rate and that typical failures stem from inability to sustain long-term execution, verify completion, and finish within budget rather than from local errors.

Dataset

The authors introduce Long-Horizon-Terminal-Bench, a benchmark of 46 long-horizon terminal tasks intended to evaluate autonomous agents over extended, multi-step workflows. The dataset is constructed as a set of containerized environments, each presenting a deliberately broken real-world project that an agent must diagnose and repair solely through the command line.

Key aspects of the dataset composition and usage:

• Task composition and sources: The 46 tasks span 21 broad categories (interactive games, multimodal audits, software engineering, robotics, climate science, chip design, etc.), with no single category dominating. They are drawn from realistic professional workflows—materials phase-diagram auditing, SLAM pipeline repair, climate extreme-event detection, audio-visual alignment, and others. Some are adapted from prior benchmarks like APEX-Agents, while others are newly created for this dataset.

• Task structure: Each task follows the Harbor format and provides:

• A natural-language instruction (the only specification the agent sees).

• A Docker image containing all code, data, tools, and helper scripts.

• A task configuration file.

• A hidden verifier with an oracle implementation used for grading.

• Grading and processing details:

• Tasks are decomposed into a small set of semantically meaningful subtasks with their own checks, enabling partial-credit scoring via subtask completion rate.

• Public checks are lightweight: they validate only command-line behavior, file formats, and a few simple examples, contributing a small reward weight to discourage overfitting.

• The majority of the reward comes from hidden stress cases that dynamically generate harder inputs and schema variations (nested manifests, gzip-plus-base64 wrappers, renamed fields, missing values, injected noise, rotated images, anomalous frames, alternative coordinate conventions). The gold solution must achieve a perfect score on the full hidden evaluation suite.

• Construction includes public asset-generation scripts, a weak baseline, a multi-step solve.sh, and the hidden verifier. Difficulty was calibrated by running Deepseek-V4-Pro under a 1.5-hour time budget, refining tasks, and filtering from 120 candidates down to the final 46.

• How the data is used: Agents are evaluated in an interactive, step-by-step fashion. After the container starts, the agent issues shell commands, edits files, runs scripts, and inspects intermediate outputs for potentially hundreds of steps. There is no training split; the benchmark is used purely for evaluation, and an agent’s long-horizon performance is measured by its subtask completion rate across all tasks.

Method

The authors design Long-Horizon-Terminal-Bench as a containerized evaluation framework where each task is formulated as a self-contained terminal environment. Every task provides a Docker image that bundles all code, data, tools, and helper scripts, along with a single natural-language instruction describing a complex, multi-stage goal. The agent interacts with the environment exclusively through the command line, issuing shell commands, editing files, running scripts, and inspecting intermediate outputs over hundreds of steps. This setup ensures that solving a task requires sustained reasoning, debugging, and progressive refinement rather than a single correct invocation.

To capture long-horizon progress, the benchmark replaces a binary pass/fail metric with a subtask-based grading scheme. Each task is decomposed into a small set of semantically meaningful subtasks $s_1, \dots, s_K$s1​,…,sK​ with corresponding normalized scores $r_k \in [0,1]$rk​∈[0,1]. The overall task reward is computed as a weighted average:

$R = \frac{\sum_{k=1}^{K} w_k r_k}{\sum_{k=1}^{K} w_k}$R=∑k=1K​wk​∑k=1K​wk​rk​​

where weights $w_k$wk​ are non-negative (default equal, with higher weight on the final goal when needed). This design provides dense partial credit, revealing how far an agent progresses through a workflow even when it does not reach the final solution.

Subtasks are evaluated using objective evidence from the final container state and fall into three categories:

• Binary subtasks use strict Boolean checks (e.g., unit tests passing, a service responding on a port) and yield $r_k \in \{0,1\}$rk​∈{0,1}.
• Continuous or thresholded subtasks handle quantitative targets. For example, a metric-reproduction score can be 1.0 when within tolerance and linearly decrease to 0 as error grows, giving meaningful credit for near-success.
• Episode-aggregating subtasks apply to campaign-style tasks (games, repeated audits) and measure the fraction of episodes or levels where an internal success condition is met, testing reliability across many trials.

A task is considered resolved when $R \ge \tau$R≥τ, and the mean $R$R across tasks is reported to capture progress on problems that are not fully solved.

The dataset construction pipeline instantiates this framework across a diverse set of real-world professional workflows—materials auditing, robotics SLAM repair, climate detection, scientific figure reconstruction, and others. For each problem, the authors build a complete but deliberately broken terminal-only project. Agents must work entirely through the terminal to inspect code, run commands, and analyze multimodal artifacts, then diagnose and repair the core pipeline. Each task includes a weak baseline, a hidden gold solution, and a multi-step solve script.

To discourage overfitting, public checks are limited to command-line behavior and a few simple examples, contributing low reward weight. The majority of the reward is assigned through hidden stress cases that dynamically generate harder inputs and schema variations—nested manifests, gzip-plus-base64 wrappers, renamed fields, missing values, injected noise, rotated images, and alternative coordinate conventions. This forces agents to implement robust parsing and core algorithms that generalize beyond visible examples. The gold solution must achieve a verifier score of 1.0 on the full hidden suite.

Difficulty calibration was performed by iteratively running a strong agent (Deepseek-V4-Pro) under a 1.5-hour time budget and adjusting tasks until they remained challenging but solvable in principle. From an initial pool of 120 candidate tasks, the final benchmark consists of 46 containerized tasks in the Harbor format, all sharing a common agent harness.

Experiment

The evaluation uses the Harbor framework with the Terminus-2 agent harness (and Codex2 for GPT-5.3) to benchmark 15 frontier models on 46 long-horizon terminal tasks, measuring pass rates, dense normalized rewards, and cost. The best model, GPT-5.5, solves only 15.2% of tasks, and dense subtask-level grading proves essential for distinguishing models, as binary pass/fail metrics collapse rankings and hide near-misses that approach completion. Failures are dominated by timeouts rather than local execution errors, and many agents exhibit false finishes where they stop prematurely despite high reward, indicating that weak self-verification and time-budgeting are the central bottlenecks. Cost analysis shows that higher inference spending alone does not guarantee better long-horizon performance.

Cost estimates for long-horizon tasks vary widely, from roughly $2.50 to$2.50to28 per task. The most expensive model, GPT-5.4, underperforms a cheaper counterpart because it uses many more episodes, while several low-cost models define the Pareto frontier, showing that high pass rates can be achieved without high spending. GPT-5.4 is the most expensive model (about $28 per task) but achieves a lower pass rate than GPT-5.5 (about$28pertask)butachievesalowerpassratethanGPT−5.5(about21 per task), driven by 302 episodes per task versus 208, despite comparable per-token pricing. Hy3, Doubao Seed 2.1 Pro, and MiniMax M3 form the low-cost Pareto frontier at around $2.50,$2.50,5, and $6 per task, respectively, proving that efficient long-horizon performance is possible at a fraction of the cost of top-tier models.

Long-horizon task costs span from roughly $2.50 to$2.50to28 per task, and the most expensive model, GPT-5.4, underperforms the cheaper GPT-5.5 because it requires far more episodes despite comparable per-token pricing. The low-cost models Hy3, Doubao Seed 2.1 Pro, and MiniMax M3 define the Pareto frontier at around $2.50,$2.50,5, and $6 per task, proving that high pass rates can be attained efficiently without heavy spending.