One-sentence Summary

Researchers from Stanford University, Anthropic, and collaborators present Terminal-Bench 2.0, a curated benchmark of $89$89 hard, realistic command-line tasks with human-written solutions and comprehensive tests, demonstrating that frontier agents score below $65\%$65% and offering an error analysis to guide future model and agent improvements.

Key Contributions

• Introduces Terminal-Bench 2.0, a benchmark of 89 tasks set in real computer terminal environments, with each task featuring a unique environment, a human-written solution, and comprehensive verification tests, distinguishing it from prior benchmarks that rely on synthetic environments or narrow CLI subtasks.
• Benchmarks frontier LLMs and agents on these tasks, showing that top models resolve less than 65% of tasks while smaller models score around 15%, and provides a taxonomy of failure modes to guide future model and agent improvements.
• Publishes the dataset and evaluation harness at tbench.ai, and incorporates reproducibility measures such as pinned package versions, pre-built Docker images, and a Big-Bench canary string to aid training-data decontamination.

Introduction

As AI agents become capable of operating autonomously over long horizons in high-stakes domains, benchmarks must evolve to reflect the complexity of real-world tasks. The terminal is a critical interface for skilled work such as software engineering, scientific computing, and cybersecurity, and it has become a standard environment for AI agents like Cursor and Claude Code. However, existing benchmarks often focus on narrow aspects of command-line usage or rely on synthetic environments, failing to capture the diversity and difficulty of professional-grade terminal work.

The authors introduce Terminal-Bench, a framework for evaluating agents on realistic command-line tasks. Each task includes a containerized environment, an instruction, verification tests, and a manually written reference solution. They also present Terminal-Bench 2.0, a dataset of 89 tasks manually verified by three reviewers. Benchmarking frontier models reveals that they resolve less than 65% of tasks, while smaller models score around 15%. The authors also provide a taxonomy of failure modes to guide future development.

Key limitations include the possibility of agents accessing the internet to locate oracle solutions, potential training data contamination, and variability in external dependencies or machine resources. Despite manual review, some tasks may still fall short of verification criteria, but the authors prioritize diversity and realism over ease of validation.

Dataset

The authors introduce Terminal-Bench, a framework for building realistic tasks that an agent can solve using a terminal. The dataset is composed of tasks that range from training a machine learning model, to building and running Linux from source code, to reverse engineering binary files.

Dataset composition and sources

• The tasks are crowd-sourced through open-source contributions, with 93 contributors creating 229 tasks.
• From those 229, the authors selected 89 tasks for the Terminal-Bench 2.0 dataset based on the author's difficulty assessment and a quality assessment by three experienced human reviewers.
• Each task consists of an instruction, a Docker image, a set of tests, an example solution, and a time limit. The instruction describes what the agent must complete within the time limit inside the Docker container.

Key details for each subset

• The tests verify that all outcomes described in the instruction have been achieved by testing properties of the final container state. They do not test the agent's commands or console output, making the framework outcome-driven.
• Contributors assigned expert and junior-engineer completion time estimates to their tasks. The distribution of these estimates is reported in the paper, along with a high-level category for each task.
• Tasks are specified using the Harbor task format and run using the Harbor harness, which supports popular agents including Claude Code, Codex CLI, OpenHands, and Mini-SWE-Agent, as well as the authors' own agent, Terminus 2.

Verification and filtering rules

• A task is considered verified if experienced reviewers determine it meets standards for specificity, solvability, and integrity.
• Specificity means the unit tests pass if and only if the container ends in an acceptable state.
• Solvability is confirmed by an oracle solution script that, when executed, causes all test cases to pass.
• Integrity ensures agents cannot cheat by taking shortcuts that would not exist in a real-world deployment, such as removing future commits from a git repository history.
• The verification process includes automated workflows that run the oracle solution, contributor checklists, an automated language model tool to find common mistakes, manual review by experienced reviewers, running tasks with multiple language models, an adversarial exploit agent to detect design flaws, and final manual review by two additional auditors.
• The authors note the benchmark has been subjected to roughly three reviewer-hours of work per task in the final benchmark.

How the data is used

• The tasks are interactive. Once the instruction and Docker container are provided to an agent, it must explore and manipulate the environment by calling tools, such as editing files or running Bash commands, to complete the task.
• The authors use the dataset as a neutral testbed for comparing model performance, including their own agent, Terminus 2.

Method

The authors introduce Harbor, a dedicated framework designed to build and execute agent evaluations at scale. Terminal-Bench tasks are implemented in the Harbor task format and are executed using the Harbor harness, ensuring a consistent and reproducible evaluation pipeline. Terminal-Bench 2.0 is distributed through the Harbor registry, allowing users to run the benchmark directly with the command harbor run -d [email protected].

Harbor is pre-integrated with multiple container sandbox providers, which enables flexible and isolated execution environments for each agent run. In the authors' experiments, they employ Daytona as the sandbox provider, running between 32 and 100 containers in parallel to achieve high-throughput evaluation. The Harbor configuration files used in these experiments are publicly available in the repository at github.com/laude-institute/terminal-bench-experiments, facilitating reproducibility and further customization by the research community.

Experiment

The evaluation ran six agents across 16 frontier models on Terminal-Bench 2.0, totaling over 32,000 trials, with a custom scaffold (Terminus 2) providing a neutral baseline for model comparison. Results show that proprietary models paired with compatible agents dominate the top rankings, with Codex CLI and GPT-5.2 achieving the highest resolution rate, and model choice generally matters more than agent scaffold. Cost and interaction patterns vary widely, but neither turn count nor token usage correlates with success. Newer models nearly doubled state-of-the-art performance in eight months, suggesting the benchmark may soon saturate. Human-predicted and empirical task difficulty correlate positively, though humans often overestimate ease for tasks requiring creative reasoning. Trajectory-level error analysis reveals distinct failure profiles: closed models mostly suffer execution errors, while an open model shows balanced errors across execution, coherence, and verification. Command-level analysis identifies missing executables as the most common failure, with error rates ranging from 9% to 27% across models.

Task completion time estimates in Terminal-Bench 2.0 show a wide range, with expert estimates mostly under one day and junior estimates mostly between one hour and one week. A small fraction of tasks are extremely long-horizon, including one that takes an expert nearly a full day and a junior ten days. Experts estimate nearly half of tasks take under one hour, while juniors estimate only about 8% fall in that range. Junior estimates are concentrated in the 1 hour to 1 day window, with over 70% of tasks in that band. A small set of tasks are estimated to take juniors over a week, and none are expected to take experts that long. One task, fixing the OCaml garbage collector, is estimated to take an expert 24 hours and a junior 240 hours.

Terminal-Bench 2.0 task duration estimates reveal a wide spread, with experts generally expecting most tasks to finish within a day and juniors predicting longer timelines, mostly between one hour and one week. Experts rate nearly half of the tasks as under one hour, while juniors place over 70% in the 1 hour to 1 day range, with only about 8% under an hour. A few tasks are extremely long-horizon, such as fixing the OCaml garbage collector, which experts estimate takes 24 hours and juniors 240 hours, and no task is expected to take experts over a week, though some do for juniors.