One-sentence Summary

Allen Institute for AI and University of Washington introduce SUPER, the first benchmark for evaluating LLMs on autonomously setting up and executing tasks from research repositories, which comprises 45 end-to-end problems with annotated expert solutions, 152 sub-problems targeting specific challenges, and 604 auto-generated problems, and shows that the best model (GPT-4o) solves only 16.3% of the end-to-end set.

Key Contributions

• The paper introduces SUPER, the first benchmark for evaluating LLM-based agents on setting up and executing tasks from low-profile ML and NLP research repositories, comprising 45 end-to-end problems, 152 sub-problems, and 604 automatically generated problems.
• The paper defines evaluation measures for task success and progress by comparing answers to gold solutions, checking landmark states such as completed training stages, and using successful execution of key scripts for automatically generated problems.
• The paper evaluates proprietary and open-source LLM agents with file-editing tools on SUPER and finds that GPT-4o solves only 16.3% of end-to-end expert tasks and 46.1% of sub-problem scenarios, with weaker performance on repository-level exploration than on narrow debugging tasks.

Introduction

In empirical ML and NLP research, reproducibility often depends on running open source research repositories, but many low profile repositories require complex environment setup, dependency fixes, data configuration, bug resolution, and script execution. Existing LLM coding and agent benchmarks usually target simple function generation, popular libraries, isolated debugging, popular repositories, or single script ML experiments, and they do not fully test whether agents can set up and execute arbitrary research repositories while producing correct result metrics. The authors introduce SUPER, a benchmark built from 45 expert curated end tasks, 152 masked sub problems, and 604 automatically generated tasks from diverse low profile repositories, and they evaluate agents using final metrics, partial landmark states, and successful script execution, showing that even strong agents still struggle with repository understanding and experiment automation.

Dataset

• The authors use the SUPER benchmark, which contains three purpose-built subsets: an Expert set for validated end-to-end research tasks, a Masked set for finer-grained sub-problems derived from the Expert set, and an Auto set for automatically generated tasks that can support agent development and improvement.

• The Expert set is built from research code repositories linked to papers in the Papers with Code database. The authors filter for repositories with text modality and selection from 2021 or later, then manually write tasks centered on running experiments from repository READMEs or available scripts. Some tasks are made harder by requiring a new dataset or model configuration, using Hugging Face Hub datasets or provided Google Drive links.

• Each Expert task includes a target GitHub repository, a task definition, the required metric or output and its format, and implementation instructions. These instructions reduce ambiguity and lower compute needs, for example by using small models such as gpt2-small, limiting datasets to the first 10 examples, or running a single epoch.

• Expert annotations are collected through Upwork workers experienced with ML and NLP experiments. Workers run tasks in Google Colab and submit solution notebooks, reported answers, the repository commit hash, and installed dependency lists. The authors review each solution, add unspecified decisions back into the task specification, and discard solutions that cannot be reproduced in their Jupyter-based environment.

• The environment stores state across executions using a Jupyter notebook engine that supports both Python commands and system shell commands, so each execution cell can build on previous Python variables and return an observation string.

• The Expert set contains 45 tasks, each paired with a gold answer and an average of 3 manually extracted landmark output patterns. Gold solutions average 44.3 non-comment lines of code and 14.4 notebook cells, reflecting the long trajectories needed for the full tasks.

• The Masked set contains 152 sub-problems extracted from the 45 Expert solutions. The authors group relevant notebook cells into masked blocks, define a narrower remaining goal, pre-execute independent context cells, and ask the agent to solve the masked portion. Each sub-problem includes a text description and a pre-executed code prefix, with accuracy and landmark metrics inherited from the Expert set.

• The Auto set contains 604 automatically generated tasks across unique repositories. The authors filter Papers with Code repositories using heuristics and LLM checks to remove repos that cannot run experiments, require GPU resources, or depend on external LLM APIs. They then prompt GPT-4o with README content to generate experiment-running tasks specifying the goal, dataset, model, and script.

• The Auto tasks are less rigorously validated than the Expert and Masked sets. A manual sample showed 81 percent feasibility, with failures mainly caused by missing resources or flawed task definitions. Because there are no expert gold solutions, evaluation uses a heuristic Script-Executed metric that checks whether the required script runs without exceptions for a minimum duration.

• The paper uses the data primarily for evaluation rather than a reported fixed training split or mixture ratio. The Expert and Masked sets provide validated benchmarking data, while the Auto set is positioned as a larger automatically generated pool for developing and improving agents. Repository popularity is used as a difficulty proxy, with a median of 14 GitHub stars for the source repositories.

Method

The authors construct the SUPER benchmark as a multi-stage benchmark building pipeline centered on executable research repository tasks. The overall workflow moves from repository collection to expert task creation, solution annotation, and then to focused subproblem extraction. It also includes an automatic task generation route intended for broader development and agent iteration. Refer to the framework diagram:

A core part of the benchmark is a stateful execution environment built around Jupyter notebooks. In this setup, each agent action is equivalent to executing a notebook cell. The cell may contain Python code, bash commands, or both. Environment state is preserved across executions, so variables, packages, files, and other runtime objects defined in earlier cells remain available to later cells. Each execution returns an observation string, allowing the agent to observe results, errors, logs, or metric outputs and then decide the next code snippet. This design supports realistic research repository workflows, where agents must install dependencies, edit configurations, load data, launch scripts, and diagnose runtime failures within one coherent execution state.

For the manually written Expert set, the authors begin by collecting research code repositories associated with papers and filtering to recent, text-based projects. They then manually inspect repositories to identify a concrete experiment that can be run or adapted. Tasks are defined around a target repository, a natural language instruction, a required output metric, an expected answer format, and implementation instructions. The instructions are deliberately included to make the task sufficiently specified and to improve reproducibility. They also constrain the task so that it can be solved with limited computational resources, such as by requiring only small models, limited data loading, or single epoch runs. Although these restrictions reduce computational cost, they can introduce practical challenges, such as finding the data loading logic, changing hyperparameters, or adapting GPU oriented scripts to CPU only execution.

Expert solutions are gathered through a vetted human annotation process. Candidate annotators are tested on a pilot task before being accepted. They solve tasks in a notebook style environment and submit the solution notebook, final answers, repository commit used, installed dependency versions, and explanations of undocumented decisions. The authors then review each solution to confirm that it follows the task, runs in the target Jupyter environment, records implicit choices, and produces stable results across repeated runs, with a tolerance around $10^{-2}$10−2. Solutions that cannot be executed or repaired in the environment are discarded.

To create finer grained evaluation items, the authors derive a Masked set from the expert gold solutions. End-to-end repository tasks can be long and provide sparse success signals, so the authors remove selected code blocks from a solution and define a smaller problem around the removed part. Each masked subproblem includes a goal statement and a prefix of already written or pre executed code. The agent is then asked to complete the missing step, such as fixing an error, resolving a dependency issue, adapting a script to CPU execution, or making a training run succeed after prior setup steps. Refer to the subproblem extraction diagram:

The automatic task generation route extends the benchmark beyond manually validated tasks. The authors use an LLM to generate candidate tasks from filtered repositories. The filtering stage removes repositories that are likely unsuitable because they lack runnable experiments, require unsupported hardware, or depend on external model APIs. For the remaining repositories, the LLM is prompted with the README file and asked to produce an experiment running task, including the goal, dataset, model, and target script. These automatically generated tasks are easier to scale but less controlled than the Expert set. They often emphasize dependency installation, experiment startup, and repository setup, and may involve more complex or incomplete environment requirements.

For the automatic tasks, evaluation cannot rely on expert final metrics or manually verified landmarks in the same way as the Expert and Masked sets. Instead, the authors use a heuristic based on whether the requested script can be executed without unresolved exceptions for a minimum amount of time. This signal helps distinguish successful setup and launch from immediate failures, silent crashes, or tasks that stop before meaningful execution occurs.

Experiment

The experiments evaluate agents on SUPER in a stateful notebook environment, using outcome and landmark metrics for expert end-to-end tasks and masked subproblems, and a script execution proxy for automatically generated tasks. Findings show that current LLM agents often make partial progress but rarely complete expert tasks, while masked subproblems provide finer-grained validation of technical components and show that strong models with file-editing tools perform best. The automated tasks support scalable development by producing consistent rankings, and error analysis reveals that agents handle concrete error-driven fixes more reliably than open-ended data or configuration changes, whereas reflection-based retries give only modest gains.

SUPER differs from related code execution benchmarks by combining repository understanding, repository setup, outcome-based evaluation, and use of low-profile repositories. It also includes expert tasks, masked subproblems, and an automated set drawn from many unique repositories. SUPER covers key benchmark challenges omitted by other datasets, including setting up less popular repositories and checking outcomes. An editing command improves agent landmark hits and final answers over a variant without direct file editing. Retry after reflection gives only a small performance gain, suggesting some failures require better reasoning rather than repeated attempts.

SUPER is organized into three complementary sets: expert-written problems, expert-solved masked sub-problems, and automatically generated problems. The first two sets include reference solutions and are evaluated against expert solutions for benchmarking and analysis, while the automatic set is larger, lacks reference solutions, and uses a proxy metric for agent development. The automatically generated set is the largest and targets development and agent improvement. Masked problems provide easier and more focused sub-problems derived from expert solutions. Proxy evaluation tracks execution success more closely than exact answer accuracy.

The masked sub-problem set is dominated by goal, data, and dependency categories, with data tasks requiring the longest gold solutions. Error analysis shows agents perform more reliably on concrete error-driven categories such as CPU, issues, and dependencies than on open-ended data, configuration, and goal tasks. Data sub-problems are common and involve substantially longer gold solutions than most other categories. Agents are weaker on configuration and goal tasks than on categories tied to explicit errors or compatibility fixes.

On the Expert task setting, GPT-4o-based agents outperform GPT-4o-mini and Llama 3.1 70B on both complete-task accuracy and landmark progress. SWE-agent with GPT-4o achieves the strongest accuracy, while ReAct-Super with GPT-4o achieves the highest landmark score. The results indicate that agents still rarely solve these tasks fully, but they often make partial progress that may still be useful for repository setup. GPT-4o agents show clear advantages over GPT-4o-mini and Llama 3.1 70B. ReAct-Super improves landmark progress over ReAct when paired with GPT-4o. Weaker open-source and smaller proprietary models show much lower partial progress and accuracy.

Results on the Auto tasks show that GPT-4o setups, especially React-Super, achieve the strongest script-executed performance, while GPT-4o mini shows a large drop for SWE-Agent. The relative ordering of models and agents remains mostly consistent with their ranking on the Masked set, supporting the Auto set as a useful benchmark for future development. React-Super with GPT-4o produces the highest script-executed score among the listed agent and model combinations. SWE-Agent performs strongly with GPT-4o but very poorly with GPT-4o mini. The model and agent ordering on the Auto set is broadly consistent with trends observed on the Masked set.

The experiments evaluate SUPER across expert, masked, and automatically generated repository tasks to assess agents on repository understanding, setup, and outcome-oriented execution. They find that direct file editing helps agents more than simple retry, and agents handle concrete error or dependency issues more reliably than open-ended goal, data, and configuration tasks. GPT-4o-based agents, especially ReAct-Super, achieve the strongest overall results, with full task completion remaining rare despite useful partial progress, while the Auto set largely mirrors the ranking patterns of the expert and masked sets.