One-sentence Summary

Researchers from DreamX Team, Alibaba Group, Beijing University of Posts and Telecommunications, UNSW Sydney, and colleagues introduce LoopArena, a benchmark that evaluates how well a model Controller guides a separate fixed Worker through long-running coding tasks across three execution-scoped settings, reporting a best Strict Success Rate of 24.69%, Spearman’s ρ = 0.9747, and a 64.4% average inference-cost reduction.

Key Contributions

• The paper introduces LoopArena, a benchmark that isolates a Controller model's long-horizon loop guidance by keeping the coding Worker and control interface fixed and scoring next-step instructions, verification decisions, and stopping decisions from structured run summaries.
• LoopArena provides three complementary evaluation settings: execution-validated next-step contract selection without running the Worker, repeated control over a selected task slice, and paired full-task execution from the original repository state. The benchmark data and evaluation code are made public.
• Experimental results show a best observed Type III Strict Success Rate of 24.69% and a paired Type II reduction in estimated inference cost averaging 64.4%, while preserving similar Controller ordering relative to full tasks (Spearman's ρ = 0.9747). A fixed-goal policy does not improve over unguided execution, indicating effective loop control must adapt to the evolving run.

Introduction

Loop Engineering reflects a shift toward letting a model manage a separate coding agent over long-running tasks, rather than requiring a developer to inspect results and write every next prompt by hand. This matters because after many steps a plausible partial result can be mistaken for completion, and the next useful instruction may need to shift from implementation to verification, recovery, or stopping. Most prior coding benchmarks evaluate the final repository state or the complete agent system, so they do not directly isolate a model’s ability to guide another coding agent at runtime. The authors introduce LoopArena, a benchmark where the evaluated model acts as a Controller over a fixed Worker and execution setup. It evaluates runtime loop control at three complementary levels: low-cost individual control decisions, task-slice runtime guidance, and end-to-end long-horizon repository-level tasks.

Dataset

The authors construct the LoopArena benchmark, which evaluates a coding agent's Controller at three distinct scopes, using tasks sourced from SlopCodeBench (SCBench) and BeyondSWE. SCBench supplies long-horizon iterative coding tasks, while BeyondSWE covers a broader range of software-engineering challenges.

• Type I: Contract selection. Each instance is a single control decision extracted at a restorable point in a Controller-guided trajectory. The input is an Evidence Packet and four candidate Loop Contracts. A correct answer is determined by executing all four candidates under two predeclared replay schedules from the same restored state, keeping only items where the same candidate uniquely wins under both schedules according to the success-and-cost rule. Once validated, evaluating a new Controller requires a four-way choice with no Worker execution.
• Type II: Condensed coding task. A task slice taken from one coherent development stage of a full task. It begins from a prepared intermediate workspace and asks the Controller–Worker loop to complete that stage. A slice is retained only if the starting workspace fails at least one stage-introduced requirement and the source-provided completed state passes all expected requirements. Each Type II case is paired with its corresponding Type III task for matched comparisons.
• Type III: Full coding task. The original full task executed from its initial state. The Controller manages the complete run from first investigation through implementation and verification to the final stop decision.

Sources and filtering:

• Source tasks come from SCBench (native checkpoints provide Type II slices) and BeyondSWE (manually divided into coherent stages using official repairs and tests).
• Type I items are anchored immediately before a recorded Controller decision; alternatives are frozen before any replay outcome is observed. Items without a unique replay-consensus winner are discarded.
• Type II slices must satisfy the workspace requirement gap described above; otherwise they are dropped.
• All subsets undergo automated consistency checks and LLM-assisted review to verify task, workspace, and evaluator consistency, and to ensure model inputs do not expose solution code, scoring results, or future events from the source trajectory. Task and slice selections are fixed before formal evaluation.

How the data is used: LoopArena assesses Controller ability at three levels: a single contract decision (Type I), a task slice (Type II), and the full task (Type III). The paired Type II and Type III cases enable direct comparisons of evaluation cost and Controller ordering across the two scopes. The benchmark’s sizes and further statistics are reported in the paper.

Method

The authors design LoopArena to evaluate and guide long-horizon coding tasks through a structured control cycle. The framework maintains a persistent Worker conversation throughout each run. The Worker is the sole component equipped with coding tools, following a native ReAct loop to inspect and modify the repository, run checks, and execute assigned work. When the Worker completes a segment, control returns to the harness, pausing the persistent conversation and initiating a control cycle.

At the start of each control cycle, the harness generates a temporary Reporter agent from a copy of the accumulated Worker conversation. The Reporter shares the same model configuration as the Worker but is restricted to read-only tools. It produces a four-part account of the run, detailing the task context, completed work and current state, available verification evidence, and remaining issues. Material claims in the report cite the corresponding Worker turns. The Reporter describes the current state without deciding subsequent actions, and it cannot execute code or modify the repository.

For an executable task $i$i at control cycle $k$k, the harness packages the report and cited Worker turns into an Evidence Packet $x_{i,k}$xi,k​, which serves as a structured, read-only summary for the Controller. Let $\pi$π denote the Controller model and $h_{i,k}$hi,k​ represent its conversation history before the decision, encompassing earlier Packets and Contracts. At each control point, the Controller receives the latest Packet alongside this history but lacks direct access to the workspace or coding tools. It then produces a Loop Contract $c_{i,k}$ci,k​:

$$c_{i,k} = \pi(x_{i,k}, h_{i,k})$$ci,k​=π(xi,k​,hi,k​)

The Contract records the decision to advance the work, request focused verification, or stop. If the Controller chooses to proceed, the Contract provides the Worker with a bounded next assignment and specifies when control should return to the harness. If the Controller chooses to stop, the harness sends the current workspace to the task evaluator.

To systematically evaluate this framework, the authors construct a benchmark using full coding tasks from SlopCodeBench and BeyondSWE, organizing them into three distinct settings.

Type III evaluates a full coding task from its original state, retaining the original specification, starting state, development process, and evaluator. Type II pairs a full task with one task slice beginning from a prepared intermediate workspace. For Type II, the case describes the work required for that specific stage, and its evaluator checks the requirements that should hold upon completion. A slice is retained only when the starting workspace fails at least one requirement introduced by the stage, and the source-provided completed state passes all expected requirements. Type I constructs execution-validated control questions at restorable points in Controller-guided trajectories. Each question is anchored immediately before a recorded Controller decision, using the Evidence Packet as context. The recorded Loop Contract is retained as one of four candidates, alongside three complete alternatives. A candidate becomes the correct option only when it is the unique winner under two predeclared matched replay schedules according to a success-and-cost rule.

Experiment

LoopArena evaluates controller ability at three levels: Type I tests a single control decision by selecting among four pre-executed Loop Contracts, while Type II and Type III score repeated Controller-Worker control over a condensed task slice and its paired full coding task. The shared Worker and reference policies allow matched comparisons across these settings. Results show that full-task control remains difficult, fixed goal restatement helps on bounded slices but not on full tasks, and the lower-cost Type II setting preserves the full-task ordering of evaluated controllers. Type I reveals meaningful differences in individual control decisions, with simple shortcuts failing to approach controller-level accuracy.

The benchmark compares three settings: Type I isolates individual control decisions without worker execution, while Type II and Type III share the same source tasks and executable controller-worker loop. Type II reduces worker turns, control cycles, and average inference cost relative to Type III while preserving similar controller ordering. Full-task Type III success remains limited, and fixed control helps in the bounded Type II setting but not in full tasks. Type II cuts average estimated inference cost relative to Type III and uses substantially fewer worker turns and control cycles per run. Controller ordering in Type II closely matches full-task Type III ordering, with no strictly ordered controller pair reversing. Type I responses all parse to one candidate, and deterministic shortcuts score well below controller accuracy.

Evaluated controllers achieve only modest full-task success, with the strongest Type III result below one quarter. Type II task-slice evaluation preserves a controller ordering close to Type III while reducing estimated inference cost by about two thirds on average. Fixed control helps over the shorter task slice but not on full tasks, and Type I contract decisions are well above deterministic shortcuts. Full-task control remains difficult: Type III strict success among controllers peaks below one quarter. Type II strongly reduces inference cost relative to Type III while maintaining similar controller rankings under the main criterion. Fixed control improves Type II success over no control but gives no gain on Type III. Type I contract accuracy is substantially higher than all tested deterministic shortcuts, which peak near one third.

The benchmark compares three settings: Type I isolates individual control decisions, while Type II and Type III share source tasks and an executable controller-worker loop, with Type II using shorter task slices. Type II substantially reduces inference cost, worker turns, and control cycles relative to Type III while preserving a similar controller ordering. Fixed control improves success in the bounded Type II setting but not in full tasks, and full-task Type III success remains limited. Type I contract decisions parse reliably and perform well above deterministic shortcuts.