One-sentence Summary

KAIST, Microsoft Research Montréal, and Microsoft AI introduce ProgramDistill, a benchmark that evaluates coding agents on features discovered through interaction with fully functional reference applications, using its mine-craft-patch pipeline to generate $4{,}063$4,063 tasks from $1{,}975$1,975 replay-verified behaviors across 26 applications, and demonstrates that frontier agents including GPT-6 Astra and Claude Opus 5 achieve only $49.2\%$49.2% and $28.8\%$28.8% success on cumulative workflows while partial-application reconstruction success drops from $100\%$100% to $64.0\%$64.0% and from $96\%$96% to $32\%$32% as restoration depth increases from 1 to 8.

Key Contributions

• Introduces ProgramDistill, a benchmark converting reference-guided web development into verifiable software-engineering tasks, built from 1,975 replay-verified behaviors and 4,063 tasks across 26 applications via the automated mine-craft-patch pipeline.
• Factors applications into replayable behaviors with prerequisite relationships, enabling controlled difficulty through restoration depth, from atomic repairs to full-application reconstruction, and supporting both evaluation and future curriculum-based training.
• On nine frontier coding agents, GPT-6 Astra and Claude Opus 5 achieve 49.2% and 28.8% success on full-application reconstruction, while partial reconstruction success drops from 100% to 64.0% and 96% to 32% as restoration depth increases from 1 to 8, demonstrating the benchmark’s diagnostic value.

Introduction

Large language models are increasingly used as coding agents that inspect and modify source code, but these systems typically assume the desired behavior has already been specified in an issue or test. In practical web development, however, developers often need to infer intended behavior directly from a working reference, such as an earlier product version, a prototype, or a demonstration video. Prior work, like ProgramBench, has begun treating running software as a specification, but only as a whole-program reconstruction target, missing the stateful dependencies and prerequisite lineages that characterize interactive web applications.

To address this, the authors introduce ProgramDistill, a framework that factorizes working interactive web applications into verifiable software-engineering tasks. It automates a mine-craft-patch pipeline that mines reproducible browser-interaction traces, crafts tasks by masking source implementations, and patches applications by having coding agents recover missing functionality from a live reference. The framework constructs 4,063 replay-verified tasks from 1,975 behaviors across 26 applications, spanning atomic repair, cumulative repair, and full-application reconstruction. Evaluation of nine frontier coding agents reveals that performance declines significantly as restoration depth increases, exposing a growing mismatch between reconstruction burden and agent effort, with observation and validation strategies emerging as a critical dimension alongside implementation skill.

Dataset

The dataset is built from 26 web applications, which come from two complementary sources: self-contained applications adapted from the OSWorld suite, and real-world open-source projects and SaaS clones from public repositories. The former offer controlled interaction diversity, while the latter contribute larger codebases, deeper workflows, nontrivial state, and heterogeneous architectures.

For each application, the authors serve two instances: a fixed production build as the reference instance and a development-server version as the editable instance. The reference stays unchanged during task construction and repair, while masking and agent edits apply to the source code and are reflected in the editable instance via hot reload. Both instances expose the same interface, so behavior observed in the reference can be reproduced and evaluated in the editable instance. To ensure reliable replay, the pipeline resets application state before collection and replay and uses a shared deterministic clock across database, backend, and frontend.

Using the full pipeline with GPT-5.6 Sol as the construction model, mining proposes 2,800 candidate goals across the 26 applications. Of these, 2,350 are collected as interaction traces. After clean-state replay, 2,165 traces reproduce successfully, and 1,975 survive the final replay check to become verified behavior traces. The mined prerequisite trees reach a maximum lineage depth of 17 (mean 3.40), a maximum width of 60, and a mean branching factor of 1.78. Lineage depth counts the number of traces from the root to a target, including replay bridges used to establish prerequisite state.

From the verified traces, the pipeline constructs 4,063 repair tasks: 2,862 atomic tasks and 1,201 cumulative tasks. Across both types, 1,997 use logic-only masks (removing the implementation but leaving the UI) and 2,066 use logic-and-UI masks (removing both behavior and UI). Masks are validated through counterfactual checks: a mask is accepted only if the masked application builds and launches, all prerequisite traces remain replayable (pass-to-pass), and the target behavior fails (fail-to-pass). A separate LLM-based mask-depth critic rejects superficial changes like toggling feature flags or removing call sites while leaving the implementation intact.

Cumulative tasks combine masks along a lineage. The pipeline first attempts deterministic composition, since all masks are defined against the same intact baseline; non-overlapping edits are combined directly and overlapping deletions are merged by union. A git three-way merge provides an independent consistency check. If the results disagree, an LLM-based merge agent resolves overlaps. Of the 1,201 cumulative tasks, 629 were composed deterministically and 572 required the merge agent. The fraction requiring the agent increases with restoration depth, from 25.6% at depth 2 to 89.7% at depth 8 and 100% above depth 8, and all agent-assisted merges succeeded.

For each validated task, reversing its masking diff yields a gold patch that restores the original implementation. The gold patch is validated by applying it to the masked application and requiring the complete task lineage to pass replay verification, providing an end-to-end positive control.

Evaluating all 4,063 tasks for every model would be too expensive, so the authors define a fixed evaluation suite, ProgramDistill-300, with 300 tasks. They stratify tasks by restoration depth using quotas of 50, 45, 45, 40, 35, 30, 30, and 25 for depths 1 through 8. Within each depth, tasks are selected round-robin across applications to preserve coverage of all 26 apps and prevent larger task pools from dominating. The resulting suite covers 269 distinct lineages and contains 50 atomic and 250 cumulative tasks, with 140 logic-only and 160 logic-and-UI masks.

Method

The authors introduce the mine-craft-patch pipeline, a synthetic task generation framework that orchestrates multiple LLM agents to synthesize verifiable tasks from web applications without relying on human-written issues, tests, or behavioral annotations.

As shown in the figure below:

The pipeline operates in three main stages: Mining, Crafting, and Patching. To support this pipeline, each application is set up as a live, self-contained execution environment accessible through a common browser interface. The environment provides a stable reference for behavior observation, an editable application for task construction and repair, and a reproducible runtime for replay-based verification. For each web application, a fixed production build serves as the reference instance, while a development-server version acts as the editable instance. To ensure reliable replay, the pipeline resets application state before collection and replay and uses a shared deterministic clock across the database, backend, and frontend. A shared Playwright-based browser helper provides a common interaction interface for mining, replay verification, and repair, resolving elements using stable observable attributes rather than volatile DOM identifiers.

The Mining stage discovers what an application can do and records those behaviors as replayable specifications for task construction and evaluation. By exploring the live application with access to its source, the pipeline builds a bank of verified traces, each containing browser actions, expected outcome signals, and an optional parent trace. Parent links preserve the prerequisite context needed to reproduce dependent behaviors. Given the current trace bank, the pipeline proposes new behavior goals grounded in source and UI evidence. An LLM agent explores the live application to pursue the proposed goal, forming an exploratory trace. Because initial exploration may include detours, the pipeline re-collects the behavior from a clean reset, providing the exploratory trace to the LLM agent as additional prompt context to elicit more concise reasoning trajectories. The resulting trace contains the actions, expected signals, and parent reference needed for deterministic replay. Only traces that pass a replay verifier are admitted.

As illustrated in the figure below:

Behavioral dependencies are organized into a prerequisite trace tree. For instance, an editor behavior is explored after replaying account setup and board or card creation. The resulting trace activates the column title editor and opens the card details, with expected signals checking the title field values and the presence of the Description label.

The Crafting stage turns replay-verified behaviors into repair tasks by removing their source implementations and checking that the resulting failures are confined to the intended targets rather than their prerequisites. The authors control the resulting tasks along two dimensions: mask scope and task composition. Mask scope determines how much of a feature is removed, either as a logic-only mask that leaves the user interface in place or a logic-and-UI mask that removes both. Task composition determines whether the task targets a single behavior or multiple dependent behaviors along a prerequisite lineage.

As shown in the figure below:

The crafting process begins with trace-conditioned masking, where Crafting agents identify the source implementation responsible for the observed behavior of a target trace and propose a mask. For atomic task validation, a mask is accepted only if the masked application builds and launches successfully, all prerequisite traces remain replayable, and the target behavior fails. These conditions yield a fail-to-pass objective for the target and pass-to-pass checks for its unmasked prerequisites. A separate LLM-based mask-depth critic rejects superficial changes. Validated atomic masks are then combined along a lineage to form cumulative tasks. Combining the corresponding masks yields a cumulative mask $M_{L} = m_{j_{1}} \oplus \dots \oplus m_{j_{r_{L}}}$ML​=mj1​​⊕⋯⊕mjrL​​​, where $\oplus$⊕ combines the accepted source modifications. Unmasked traces serve as replay bridges that establish prerequisite state without becoming repair targets themselves.

The Patching stage evaluates whether a coding agent can recover application behavior by observing a working reference and implementing it in an incomplete application. The authors consider two settings: partial-application reconstruction, which starts from the masked repository produced by crafting, and full-application reconstruction, which starts from a minimal executable scaffold. In both settings, the agent interacts with the reference without access to its source. Evaluation uses a verifier with recorded actions, selectors, and expected signals unchanged. For repair targets indexed by $j_{1} < \dots < j_{r_{L}} = d$j1​<⋯<jrL​​=d, the authors define a BinaryScore and a ChainScore:

$$\operatorname{BinaryScore}(L_{d}, \Delta) = V(A[M_{L}, \Delta], L_{d}), \quad \operatorname{ChainScore}(L_{d}, \Delta) = \frac{1}{r_{L}} \sum_{k=1}^{r_{L}} V(A[M_{L}, \Delta], L_{j_{k}}).$$BinaryScore(Ld​,Δ)=V(A[ML​,Δ],Ld​),ChainScore(Ld​,Δ)=rL​1​k=1∑rL​​V(A[ML​,Δ],Ljk​​).

The binary score is 1 only when the complete task lineage passes, while the chain score gives partial credit for the recovered prefix. For full-application reconstruction, atomic recovery measures the fraction of atomic behavior tests that pass, while cumulative workflows use binary and chain scores to evaluate the complete lineage or recovered prefixes.

Experiment

The experiments evaluate two reconstruction settings using the ProgramDistill benchmark. In partial-application reconstruction, nine frontier models repair masked applications by observing a working reference, with GPT-6 Astra achieving the highest binary score of 84.3%, followed by Claude Opus 5 and GPT-5.6 Sol. Performance degrades with restoration depth, and analysis shows Astra's success stems from an observation-intensive, edit-light workflow, while deeper tasks correlate with reduced per-target effort and more unrestored code. Full-application reconstruction, tested on three models across twelve applications, proves more challenging, with Astra reaching 58.98% atomic and 49.15% cumulative recovery; failure analysis reveals that 59.2% of errors stem from behaviors never observed in the reference, and agent validation often fails to recheck the exact failing workflow after final edits.

Full-application reconstruction remains difficult even for the strongest models, with cumulative recovery consistently lower than atomic recovery across all models. The best model, GPT-6 Astra, leads on both metrics, and performance varies widely by application, from 81.8% cumulative recovery on MailHub to 5.9% on Baserow. GPT-6 Astra outperforms both other models on atomic and cumulative recovery, with the same ranking as in masked-repair tasks. Cumulative recovery is always lower than atomic recovery for every model, showing that completing atomic behaviors does not guarantee full workflows. Performance varies sharply by application; for example, MailHub shows high cumulative recovery while Baserow shows very low recovery. Failure analysis shows that 59.2% of atomic failures are due to behaviors not observed in the reference, and many remaining failures involve wrong state or observable form. Agent validation often checks related functionality instead of rechecking the exact failing workflow after final edits, leaving mismatches undetected.

Full-application reconstruction runs are substantially longer than partial-application runs, averaging around 700 agent steps and reaching up to nearly 2,000. Opus 5 stands out by issuing multiple browser CLI calls per agent step, leading to a much higher number of browser state returns per step compared to other models. Opus 5 collects about 7.9 browser state returns per agent step on average, while Astra and Sol collect about 1.6 and 1.2, respectively. Full-application reconstruction averages around 700 agent steps, with maximum runs exceeding 1,900 steps. Opus 5 shows a wide range in browser state returns, from 1,427 to 12,821, whereas Astra and Sol have narrower ranges.

Full-application reconstruction remains challenging even for the strongest models, with cumulative recovery consistently below atomic recovery across all models, indicating that completing individual behaviors does not guarantee entire workflows. GPT-6 Astra leads on both metrics, though performance varies sharply by application, from 81.8% cumulative recovery on MailHub to just 5.9% on Baserow, and failure analysis shows most atomic failures stem from behaviors not observed in the reference, with agent validation often missing mismatches by checking related functionality instead of rechecking the exact failing workflow. These runs are substantially longer than partial-application ones, averaging around 700 agent steps and reaching nearly 2,000, with Opus 5 distinct for issuing multiple browser CLI calls per step and collecting roughly 7.9 browser state returns per agent step, far more than Astra and Sol.