One-sentence Summary

Researchers at University of Waterloo and Google introduce ReproBridge, an agent workflow that builds reproducible ML artifacts comprising refined codebases, environment setups, experiment launchers, and reproduction guides, and ReproBridge-Bench, a benchmark of 1,029 tasks from 387 experiments across 36 papers; across four evaluated models, this workflow raises mean reproduction success from 76.4% to 92.6%, narrows the cross-model spread from 21.6 to 3.9 percentage points, and lowers API spending by 19.7%.

Key Contributions

• The paper introduces ReproBridge, an agent workflow that packages recovered reproduction procedures into four standardized reusable components: a refined codebase, environment setup, experiment launchers, and a reproduction guide.
• ReproBridge-Bench provides 1,029 reproduction tasks from 387 experiments across 36 papers, with five human-verified reference scores per task and a separate Auditor that checks execution evidence.
• Experiments across four models show that refined artifacts raise mean reproduction success from 76.4% to 92.6%, reduce downstream API spending by 19.7%, and narrow the cross-model success-rate spread from 21.6 to 3.9 percentage points; ablation results identify code and environment removal as the largest sources of loss.

Introduction

Reproducing machine learning experiments is difficult because papers and released code often leave critical execution details implicit, such as compatible dependencies, intended configurations, data-loading repairs, and the mapping from run outputs to reported scores. Coding-agent benchmarks expose this burden through unresolved dependencies, missing implementation details, and incomplete plans, while consulting papers recovers only part of what is needed. The authors argue that preserving the procedures and decisions recovered during reproduction is as important as releasing code. They introduce ReproBridge, which packages four components: a refined codebase, environment setup, experiment launchers, and a reproduction guide. They also contribute ReproBridge-Bench, covering 1,029 tasks from 387 experiments across 36 papers with five human-verified reference scores per task, and show that refined artifacts improve coding-agent reproduction success by 5.5 to 23.2 percentage points while reducing downstream API spending by 16.8% to 23.7%.

Dataset

The authors build a curated reproduction benchmark from machine learning papers that report main experimental results and have accessible official implementations.

• Sources and paper selection

  • The collection starts from 742 ICML 2026 Spotlight papers.
  • The authors identify 241 papers with code and non-purely theoretical content, then randomly sample 50.
  • They exclude 6 papers for artifact or code availability issues, 4 for compute requirements, and 4 for closed-model access, leaving 36 retained test papers.
  • They also use 10 development papers, 5 from NeurIPS 2026 and 5 from ICLR 2026, to develop the ReproBridge and evaluation workflows.

• Task definition and curation

  • A task is one own-method measurement in a main table, identified by dataset, metric, reporting statistic, and location.
  • Only selected main-table variants are retained.
  • Baselines, stand-alone ablations, figures, and appendix-only results are excluded.
  • One experiment can emit several task scores.
  • The 36 papers yield 1,463 curated tasks from 502 experiments.

• Reference collection

  • Five human reproducers work on separate servers, starting from the same pinned raw commit, with separate execution directories and coding-agent sessions.
  • They cannot reuse another participant's generated scripts, measurements, refined artifacts, or downstream reproduction traces.
  • Coding-agent assistance is allowed, but humans inspect the scripts, configurations, edits, and measurement extraction actually used.
  • Before aggregation, the five reproducers' execution setups are cross-verified against the paper's specified setup and the common task definition.
  • Verifiers fix unspecified repetition and seed choices before inspecting scores, and they must not tune procedures to approach the published value.

• Numerical compatibility and filtering

  • Each reference observation measures the specified task statistic.
  • The five observations estimate the reference mean and variation.
  • A reported range is represented by its midpoint.
  • The compatibility interval is a 95% prediction-style interval around the reference mean, with a 1% tolerance floor.
  • A published score is compatible if it falls inside that interval.
  • Of the 1,463 tasks, 1,029 tasks, or 70.3%, have numerically compatible reference results and form the downstream evaluation set, spanning 387 experiments.
  • The remaining 434 tasks distinguish numerical disagreement from execution and resource barriers.
  • Tasks whose reference range is at most 10% of the absolute published score are considered stable; higher-variability tasks exceed that threshold. Both categories enter downstream evaluation and are pooled for the main comparison.

• Scale and usage

  • The evaluated set contains 631 full-procedure tasks and 398 evaluation-only tasks.
  • Hardware usage includes 730 GPU tasks and 299 CPU-only tasks.
  • Evaluating an explicitly permitted official checkpoint does not require repeating its training.
  • Evaluation alone cannot establish completion of a training task.
  • The benchmark includes 586 lower-is-better and 443 higher-is-better measurements, all assessed by the same two-sided compatibility rule.
  • A score beyond either bound remains numerically incompatible, including a score that improves on the published value.

• Metadata and traceability

  • Reference submissions must link each score to the starting code commit, executed scripts and edits, environment and hardware, configuration and seeds, raw outputs, and score extraction or aggregation procedure.
  • Published targets remain visible in the paper and task specification, but the Auditor checks that submitted values come from new executions.
  • An allowed checkpoint may serve as input, but a shipped score file never counts as a measurement.

Method

The authors introduce ReproBridge, a framework that transforms the process of reproducing experimental results into a reusable artifact for future researchers or agents. The system is designed to preserve both the executable steps and the evidence behind execution choices, ensuring that reproduction work can be utilized without access to the original construction conversation.

The construction workflow is organized into three distinct stages: Specify, Execute, and Preserve. In the Specify stage, the Intake role maps inputs and tasks, while the Planner role specifies measurements, resulting in detailed task specifications. The Execute stage involves the Builder preparing the environment and the Reproducer executing and recording the process to generate execution evidence. Finally, the Preserve stage utilizes the Distiller to document decisions and the Packager to verify the handoff, producing reusable procedures. Throughout this process, an Independent Reviewer inspects plans, runs, and the final handoff to request revisions and rechecks, ensuring the integrity of the artifact.

To ensure that a later reproducer knows what code to run, how to prepare the environment, how to obtain measurements, and why specific choices were made, the authors design a four-component Refined Reproducible Artifact (RRA). The first component is the refined codebase, which repairs execution defects and supplies missing experiment support. The second is the environment setup, specifying compatible dependencies and data or checkpoint acquisition. The third component consists of experiment launchers that connect tasks to configurations, execution scope, outputs, and measurement extraction, distinguishing scored runs from smoke tests. The fourth component is the reproduction guide, which includes a change record explaining settings, paper-code discrepancies, reporting statistics, and resource or numerical sensitivities.

The guide connects instructions to their rationale by recording the source location, the gap, the adopted resolution, sensitivity, and affected tasks for each ambiguity. The change record distinguishes between execution fixes, correctness fixes, and convenience changes while retaining pitfalls and open choices. This structured approach ensures that the delivered environment matches the one used for construction measurements and that all execution decisions are explicit before their results are known. When paper and code conflict, agents follow the paper's setup and flag the discrepancy, grounding the guide and change record in stored specifications and justified repairs.

Experiment

The evaluation compares raw released code with refined artifacts across four models on 1,029 tasks from 36 papers, using independent auditing of execution evidence. Refined artifacts consistently improve reproduction success and lower token and API costs, with gains persisting across papers, stricter numerical criteria, and different auditors. Qualitative analyses show the artifacts help by clarifying measurement specifications, preserving executable procedures and repaired environments, and fixing bugs, which reduces downstream reconstruction; ablations indicate code and environment components contribute most while guides and launchers also matter. Repeated reuse can eventually offset construction API costs, supporting the approach for shared, repeated reproduction.

Across 1,463 curated tasks, the majority produced reference measurements compatible with published scores, with stable reproductions as the largest single category. A smaller share was not directly usable due to numerical disagreement, execution blockers, or resource and access limits. The compatible outcomes form the downstream evaluation set. Stable reproductions account for nearly half of curated tasks, and together with variable reproductions they provide roughly seven in ten usable downstream tasks. Numerical mismatches and execution blockers each affect a comparable minority of tasks, while resource or access limits are less frequent.

The construction workflow assigns distinct roles that map inputs, plan measurements, build environments, execute, distill guidance, package artifacts, and review evidence. Persistent outputs link the stages, with phase gates checking required evidence and independent review challenging unsupported claims at planning, reproduction, and final handoff. Execution requires file-backed measurements and explicit failure reasons, while delivery includes clean-shell verification. Persistent outputs connect stages, from input manifests and task specifications through execution records, reproduction guides, and change records. A checkpoint reviewer operates in contexts separate from the roles being reviewed and returns localized findings for revision and rechecking.

Refined reproducible artifacts improve success for every tested model, with larger gains for models that had lower raw success. They also reduce token consumption and API spending, including failed attempts. The success-rate spread across models narrows sharply, and gains persist under equal paper weighting and stricter numerical compatibility. Refined artifacts raise success by 5.5 to 23.2 percentage points, while the cross-model spread narrows from 21.6 to 3.9 percentage points. Token use falls by 6.3 to 19.8 percent and API spending falls by 16.8 to 23.7 percent, with savings including failed attempts.

On a hard reproducibility benchmark, refined artifacts achieve the highest success rate and the lowest median time and spending per successful capsule. VERITAS-patched artifacts perform between raw code and refined artifacts. Refined artifacts match or exceed raw-code success in every assessed area. Refined artifacts yield the highest success rate and reduce median reproduction time by more than half compared with raw code. Spending per successful capsule is lowest for refined artifacts, with VERITAS sitting between raw and refined performance.

Under primary Codex scoring, refined artifacts improve success over raw outputs for every listed model in both paper-macro and task-level evaluations. The gains persist with equal paper weighting and a stricter task-level interval, with improved papers outnumbering worse papers across models. This supports the value of protocol-guided construction and execution reuse beyond raw code performance. Task-level evaluation shows the largest raw-to-RRA gain for Gemini 3.8 Flash and the smallest gain for Opus 5. Paper-macro improved/tied/worse counts favor RRA for all models, with improved papers far outnumbering worse papers.

Using 1,463 curated paper tasks, the workflow first identifies compatible measurements, with stable reproductions forming the largest group and stable plus variable reproductions supplying roughly seven in ten usable tasks. The construction process enforces role separation, phase gates, file-backed execution, clean-shell verification, and independent checkpoint review. Refined reproducible artifacts improve success for every tested model, narrow the cross-model spread, and reduce token use and API spending including failed attempts. On the hard reproducibility benchmark they achieve the highest success and lowest median time and spending per successful capsule, while under Codex scoring these gains hold across paper-level and task-level evaluations, with improved papers consistently outnumbering worse ones.