One-sentence Summary

The authors develop Replica, a scalable task space for paper replication, and introduce an auto-generated rubric-based judge with low noise that agrees with human assessments of replication quality; post-training Faraday, a 27B-parameter AI Scientist agent that uses coding agents as tools, surpasses Claude Opus 4.8 and GPT-5.5 on held-out replication tasks and adopts a more scientifically principled approach, providing a stepping stone toward long-horizon scientific innovation.

Key Contributions

• Introduces Replica, a scalable task space for paper replication.
• Develops an auto-generated rubric-based judge that provides low-noise reward signals and agrees with human assessments of replication quality.
• Post-trains Faraday, a 27B-parameter AI Scientist agent that uses coding agents as tools. Faraday surpasses Claude Opus 4.8 and GPT-5.5 on held-out replication tasks, and qualitative rollouts show a more scientifically principled approach.

Introduction

Replication underpins scientific knowledge, yet machine learning is experiencing a replication crisis. LLM-based AI agents could scale in silico replication, but existing agents struggle because paper replication is underspecified, requires open-ended exploration to infer missing details, and offers no definite reward for harnesses such as autoresearch or AlphaEvolve. Prior replication benchmarks also face a tradeoff between ease of evaluation and construct validity. The authors introduce Faraday, a 27B-parameter post-trained AI Scientist that uses a larger coding agent, Codex GPT-5.5, as a tool. Faraday is trained on Replica, a scalable space of 310 figure-replication tasks from 100 ML and AI-for-science papers, using a stable GRPO recipe with a rubric-based judge, multi-sample judge aggregation, and turn-level credit assignment.

Dataset

The Replica dataset is described as follows.

• Sources and scale

  • It contains 242 training tasks and 68 test tasks drawn from 100 well-known ML and AI-for-science papers.
  • Training tasks come from ML papers from 1990 to 2026; test tasks come from AI-for-science papers from 2012 to 2026.
  • Each source paper contributes between 1 and 13 tasks, with a median of 2.

• Task construction and filtering

  • Given a paper, three vision-language stages powered by Gemini 2.5 Pro scan for every main-text results plot and its caption, localize the figure bounding box inside an LLM-verifier repair loop, and irreversibly redact the figure from the PDF.
  • Each task is stored as a triple: the caption, the extracted figure as the gold plot, and the redacted paper.
  • All tasks are manually inspected. Low-quality tasks are removed, for example when a figure is insufficiently redacted, is not a results plot, or has an incorrectly identified caption.

• Task setup

  • An agent receives the redacted paper, the task caption, a system prompt, and a task prompt.
  • The setup includes a 60-minute limit, one-seventh MIG slice of an H200 GPU, a containerd container with research libraries, and internet access.
  • If the full experiment cannot be completed within the time budget, the prompt asks for a faithful scaled-down version.

• Dataset usage

  • The authors use the 242 tasks from ML papers as the training split and the 68 tasks from AI-for-science papers as the test split.
  • No mixture ratios are specified in this section; the task format is used directly as benchmark episodes where the target is the gold plot.

Method

The authors construct Replica as a scalable benchmark and training environment in which an agent must replicate a results figure from a machine learning or AI-for-science paper under constrained compute and time. The task generation pipeline is automated and uses three vision-language stages powered by Gemini 2.5 Pro. Given a source paper, the pipeline first scans for every main-text results plot and its caption, then localizes the plot through a bounding-box stage embedded in an LLM-verifier repair loop, and finally redacts the figure from the PDF. Each generated task is represented as a triple containing the caption, the extracted gold plot, and the paper with the figure removed. The authors manually inspect every task and filter out low-quality examples, such as cases where the figure is insufficiently redacted, the target is not a results plot, or the caption is incorrectly identified.

Because paper replication is inherently difficult to verify automatically, the authors design a rubric-based judge rather than relying on exact plot matching alone. A short hand-designed meta-prompt is used with Claude Opus 4.7 to generate task-specific rubrics. The gold plot is hidden from the rubric generator so that rubrics focus on the paper's scientific claims instead of over-indexing on axis ranges, formatting, or exact numerical values. Each rubric covers five dimensions: visual similarity to the original figure, support for the paper's scientific claim, whether the underlying experiment implements the described method, effective use of the compute budget, and scientific integrity. Since full-scale replication is often infeasible, rubrics explicitly reward faithful scaled-down experiments.

Rollouts are assessed by Codex GPT-5.5 acting as a judge. The judge receives the task-specific rubric, the redacted paper, the agent's workspace and tools, the replication codebase, git history, the full interaction trace, and the gold plot. It has ten minutes to inspect these materials and may re-execute the agent's code to verify the replication process. Each rubric dimension receives a continuous score between 0 and 1, and the overall rollout reward is the average across dimensions. During training, the authors sample the judge three times per rollout to reduce reward variance. They also instruct the judge to produce turn-level credit assignment weights, which are used later in the reinforcement learning update.

Human evaluation is used to assess whether the judge captures human research taste. Human experts are given the redacted paper, the gold plot and caption, the rollout transcript, and the resulting repository. They rank rollouts and justify their rankings while also explaining what the gold plot shows and how they would reproduce it. Participants are selected from PhD-level researchers with strong ML publication records.

The agent harness is designed to be simple, interpretable, and permissive. Faraday interacts with the environment through five function-calling tools: apply_patch, read_file, list_dir, grep_files, and shell. The shell tool supports background processes, enabling the agent to manage parallel work across many turns. Tool calls within a turn execute concurrently, and their results are appended in call order. The conversation history is linear and append-only. A rollout ends when the agent stops making tool calls, exhausts its wall-clock time, or encounters an error. In addition to these tools, Faraday can invoke a frontier coding agent as a tool through a wrapper around the Codex CLI. Successive invocations resume the coding agent's session by default, but Faraday may reset context or launch multiple coding agents in parallel. The wrapper enforces a deadline and returns a partial transcript if exceeded.

For post-training, the authors initialize Faraday from Qwen3.6-27B and train it in the Replica harness using a modified version of GRPO. They use LoRA fine-tuning with rank 128 and $\alpha = 128$α=128, adapters on all linear projections, a 128K-token context window, and a constant learning rate of $6 \times 10^{-6}$6×10−6. Each optimizer step samples 10 tasks from the 242-task training split, with eight rollouts per task. Tasks are sampled so that each batch spans the corpus year range evenly, and each epoch visits every task exactly once.

To stabilize long-horizon reinforcement learning, the authors use the mean of three independent judge evaluations as the rollout reward and redistribute credit inside each rollout using judge-generated turn weights. The judge produces a weight $u_k$uk​ for each turn, normalized as

where $n_k$nk​ is the number of tokens in turn $k$k. This normalization preserves the overall reward scale while allowing per-token advantages in GRPO to be scaled by the corresponding turn weight.

Experiment

The experiments evaluate a rubric-based judge and Faraday, a trained agent for paper replication on the Replica benchmark. Human expert rankings show that the per-task rubric judge aligns better with human preferences and is more consistent than a baseline judge, while Replica tasks remain challenging for frontier agents, especially for recent and interdisciplinary papers. Faraday outperforms Claude and Codex on most tasks, an advantage not achievable through prompt optimization alone, and qualitative analysis indicates it more faithfully implements experimental mechanisms and conducts more thorough experiments, with humans preferring its strongest rollouts.

Qualitative inspection of rollouts where Faraday most outperforms Claude and Codex shows Faraday tends to implement the mechanism an experiment is designed to test rather than hardcoding expected outputs, and it runs experiments with greater fidelity and scope. This pattern appears both for papers inside and outside Faraday's training distribution. In a focused human study, people preferred Faraday rollouts over both baselines in the majority of cases examined, though the study design does not support conclusions about average preference. Faraday implements the evolutionary self-improvement procedure in the Darwin-Gödel task while the baseline bypasses it by hardcoding a discovered agent. In the Voyager task, Faraday learns and transfers skills through a dedicated acquisition phase whereas the Claude baseline supplies a prepopulated library containing target skills. For the AI Scientist task, Faraday follows the paper-reviewing method more closely and at larger scale, and its reflection loop improves accuracy while the Codex baseline barely changes. In the ChemVAE task, Faraday more faithfully decodes optimized latent points back into molecules instead of relying primarily on nearest known molecules.

The evaluation compares Faraday against Claude and Codex through qualitative inspections of agent rollouts and a focused human study. Faraday tends to implement the intended experimental mechanism rather than hardcoding outputs, and it runs experiments with greater fidelity and scope; for example, it performs evolutionary self-improvement in Darwin-Gödel, acquires skills through a dedicated phase in Voyager, follows the reviewing method more closely in AI Scientist, and decodes latent points into molecules in ChemVAE. Human reviewers preferred Faraday rollouts in the majority of examined cases, though the design does not support average preference conclusions.