Command Palette
Search for a command to run...
27B AI's Ability to Reproduce scientists' Papers Surpasses GPT-5.5/Claude Opus 4.8; Faraday Explores long-term Scientific innovation.
The reproducibility of a research paper is the cornerstone of a scientific knowledge system. It not only ensures the reliability of existing research results but also provides a basis for further experiments.
However, various scientific fields are currently facing a "reproducibility crisis," especially machine learning. Theoretically, AI agents based on Large Language Models (LLMs) could provide a scalable solution to this crisis; however, in practice, reproducibility challenges existing AI agents in three main areas—first,By definition, the problem of reproducing a paper is that the information is incomplete. That is, the paper only provides a lossy compression of the complete research process that led to a discovery, and therefore inevitably omits some details.Secondly,Existing AI agents are mainly trained extensively on well-defined, closed problems, while paper reproduction requires agents to conduct open-ended exploration; finally, for general reproduction tasks, there is no clear reward function that can be used for continuous "climbing" optimization.
In response to this challenge,A research team from Inherent Labs has trained an AI scientist agent called Faraday that can reproduce scientific research papers.This LLM-based agent uses Codex GPT-5.5 as a tool, similar to how human AI researchers use programmed agents. In a criterion-based evaluator evaluation, Faraday outperformed Claude Opus 4.8 and GPT-5.5 on a 73% in-distribution machine learning task and a 60% holdout AI for Science task.
It is worth noting that Faraday has only 27 billion parameters, yet it can guide a model with 5 trillion parameters, thereby achieving a practically significant performance improvement compared to using this larger model alone. Qualitative analysis of a single run shows that Faraday employs a more scientifically sound approach.
The related research findings, titled "Training AI Scientists to Replicate Research," have been published as a preprint on arXiv.
Research highlights:
* This study constructed an automatically generated task space containing 310 graph reproduction tasks from 100 papers in Machine Learning and AI for Science, spanning from 1990 to 2026.
* This study proposes a stable GRPO post-training method suitable for long-term, unverifiable tasks. This method includes a rating criterion evaluator for each task, aggregation of multi-sample evaluation results, and a round-by-round credit allocation mechanism.
* This study trained an agent with 27 billion parameters capable of using coding agents as tools (CAT). Experiments, both quantitatively and qualitatively, demonstrate that Faraday exhibits greater scientific rigor.
Paper address:
https://hyper.ai/papers/2608.13331
Build an automatically generated task space Replica
To train Faraday, researchers constructed a scalable task space called Replica. The Replica task space consists of 242 training tasks and 68 test tasks, which are drawn from 100 well-known machine learning (ML) and AI for Science papers.Each task requires the agent to reproduce a result figure from the paper. The agent can obtain the original paper after figure masking and is subject to a 60-minute time limit, while only being able to use a 1/7 MIG slice of an H200 GPU.
The system provides the agent with a containerized environment, pre-loaded with useful research libraries, and offers internet access, system prompts, and task prompts. If an experiment in the paper cannot be completed within the allotted time, the prompts will instruct the agent to scale up the experiment as faithfully as possible to the original experiment before completing it.
* Training tasks are drawn from machine learning papers published between 1990 and 2026.
* The test tasks are drawn from AI for Science papers published between 2012 and 2026.
It is worth noting that these tasks are all automatically generated, thus the entire task space has good scalability. For a given paper, the system transforms it into a task through three visual language processing stages, all driven by Gemini 2.5 Pro—first, the scanning stage identifies all resulting figures and their captions in the text; then, the localization stage determines the bounding boxes of the figures in the repair loop of the LLM validator; finally, the figure is irreversibly masked from the PDF. A complete task consists of three parts: the caption, the extracted original figure ("gold plot"), and the paper with the figure masked.
The research team will manually review each task and filter out tasks of lower quality, such as those with insufficient chart coverage, charts that are not the result charts, or incorrect captions.Each paper contributes 1–13 tasks, with a median of 2 tasks.
A stable GRPO post-training method suitable for long-term, unverifiable tasks
Simple agent framework
Agent Harness provides an interface between large language models (LLMs, where both input and output are tokens) and the environment (input actions/output states):
* Available capabilities and context
The agent executes operations through five function calls: `apply_patch`, `read_file`, `list_dir`, `grep_files`, and `shell`. Faraday can use the shell tool to detach background processes from the current session, allowing it to execute multiple rounds of operations while running multiple commands in parallel. Tool calls within a round are executed concurrently, and their results are appended to the history in the order they are called. The run terminates if a context overflow occurs, the 16K token limit per round is exceeded, or an inference error occurs; even incomplete runs are evaluated like other runs. Otherwise, the run terminates when Faraday responds directly without calling any tools, or when the specified actual runtime limit is reached.
* Using programmable agents as tools (CAT)
Faraday is equipped with a cutting-edge programming agent and uses it as a tool. A wrapper script runs the Codex CLI in a non-interactive manner. Faraday can invoke this script via a shell tool and obtain a log of the programming agent's commands, outputs, and messages, including the time taken for each step. By default, consecutive calls continue using the programming agent's previous session state; however, Faraday can also choose to reset the context or run multiple programming agents in parallel. The wrapper script enforces a deadline, which can be set individually by Faraday for each request. If the deadline is exceeded, a partial log of interactions is returned. The model used by the programming agent is a runtime parameter: GPT-5.4 mini is used for most of the training process; GPT-5.5 is used in the final stage and during evaluation.
Post-training scheme
To obtain Faraday, researchers post-trained Qwen3.6-27B within the Faraday agent framework, using a modified GRPO based on the Replica task space. They employed LoRA fine-tuning with a rank of 128 (k = 128); trained the adapter on all linear projection layers using a 128K token context window and a fixed learning rate of 6 × 10⁻⁶. Each step of the Adam optimizer involved drawing a batch of 10 tasks from a Replica training set partition containing 242 tasks, with each task corresponding to 8 runs.
The task sampling method ensures that each batch evenly covers the entire corpus's year range; at the same time, each training epoch accesses each task exactly once, thus avoiding the dominance of data from a particular scientific period in a certain parameter update.
Long-term training stability
Post-training requires long-term reinforcement learning in an unverifiable domain, a setting known to be prone to training instability and even model collapse. Two factors contributing to this instability are the high variance of the reward signal and uniform credit assignment. To address this issue, researchers made two modifications to the evaluator during the training phase.
First, it uses the average of three independent evaluation results to calculate the reward at the rollout level. Second, it requires the judge to generate round-by-round weights to assign credit to each round of operations during the rollout. In this way, credit can be reallocated within a single rollout without changing the overall magnitude of parameter updates.
Faraday is more rigorous in both qualitative and quantitative aspects of scientific research.
Before evaluating Faraday's capabilities, the experimental results first verified that the Replica task itself is sufficiently challenging.Claude Opus 4.8 is one of the best-performing baseline models in the study, but none of the leading Coding Agents, including Claude and GPT-5.5, have reached saturation on Replica tasks.The study also found that the more recent the publication date of a paper, the more difficult it is to reproduce; AI for Science papers are generally more difficult to reproduce than traditional machine learning papers. The research team speculates that this may be related to the lower information density of newer papers in the model's pre-training data, and also because recent research usually requires higher computational resources, making it more difficult to achieve reasonable scaling down.
Faraday's ability to reproduce the paper is superior to Claude and Codex.
Researchers compared Faraday with various baseline models across the entire Replica task distribution (see figure below).Faraday's performance is comprehensively improved compared to the base Qwen model, both in training and testing tasks.In distributed tasks, Faraday outperforms Claude and Codex in 73% tasks; in out-of-distribution tasks, Faraday outperforms Claude and Codex in 60% tasks.
Further decomposition of the judge's total score into different sub-dimensions reveals that...Faraday outperformed the baseline model in experimental depth, assertion reproducibility, and visual fidelity, while performing comparably to Claude in scientific integrity and realization fidelity.(See the image below).
To examine whether Faraday's advantages can be obtained solely through prompt word optimization, researchers performed 24 rounds of automatic prompt word optimization on the Codex baseline model, comparing the final prompt words with the original Codex and Faraday. The results are shown in the figure below:
The optimized suggestion words did not achieve any meaningful performance improvement, so the gap with Faraday still exists.The optimized prompts did identify the specific failure patterns in the results, but they did not solve these problems: the performance improvement brought by post-training does not seem to be obtained solely through prompt engineering.
Faraday demonstrated more rigorous research capabilities at the qualitative level.
To understand the specific improvements Faraday makes compared to the Claude and Codex baseline models, researchers manually analyzed some specific results, focusing on cases where Faraday's scorer scores significantly outperformed the highest scores from Claude and Codex. The table below shows some representative examples:
Two patterns repeatedly emerged in the analysis: First, Faraday truly realizes the mechanism that the experiment is trying to test, while the baseline model often directly hardcodes the expected output or resorts to an oversimplified approach, thus failing to truly reproduce the core conclusions corresponding to the charts; Second, Faraday is more comprehensive in terms of experimental scope, able to reproduce more of the content in the original experiment, while avoiding unnecessary deletions.
Generalization ability of "imagined replications"
The researchers further evaluated Faraday's generalization ability for "imaginary" reproduction (as shown in the figure below). They required Claude Opus 4.8 to randomly select five papers from both the Replica training and test sets, generating two variants for each paper, and then evaluate Faraday and Codex GPT-5.5 on these tasks.
On 19 out of 20 tasks, the judges favored Faraday's results over Codex's. In a weaker sense, Faraday not only demonstrated better reproducibility than state-of-the-art models but also exhibited stronger innovation capabilities.However, the researchers also cautiously pointed out that their rating scale has never been validated for "imaginative" tasks, so this conclusion still needs further verification in future research.
Conclusion
If past AI scientists were more like engineers who could conduct experiments, then Faraday's CAT paradigm attempts to further equip AI with the scientific judgment to know what to do. The abilities to determine research direction, reasonably define the scope of experiments, and judge the reliability of reproducible results—once these are embedded in the outer layer of the intelligent agent, can be continuously amplified as the underlying cutting-edge models are upgraded.
More importantly, this paradigm also offers new insights into the development and security of AI capabilities: allowing a relatively small model to handle scientific research judgments, while a more powerful model handles engineering execution. In the future, competition among AI scientists may no longer be just about the size of model parameters, but will shift towards how to develop better scientific research judgments, tool utilization, and human-machine collaboration capabilities.
References:
