One-sentence Summary

ScienceIDE, introduced by PhAI-Labs and Qwen, converts scientific repositories into programmable, agent-learnable environments that support task generation, execution, and scientific verification, and yields the $\text{PhAI-IDE-72B}$PhAI-IDE-72B, $\text{PhAI-IDE-9B}$PhAI-IDE-9B, and $\text{PhAI-IDE-4B}$PhAI-IDE-4B models, which demonstrate gains in held-out scientific-code repair and selected general-purpose benchmarks, providing evidence of positive transfer from scientific experience to broader capabilities and laying the foundation for shared scientific intelligence.

Key Contributions

• Introduces ScienceIDE, an infrastructure that converts expert-defined scientific cases and acceptance criteria into executable environments for scientific agents, enabling task generation, execution, and verification from existing scientific code repositories.
• Trains PhAI-IDE-72B, PhAI-IDE-9B, and PhAI-IDE-4B on verified interaction trajectories, establishing a shared foundation for supervised fine-tuning, reinforcement learning, and evaluation.
• Reports gains in held-out scientific-code repair and on selected general-purpose benchmarks in code, reasoning, and knowledge, providing evidence of positive transfer from scientific experience to broader capabilities.

Introduction

Language models improve with verifiable experience, yet scientific discovery remains hard because tasks are open ended, require hypothesis testing, and depend on complex toolchains that are difficult to reproduce. Prior benchmarks evaluate scientific coding but do not offer the production infrastructure needed to convert repositories into trainable environments, especially for accelerator porting where each port is hand written and tied to vendor-specific defaults. The authors introduce ScienceIDE, which turns expert-defined checks and tolerances into executable environments, decoupling verification from task authoring. This lets a single environment support repair, implementation, and acceleration tasks, with a registry of over 2,800 tasks. They address the bottleneck of turning scientific code into reusable learning experience by having agents generate and validate tasks, producing graded trajectories for supervised fine-tuning and reinforcement learning.

Method

The authors introduce ScienceIDE, a framework that transforms scientific expertise into reusable agent experience by defining scientific responsibilities and acceptance criteria within executable environments. Task factories generate challenges from these environments, and agent interactions provide the evidence required for evaluation and learning.

As shown in the figure below:

The construction unit is a scientific module within a versioned codebase. A module owns a coherent scientific responsibility and executable coverage rather than simply grouping related files. An environment packages an approved module with its runtime and scientific checks. Scaling is achieved by adding modules and codebases while reusing their acceptance criteria across task families.

Construction begins with production scientific software. An agent inspects a pinned upstream revision, dependencies, and build assumptions, then builds the source and runs official tests and examples. These runs expose output formats, numerical variability, and execution hazards that inform the environment boundary. The agent proposes modules defined by scientific responsibility and executable coverage, identifying inputs, outputs, algorithm stages, and supporting tests. A domain expert reviews the decomposition and coverage. The agent then implements the environment-specific experiment adapter and proposes scientific outputs for review. An approved module is packaged with an editable workspace, checks, and a private verifier.

Refer to the framework diagram:

Once a module is approved, its upstream unit tests, regression tests, and shipped example problems are surveyed together. Each case is traced to the module responsibility and either retained as a check, explicitly excluded, or identified as a gap. The unit of reward is a check, consisting of fixed inputs and graded outputs paired with a pass policy. The pass policy takes two forms. A pointwise policy compares every graded value using the bound $|y - \hat{y}| \le \epsilon + \rho |y|$∣y−y^​∣≤ϵ+ρ∣y∣, where exact equality is the special case $\epsilon = \rho = 0$ϵ=ρ=0. This grades physical observables, aligning unordered collections by an identity carried in the output before comparison. An invariants policy compares quantities such as moments, distributions, conserved values, or integral norms when no pointwise bound can contain the run-to-run variation. Nominal and variant initial conditions make this choice measurable, where the variant perturbs the smallest sufficient set of active inputs to reveal numerical sensitivity.

A factory combines reusable authoring procedures with an environment's scientific context to generate tasks. Shared procedures handle reversible edits, execution, and artifact assembly, while local rules identify active paths and meaningful transformations. The agent proposes semantic edit sites and objectives, while the curator and domain expert retain decisions about observables and acceptance. Deterministic procedures expand approved rules into mutation or excision candidates.

As shown in the figure below:

The authoring taxonomy includes seven categories: Acceleration, Repair, Discovery, Reproduction, Integration, Calibration, and Implementation. Repair and Implementation expose automatable candidate expansion through reversible mutation and excision. The other categories provide interfaces for expert-specified objectives, data, procedures, or resource constraints.

Factory proposals become tasks only after executable evidence establishes that they are observable, solvable, and trustworthy. AI authors can propose repairs, implementations, accelerations, reproductions, or other transformations, but a proposal is not evidence of validity. Factory rules turn AI-proposed transformations into candidates. Each candidate is tested in the final grading environment. For an injected repair, a known-valid witness must pass, the unfixed baseline must leave headroom, and the change must produce a check failure removed by the reference repair.

Refer to the framework diagram:

Scientific checks expose partial accomplishment. Repair scores normalize this against the defective starting point using the formula:

where $r$r measures scientific agreement and $f$f is the unfixed build's score. Packaging also screens answer leakage and records execution-integrity failures.

A validated task exposes a common episode interface. The agent receives an editable workspace, inspects code and scientific inputs, makes changes, runs experiments, and submits artifacts to a private verifier. The harness records actions, observations, check rewards, execution status, and resource use.

As shown in the figure below:

Evaluation holds scientific tasks and interaction budgets fixed while measuring success. Supervised fine-tuning consumers select trajectories and supervise agent actions, including tool calls. Reinforcement learning consumers connect policy-controlled rollouts to verifier rewards. Tasks can be selected by environment, domain, family, or measured difficulty for held-out evaluation and training curricula.

For reinforcement learning, the verifier scores an agent's repair by compiling the modified code and running the underlying scientific simulation. This signal is used directly as the RL reward. The training stack runs generation and optimization concurrently on disjoint GPUs. Rollout and training run concurrently, with token-level truncated importance sampling correcting the resulting one-step policy staleness.

For a prompt group of $G = 8$G=8 trajectories with rewards $R_i$Ri​, the group-relative advantage is calculated as:

Tokens are updated with a PPO-style clipped ratio whose bound is asymmetric ($\varepsilon_{\text{low}} = 0.2, \varepsilon_{\text{high}} = 0.3$εlow​=0.2,εhigh​=0.3), allowing rarely sampled repair actions to retain room to gain probability mass. The loss is aggregated by token mean.

In long-horizon scientific tasks, an episode can end because the repair finishes, the turn cap is exhausted, or the response budget is exhausted. When a budget-truncated episode receives zero reward in a group with positive mean reward, the advantage becomes negative, suppressing its generated tokens. To prevent the policy from learning to simply shorten trajectories rather than solving the scientific task, the authors separate the role of a truncated trajectory as a reward observation from its role in policy optimization. Budget-truncated trajectories are kept in the group baseline but masked out of the loss. Their reward still enters the advantage calculation, ensuring completed episodes receive a positive advantage when their reward exceeds the group mean.

Experiment

The experiments evaluate scientific task execution in the ScienceIDE benchmark, comparing fifteen agents on an 85-task hard subset and finding that the top models achieve around 63-67% success, with most agents scoring below 40% and considerable variation in time and resource efficiency. Fine-tuning on scientific demonstration trajectories improves repair reward across held-out codebases and transfers positively to public code, reasoning, and knowledge benchmarks. Online reinforcement learning with verifier feedback further boosts held-out repair reward by 2.0-2.4x, but requires masking budget-truncated trajectories in the policy loss to avoid a pathological shortening shortcut that otherwise collapses performance.

Fine-tuning with online verifier feedback and truncation masking improves scientific code repair reward across multiple model-environment pairs, with notable gains in PLUTO-Particles-Dust and PLUTO-RMHD/ResRMHD. The method prevents the policy from being penalized for long exploration, leading to higher rewards and reduced budget truncation in held-out tasks. The largest reward gain is on PLUTO-Particles-Dust with Qwen3.5-4B, improving from 0.0 to 0.33. LAPS shows moderate improvement from 0.31 to 0.50 on PLUTO-RMHD/ResRMHD, and MITgcm-Biogeo gains 6.25 percentage points. Masking truncated trajectories out of the policy loss while keeping them in the group baseline prevents reward collapse and encourages longer, more effective repair attempts. Training with the truncation mask leads to higher held-out rewards and fewer budget cutoffs, without shortening responses.

Fine-tuning with online verifier feedback and truncation masking consistently improves scientific code repair rewards across multiple model-environment pairs, with the largest gain observed on PLUTO-Particles-Dust (0.0 to 0.33) and solid improvements on PLUTO-RMHD/ResRMHD and MITgcm-Biogeo. Masking truncated trajectories from the policy loss while retaining them in the group baseline prevents reward collapse and encourages longer, more effective repair attempts, resulting in higher held-out rewards and fewer budget cutoffs without shortening responses.