One-sentence Summary

The authors introduce FRONTIERPHYSICS, an end-to-end physics research benchmark comprising 56 tasks created by 42 physicists across nine theoretical, computational, and experimental domains, and by combining deterministic verifiers with expert-authored rubrics it evaluates both correctness and publishability, finding that no configuration among 22 models in 26 configurations passes more than 46 of its 168 rollouts.

Key Contributions

• FRONTIERPHYSICS is an end-to-end benchmark with 56 tasks authored by 42 physicists from their own research, spanning nine domains across theoretical, computational, and experimental physics. Each task requires literature review, research planning, conducting research and experiments, and producing numerical outputs and a scientific manuscript.
• The benchmark combines deterministic verifiers for numerical results with expert-authored rubrics for manuscript quality, evaluating both correctness and publishability. Tasks undergo manual general review, domain-expert peer review, and failure-mode analysis to preserve open-endedness while reducing false positives and false negatives.
• Across 22 models in 26 configurations, no configuration passes more than 46 of its 168 rollouts. Manual trajectory auditing shows frontier agents' main weaknesses are insufficiently rigorous reasoning and limited expert intuition, rather than gaps in physics knowledge or execution errors.

Introduction

Large language models are moving from passive scientific assistants to tool-augmented agents that can search literature, formulate hypotheses, write and debug code, and produce technical reports. Evaluating this progress requires benchmarks that capture open-ended, multistage research, but existing research-level benchmarks target isolated capabilities, rely on closed-ended deterministic answers that models quickly saturate, or in full-paper reproduction settings reveal the original publication and limit open-ended inquiry. The authors introduce FRONTIERPHYSICS, a benchmark built by active physics researchers from projects they personally conducted, with manual review, domain-expert peer review, and failure-mode analysis to preserve open-endedness while reducing false positives and negatives. Each task gives an agent a high-level principal investigator-style prompt and requires both a scientific manuscript and deterministic numerical outputs, graded by a verifier and rubric-based reviewer agents.

Dataset

Dataset sources and scale

• FRONTIERPHYSICS contains 56 accepted research tasks from 42 physics researchers at 26 institutions, spanning 9 physics domains.
• The tasks were selected from 182 initial submissions through authenticity checks, domain peer review, and final admin approval.
• 18 tasks are based on published research co-authored by contributors, including four Nature papers and work in Nature Communications, Physical Review Letters, and Physical Review X.
• Contributors report an average of about 270 days of original research per task and more than 8,000 total hours of research effort.

Task structure and schema

• Each task reconstructs an end-to-end research project. The main instruction is stored in task.md, evaluation criteria are stored in rubric.json, and input data or supporting files are provided in the task environment when needed.
• Prompts may include important questions or directions, but no explicit methodology or solution pathway is specified.
• Each task requires deterministic artifacts, such as computed quantities, simulation outputs, or processed data, plus a final manuscript submitted as paper.pdf.
• The manuscript must meet peer-reviewed standards and present research context, methodology, results, and conclusions while incorporating the evidence and artifacts produced during execution.

Curation, filtering, and quality control

• Contributors must provide provenance evidence such as reference papers or patent links, project background, a research timeline, and time spent. The original project must represent at least two weeks of research effort.
• task.md and rubric.json are checked to be human-readable and genuinely human-authored. Purely AI-generated submissions are rejected.
• Domain reviewers verify that tasks are not over-specified and that rubrics encode real physics insight rather than generic or AI-like criteria.
• Contributors must perform at least three local runs with state-of-the-art agents and provide an initial failure-mode analysis. Reviewers inspect these traces for false positives and false negatives.
• Common issues include unjustified numerical tolerances and ambiguous instructions.
• Domain reviewers are mostly PhD students, 47 percent, and PhD holders, 37 percent. They are onboarded individually and their first reviews are audited.
• A final admin team of six physics PhD researchers approves and merges accepted tasks.

How the data is used

• The dataset is used as an evaluation benchmark for physics research agents rather than as a training corpus in the described sections.
• Agents receive the task instruction and environment files, then must complete the required artifacts and write the final paper manuscript.
• Assessment combines global rubrics, covering reward hacking, hallucination, literature synthesis, result validation, and evidence use, with task-specific rubrics designed by the contributing domain expert.
• The provided text does not specify a training split, mixture ratio, or separate validation partition.

Method

The authors establish a rigorous pipeline for task curation and quality review to ensure the benchmark reflects genuine, high-level physics research. The process begins with a general quality review where contributors must provide supporting information, such as reference papers or patent links, to prove they personally conducted the project. The underlying project must involve at least two weeks of research effort. The authors also verify that the task description and rubric contain genuine human input, rejecting purely AI-generated submissions.

Following the initial check, the authors implement a domain peer review system. Contributors who pass the first stage are invited to become domain reviewers, ensuring that every task is evaluated by an expert in a closely related field. Reviewers use a checklist to verify that the task prompt does not over-specify the problem, that the rubric encodes real physics insight, and that failure modes are properly analyzed through local runs with state-of-the-art agents. Once all checks pass, an admin team of physics PhD researchers approves and merges the submission.

The final task set comprises 56 tasks across 9 different physics domains, as summarized in the figure above. The authors note that for many of these tasks, the underlying research has been published in top-tier journals, representing thousands of hours of original research effort.

For the evaluation phase, the authors design a scheme that assesses both the objective correctness of an agent's results and the overall quality of its research process. Each task rollout combines programmatic deterministic verifiers with an Agent-as-judge. The deterministic verifier checks structured outputs against reference values. Subsequently, a reviewer agent evaluates the trajectory files and generated papers based on a predefined rubric. The authors utilize an agent rather than a single large language model call because many criteria, such as detecting reward hacking or hallucinated results, require examining the agent's full trajectory and code.

The rubric-based grading system divides criteria into blockers and non-blockers. Blockers act as binary gates representing requirements essential to scientific validity; an agent must satisfy every blocker and the deterministic verifier to pass. Non-blockers capture qualitative aspects and are scored on a three-level scale (0, 1, or 2). The task author assigns an integer weight to each criterion. The final score is calculated as:

where $\mathcal{N}$N and $\mathcal{B}$B denote non-blockers and blockers respectively, $s_i \in \{0, 1, 2\}$si​∈{0,1,2}, $w_i$wi​ is the criterion weight, $D$D is the deterministic verifier result, and $b_j$bj​ is the blocker result. A score of zero indicates the results are not publishable, while a non-zero score measures publication quality.

To understand why agents fail, the authors conduct a failure mode analysis of the agent rollouts, as detailed in the figure above. The analysis reveals that agents predominantly fail not due to fundamental physics mistakes, but rather due to a lack of rigorous thinking, missing expert intuition, and difficulties in synthesizing literature. Common issues include circular self-verification and failure to consider edge cases, highlighting the challenges agents face in conducting open-ended scientific research.

Experiment

These experiments evaluate a physics research benchmark by combining deterministic verification with rubric-based reviewer agents that use binary blockers and weighted qualitative criteria, while also validating grading through expert audits and checks against authors' own manuscripts. The main comparison across frontier models and agent harnesses shows that overall pass rates remain low, with no model clearing thirty percent, and higher compute cost generally correlates with better performance but not uniformly across model families. Reliability studies reveal that a small number of correct rollouts were initially failed due to grader or rubric issues, and reviewer rubrics align well with human expert reference work. Failure mode analysis indicates agents rarely fail because of physics knowledge errors; instead, their main weaknesses are rigorous reasoning, physical intuition, literature synthesis, and instruction following.

The compared benchmarks rely on tasks drawn from contributors’ own research and withhold the original reference, but their evaluation does not grade literature review or manuscript write-ups. Instead, evaluation uses deterministic verifiers, expert rubrics, or failure analysis, with failure analysis present in full for one benchmark, partial in another, and absent in a third. Physics task coverage is uneven, ranging from an entirely physics-focused set to a small physics subset. All listed benchmarks use tasks from the contributors’ own research and provide no reference to the agent. None of the listed benchmarks grade literature review or write-up quality; evaluation focuses on expert rubrics or deterministic verifiers. Failure analysis is documented fully for one benchmark, partially for another, and not at all for the third. Physics coverage varies widely, from an all-physics task set to a benchmark where physics is a small subset.

Across 56 tasks, all evaluated frontier agents record overall pass rates below 30 percent. Claude Opus 5.5 leads on score, deterministic pass rate, and overall pass rate, while deterministic pass rates are consistently higher than overall pass rates because blocking rubric criteria are stricter. Pass@3 exceeds Pass@1 for every configuration, showing that multiple rollouts improve the chance of at least one successful run. Claude Opus 5.5 has the highest overall pass rate and score among the evaluated agents. Deterministic pass rates are consistently higher than overall pass rates, indicating that many verifier-passing runs still fail blocking rubric criteria. GPT-6 Astra at maximum effort outperforms its xhigh setting on score and overall pass rate. The lowest overall pass rates occur with GPT-6.1 Terra and GPT-6 Sol; GPT-6 Sol has a higher Pass@3 despite a similar score.

The first analysis surveys benchmark design and finds that all compared benchmarks draw tasks from contributors’ own research and withhold references, but they differ in evaluation style, failure-analysis completeness, and physics coverage, with no benchmark grading literature review or manuscript write-up quality. The second experiment evaluates frontier agents across 56 contributor-derived tasks and shows that overall pass rates remain below 30 percent, with Claude Opus 5.5 leading; deterministic verifier pass rates are consistently higher than overall pass rates because blocking rubric criteria are stricter, and Pass@3 improves over Pass@1, indicating that repeated rollouts help. Overall, the results highlight uneven benchmark design and substantial headroom for agents on these research-style tasks.