Shanghai AI Lab Benchmark Finds Large Models Cannot Reliably Engineer Proteins

Shanghai AI Laboratory, Tsinghua University's Institute for AI Industry Research, and Tsinghua's School of Pharmaceutical Sciences have released PFArena, a protein engineering benchmark within the laboratory's open science-evaluation framework. Building on the lab's AMix protein foundation models, the benchmark uses 202 real protein assays and more than 600,000 candidate records from public and curated deep mutational scanning datasets to test six protein language models, six general large language models, and five science agent combinations across four progressively harder tasks. It evaluates whether models can generate mutation candidates, rank multi-mutation combinations, and use experimental information under constrained budgets, moving beyond static mutation prediction toward practical laboratory decision-making. PFArena finds that current AI systems cannot yet reliably engineer proteins. The best model, when allowed to propose only 40 candidates, recovers fewer than two high-scoring mutations on average, and results weaken as mutation complexity increases. Protein language models are stronger when no target-specific experimental data is available because they embed sequence, structural, and evolutionary knowledge. General large models and agent systems become more effective once experimental anchors exist, using evidence to reason about complex mutation combinations. Iterative feedback substantially improves performance: with the same 40-candidate budget, successive rounds of experimental feedback raised GPT-6 Astra's recall of high-quality mutations from 2.54% to 4.65%. The findings support the view expressed by Shanghai AI Laboratory director Zhou Bowen that artificial intelligence is accelerating quickly in software and coding while scientific task completion remains slower. The team argues that the path forward lies in general-specialist fusion, combining specialized protein models, general reasoning, tool use, and closed-loop experimental feedback. The researchers introduced AMixKit, a lightweight toolkit that allows general models to call protein analysis functions and perform multi-step candidate assessment, improving performance on the benchmark. They identify three priorities for future work: better modeling of non-additive fitness landscapes, budget-aware multi-round experiment design, and autonomous systems that can propose hypotheses, select assays, and update strategies from measured data. The work positions PFArena as a step toward AI that can move from reading protein information to supporting real-world scientific discovery.