One-sentence Summary

Harvard University and Capable researchers introduce PG-LLM, a ProteinGym-based benchmark evaluating 13 general-purpose language models on 217 protein variant ranking tasks, revealing that Claude Opus 5 leads with $\rho=0.406$ρ=0.406, outperforming many sequence-only predictors without multiple-sequence alignments yet still lagging behind the specialist VenusREM ($\rho=0.523$ρ=0.523), while test-time compute gains taper.

Key Contributions

• The benchmark PG-LLM evaluates general-purpose language models on 217 protein-variant prioritization tasks from ProteinGym, using only the wild-type sequence, an assay description, and 50 mutants without alignments or structures.
• Comparing thirteen language models and 95 published predictors reveals that Claude Opus 5 attains a Spearman correlation of 0.406, surpassing 49 of the 95 predictors including most sequence-only methods, while remaining below the specialist VenusREM at 0.523.
• Increased test-time compute improves variant ranking across GPT, Claude, and Gemini models, but gains taper before matching specialist predictors, and LLM accuracy is largely insensitive to alignment depth, unlike sequence-only methods.

Introduction

Prioritizing protein variants for functional testing is a central challenge in protein engineering, where deep mutational scanning (DMS) provides dense maps of sequence–function relationships. Current state-of-the-art specialist predictors integrate evolutionary, structural, and protein-family information, but general-purpose language models could potentially streamline this task by interpreting assay objectives and reasoning over candidate sequences without a dedicated protein-modeling pipeline. Prior evaluation of this capability was limited to a single model family on a subset of ProteinGym assays and lacked comparisons with broader predictor landscapes or analysis of key performance factors. The authors introduce PG-LLM, a benchmark that compares thirteen general-purpose language models against 95 published protein predictors across all 217 ProteinGym substitution assays. They assess each model on a ranking task where only a wild-type sequence, an assay description, and a shuffled set of mutant sequences are provided, without alignments or structures. The work reveals that frontier models capture substantial protein-variant signal but do not yet match the strongest specialist systems, establishing variant ranking as an emerging capability of general-purpose language models.

Dataset

The authors construct the PG-LLM benchmark from the substitution component of ProteinGym v1.3, using all 217 assays across 186 proteins. Each assay contains variants with experimental fitness scores (oriented so that larger values indicate better performance). The benchmark comprises 148 assays with single substitutions and 69 assays with multi-mutant variants.

• Candidate-set construction For each assay, variants are sorted by their measured fitness, split into ten equally populated bins, and roughly the same number of variants are sampled from each bin. This stratified sampling ensures the candidate set spans the full performance range rather than concentrating near the mean. The sampled variants are then randomly shuffled and assigned unique identifiers.

• Draws Per assay, three independently sampled candidate sets (draws 1–3) are generated using fixed seeds 1–3. Every evaluated model and predictor is tested on the same variants within a draw. Draw scores are reported separately, and assay-level analyses average the available draw scores.

• Candidate-set size The primary leaderboard uses a fixed candidate-set length. Separately, the benchmark evaluates the effect of candidate-set size by testing three fixed draws at N = 10, 50, and 100 (as noted in the text).

• Prompt format Each prompt contains the protein and organism names, a short description of the measured phenotype, the direction that corresponds to better performance, the full wild-type sequence, and the shuffled list of full-length mutant sequences. The model must infer the relationship between sequence changes and the stated assay, compare all candidates jointly, and return a JSON ranking from highest to lowest predicted performance. The prompt deliberately withholds the source publication, mutation notation, experimental measurements, multiple-sequence alignments, and structural information.

• Usage The PG-LLM benchmark is a zero-shot evaluation suite. It does not provide training data. LLMs and published predictors are compared by their ability to rank variants on the same candidate sets. All aggregate results, assay-level predictions, and provider-visible traces are available at ProteinGymLLM.com, and the evaluation code is released through the PG-LLM GitHub repository.

Method

The authors design the PG-LLM benchmark to evaluate large language models on a specific protein engineering task: ranking a set of variant amino acid sequences by their expected experimental performance given an assay objective.

Task Formulation and Candidate-Set Construction Every prompt follows a standardized template that provides the protein and organism names, a short description of the measured phenotype, the direction corresponding to better performance, the full wild-type sequence, and a shuffled set of full-length mutant sequences. The prompt intentionally withholds the source publication, mutation notation, experimental measurements, multiple-sequence alignments, and structural data. The model must infer how sequence changes relate to the stated assay and return a single JSON ranking from highest to lowest predicted performance.

To construct the candidate sets, the authors utilize 217 assays across 186 proteins from the substitution component of ProteinGym v1.3. Variants within each assay are sorted by measured fitness and divided into ten equally populated bins. Approximately the same number of variants are sampled from each bin to ensure the candidate set spans the assay's measured fitness range rather than concentrating near its center. The variant order is then shuffled randomly, and each variant is assigned a unique identifier. For each assay, three independently sampled candidate sets, referred to as draws, are generated using fixed seeds.

Model Inference and Output Parsing The authors evaluate thirteen general-purpose language models under text-only conditions, without external tools, retrieval, or programmatic execution. Reasoning effort is configured at varying levels depending on the model, such as max reasoning for specific models and high or xhigh for others. Models are instructed to return a JSON object containing a "ranking" array of candidate identifiers. The parsing pipeline first searches for an exact double-quoted "ranking" array. If this fails, it extracts candidate identifiers based on the order of their first occurrence in the response. A response is considered eligible if it contains at least 80% of the expected identifiers, with any omitted identifiers appended after the explicit ranking in the prompt order.

Scoring, Aggregation, and Baseline Comparison For each benchmark cell, the returned candidate ranks are compared with experimental values using Spearman rank correlation, denoted as $\rho$ρ. To aggregate these scores, the primary leaderboard employs a nested-macro approach. The full nested-macro score is calculated separately for each draw and then averaged. Within a draw, repeated assays within each protein-functional category cell are averaged, proteins are averaged within each functional category, and equal weight is given to the five category means. This hierarchy prevents proteins with many assays or categories with many proteins from dominating the overall score.

The authors also compare the LLMs against 95 published predictors, comprising 46 sequence-only methods and 49 methods utilizing alignments, structures, or both. For each set of mutant sequences, every predictor's released score table is restricted to the selected variants, ranked, and evaluated using the same metric and aggregation pipeline.

Experiment

The PG-LLM benchmark evaluates general-purpose LLMs on ranking protein variants by expected fitness from assay descriptions and sequences, using Spearman correlation against experimental measurements. The best LLMs, Claude Opus 5 and GPT-5.6 Sol, surpass many sequence-only specialist predictors but fall below alignment/structure-based methods, and their accuracy drops with larger candidate sets while specialist models are unaffected. Test-time scaling brings diminishing returns, and LLM performance does not benefit from deeper evolutionary alignments, indicating reliance on different knowledge sources. These results demonstrate emergent protein reasoning in LLMs but not a replacement for specialist models.