One-sentence Summary

A multi-institutional team featuring Carnegie Mellon University and Tsinghua University introduces Benchmark Radar, a continuously updated living database and search engine that aggregates AI benchmarks from 37 sources and provides a web dashboard with score histories, Pareto frontier views, trend analyses, and a command-line interface for reproducible evaluation design and prior-art search.

Key Contributions

• Benchmark Radar is a living search engine and database that consolidates daily discovery of AI benchmark papers, repositories, and releases from 37 sources with a searchable catalog of 1,283 source records, model-card mentions, and score histories that retain source identities and citations.
• A full audit of the catalog covers 12,916 numeric observations across 790 records and examines benchmark saturation, adoption trends, and limits of score comparisons, supported by reproducible analysis and visualizations.
• A worked prior-art search example demonstrates how the system supports evaluation design by enabling queries to the catalog and inspection of benchmark evidence and reporting settings, and the release includes a web dashboard with a benchmark leaderboard, a Pareto frontier view of score against measured use, saturation and trend views, daily feeds, downloadable evidence, a command-line interface for offline queries, and reproducible analysis.

Introduction

The evaluation of large language models (LLMs) depends on benchmarks that measure coding, reasoning, and domain-specific capabilities, but finding a benchmark’s materials, usage history, and reported scores often requires manually searching across paper servers, code repositories, leaderboards, and vendor blogs. Existing catalogs and leaderboards provide partial coverage, yet connecting a new benchmark to its dataset or tracking how it is used in later model reports remains fragmented. The authors introduce Benchmark Radar, a living search system that aggregates benchmark records from public sources, links related entries with shared identifiers, and surfaces daily discoveries, model-report mentions, and score histories with documented evaluation settings, making the evidence behind model performance claims more discoverable and inspectable.

Dataset

The authors construct a benchmark catalog dataset that combines static source records with daily discovery observations. The data is not used to train a predictive model; it forms the underlying knowledge base for a browsing and search system.

Dataset composition and sources

• Source records are benchmark entries ingested from four core upstream sources (arXiv, Hugging Face Hub, GitHub Search, plus additional scholarly indexes, dataset hosts, repository releases, and institutional feeds).
• Discovery observations are collected daily from 13 direct connectors and 24 first-party feeds. They record mentions, releases, and updates found across public sources and are linked to artifacts via exact identifiers (DOIs, arXiv IDs, repository URLs). Discovery observations are kept separate from the benchmark record count.

Key details for each subset

• Source records (static catalog)
  • 1,283 records drawn from four sources; two sources may describe the same benchmark and both are retained.
  • 790 records carry 12,916 numeric score observations; 493 records have no numeric score, but 464 of those still link to at least one paper, repository, or dataset.
  • In the full catalog, 475 records link to papers, 506 to repositories, and 293 to datasets (a record can contain multiple link types).
  • Records are normalized to common fields (names, identifiers, artifact links, score observations, model identities, cited documents) and persist even when a score, date, or citation is absent. Reviewed identity links connect related records without merging their separate observations.
• Discovery observations (daily stream)
  • Each collection run scans a 48-hour window and requires that core sources are healthy before publication. Futuredated rows are removed.
  • Observations do not inflate the benchmark count; they populate a parallel history linked by identifiers.

Processing pipeline

• Record normalization: source records are mapped to a unified schema that standardizes names, identifiers, artifact links, model identities, and cited documents. The catalog retains entries with missing fields.
• Discovery cleaning: daily ingestion filters out futuredated observations and validates the health of core sources (arXiv, Hugging Face Hub, GitHub Search) before publication.
• Identity linking: related records are connected via reviewed identity links while keeping their observations and counts separate from the raw discovery stream.

How the data is used

• The processed source records and discovery observations feed a search interface that supports full catalog export and view‑specific display filters (Table 2).
• The resulting catalog enables browsing by date, reported score, and number of scored models (Figures 3 and 4), serving as the reference dataset for the benchmark tracking system rather than being a training corpus.

Method

The authors design Benchmark Radar to systematically collect, preserve, and retrieve benchmark records and discovery observations from public sources. The system architecture separates the benchmark catalog from the discovery history, ensuring that evaluation metrics remain distinct from temporal mentions and release updates.

As shown in the framework diagram, two primary input paths support the system publication pipeline. The benchmark catalog ingests data from registries and model reports, including model cards, technical reports, system cards, and release posts. These inputs contribute through a unified record structure. Simultaneously, the discovery history captures dated mention and release snapshots. Both paths converge into a shared evidence layer that maintains source record IDs, scores, model identities, citations, and links to discovery evidence. This shared layer subsequently serves web interfaces, downloads, and offline query clients using stable benchmark IDs.

The daily discovery collection operates by searching a 48 hour window across multiple public sources. The system relies on core sources such as arXiv, Hugging Face Hub, and GitHub Search, supplemented by additional routes covering scholarly indexes, dataset hosts, and institutional feeds. Each collection run records counts and errors, removes future dated rows, and requires healthy core sources before publication. Discovery observations, such as collected mentions and updates, are linked to artifacts using exact identifiers like DOIs, arXiv IDs, and repository URLs. These observations are retained alongside the catalog to provide historical context but do not inflate the benchmark catalog total.

To ensure data integrity before comparing measurements, the authors normalize source records into common fields. This process standardizes names, identifiers, artifact links, score observations, model identities, and cited documents. Reviewed identity links connect related records while preserving their separate observations and counts.

For candidate retrieval, the system employs a shared query service that supplies both CLI and HTTP interfaces with a consistent response format. Lexical search utilizes BM25F, a field weighted word matching score, applying bounded boosts for name and phrase matches. Each search result exposes matched and missing query words, their occurring fields, and the score components, ensuring that source membership does not alter the ranking. This design allows analysts or agents to inspect a record tasks, score settings, and citations, preserving the necessary evidence for suitability judgment.

Finally, the system conducts a comprehensive audit of the full population. The census process iterates through every record in the rebuilt catalog index, counting finite numeric score observations, scored models, and cited documents without applying date or score filters. Eligibility for summary calculations depends on specific measurement criteria, such as declared percentage units and known score directions, ensuring that rescaling or aggregator declarations do not falsely establish measurement scales.

Experiment

The evaluation setup combines a rebuilt catalog of 1,283 benchmark records with a daily discovery collection of over 11,000 observations to map the landscape. Benchmark records are classified into a multi-faceted taxonomy that separates capabilities from interaction paradigms, revealing that agentic evaluations are spread across coding and other domains rather than confined to a single class. The discovery stream, dominated by five sources, complements the catalog by tracking new releases and updates over time. A worked example demonstrates how the system lets a researcher or agent efficiently check prior work and avoid proposing duplicate evaluations.

The v0.11.0 benchmark catalog collects 1,283 records from four distinct sources. LLM Stats supplies the largest share (687 records) with benchmark names and provenance, while model reports contribute 110 entries, some without scores, alongside 461 from OpenCompass Hub and 25 from Artificial Analysis for commercial evaluations. LLM Stats dominates the catalog with 687 of 1,283 records, providing benchmark names, descriptions, and source provenance. Model reports add 110 records, including benchmarks mentioned even when no score is reported.

The platform offers six interfaces, each aligned with a specific reader question and a defined evidence scope. The Today interface surfaces daily discovery observations and source health, while Search provides access to the complete catalog, including unscored records. Leaderboard and Saturation serve different analytical needs: the former restricts evidence to scored records with distinct model or document counts, and the latter exposes full score histories and benchmark browsing across all sources and years. The Leaderboard only considers scored source records and counts distinct scored models or cited documents to answer which evaluations combine lower scores with broader measured use. The CLI offline client relies on the same benchmark IDs and a stable JSON response format, enabling local inspection of the evidence.

The benchmark catalog contains 1,283 records from four sources, 790 of which carry numeric scores totaling 12,916 observations. Three sources mix scored and unscored records, while OpenCompass Hub contributes 461 records with no scores at all. Artificial Analysis supplies the densest scoring per record, providing more than half of all numeric observations from a tiny fraction of the catalog. OpenCompass Hub accounts for 461 records but none of the 12,916 numeric scores. Artificial Analysis contributes only 25 records yet provides over half (7,050) of all numeric score observations.

A set of recent works on credit assignment in agentic training, all published in August 2026 and employing small Qwen-series base models, demonstrates a trend toward finer-grained reward signals beyond sparse terminal rewards. Techniques include reflection-conditioned dense token-level credit, action-level branching, selector credit partitioning, turn-level evidence reward, and hierarchical two-layer assignment, each evaluated on distinct agentic benchmarks. All surveyed works use Qwen base models from 3B to 14B parameters, with 8B models appearing most frequently. Credit assignment strategies shift from sparse terminal rewards to dense signals at the token, action, or turn level. The works cluster in a three-week period in late August 2026, indicating simultaneous and active investigation of the problem. Evaluation benchmarks cover long-context QA, web navigation, multi-hop reasoning, code generation, and multi-agent tasks.

The benchmark catalog evaluation aggregates 1,283 records from four heterogeneous sources, revealing that the majority of numeric scores come from a very small subset of records, while many entries lack scores entirely. Six platform interfaces serve distinct analytical needs, from daily discovery to full historical score browsing. A survey of contemporary credit assignment works in agentic training shows a clear shift from sparse terminal rewards to dense, fine-grained signals at the token, action, or turn level, with small Qwen-series models evaluated across varied agentic benchmarks in a compact August 2026 timeframe.