One-sentence Summary

Researchers from Google Research and Technion propose a behavioral framework and the WikiProfile benchmark, showing that while GPT-5 and Gemini-3 encode 95–98% of facts, recall failures are the primary bottleneck for parametric factuality, and inference-time thinking can recover many errors, especially for long-tail facts and reverse questions.

Key Contributions

• A behavioral framework is proposed that profiles factual knowledge at the level of individual facts, distinguishing whether a fact is encoded and whether it can be recalled directly or only with inference-time computation.
• The WikiProfile benchmark is introduced, built via an automated pipeline using a prompted LLM grounded in web search to generate fact-level probes for measuring encoding and recall.
• Across 4 million responses from 13 LLMs, encoding is near-saturated (GPT-5 and Gemini-3 encode 95–98% of facts), while recall failures are systematic and concentrated on long-tail facts and reverse questions; inference-time computation (thinking) acts as a recall mechanism that recovers a substantial fraction of these failures.

Introduction

Large language models encode extensive factual knowledge but still produce errors, and standard accuracy metrics cannot tell whether a mistake stems from missing knowledge (an encoding failure) or an inability to retrieve what the model already knows (a recall failure). The authors address this by proposing knowledge profiling, a behavioral framework that separates encoding from recall, and introduce WIKIPROFILE, a benchmark of 2,150 naturally sourced facts each paired with probes for encoding, direct recall, and recognition. Evaluating 13 models, they find that frontier LLMs encode over 95% of facts yet fail to recall up to a third without inference-time computation (thinking), revealing recall as the primary bottleneck for factuality and showing that thinking recovers a large share of otherwise inaccessible knowledge.

Dataset

The authors introduce WIKIPROFILE, a benchmark for evaluating factual knowledge in large language models (LLMs). It consists of 2,150 facts extracted from Wikipedia, each paired with ten questions that probe encoding, knowledge, and fact verification. Here is a breakdown of the dataset’s construction, composition, and usage:

• Composition and question types

  • 2,150 facts, each equipped with ten questions.
  • 2 encoding questions: one proposition-completion item and one contextual question (formed by appending the direct question to its left context).
  • 4 knowledge questions: direct and reverse formulations, each instantiated with two distinct phrasings (one high-verbatim and one natural rephrasing).
  • 4 multiple-choice variants: used for fact verification, with three plausible distractors per question matched by entity type and thematic similarity.
  • All questions have a single, unambiguous gold answer, enabling automatic evaluation.

• Data sources and scale

  • Source: 10,000 Wikipedia pages, sampled with their visit counts and categorized into nine topics (figures in the paper show topic and entity-type distributions).
  • Entity selection: NER identifies candidate object entities (PERSON, LOCATION, DATE, etc.). Up to three candidate objects are chosen per document, excluding entities from the first sentences.
  • Strict criteria ensure each fact is non-trivial, non-guessable, time-independent, and resolves to a unique answer from its left context.
  • After downsampling to balance categories and entity types, 5,000 candidates are retained; the median left context length is 100 words, the mean is 116 words.

• Processing and filtering

  • A fully automated pipeline (with manual optimization on a held-out subset) handles extraction and question generation.
  • Question generation proceeds in three steps: generation, refinement, and filtering. Reverse questions are derived by swapping subject and object roles.
  • All questions pass through a Google Search-grounded LLM filter: if a query returns multiple answers or requires clarification, the entire fact is discarded.
  • After manual validation, fewer than 2% of facts are removed, yielding the final 2,150 facts.

• How the data is used in the model

  • WIKIPROFILE serves exclusively as an evaluation benchmark; it is not used for training.
  • It measures an LLM’s encoding (via completion and contextual priming) and factual knowledge (via direct and reverse probes), as well as its fact-verification capability (multiple-choice).
  • The pipeline is designed to be applicable to other corpora for domain-specific knowledge profiling.

Method

The authors introduce a knowledge profiling framework to systematically characterize factual behavior in LLMs. They define a fact as a proposition involving an ordered pair of entities: a subject and an object. To determine whether errors arise from encoding or recall, they introduce two operational notions defined purely by observable behavior.

Encoding is assessed using an encoding-via-memorization perspective, where an LLM encodes a fact if it can correctly reproduce it in a pre-training-like context. This is operationalized using tasks like proposition completion and contextual questioning, which strongly prime the model with the original source text. Knowledge, in contrast, is defined as the ability to correctly answer questions about a fact across a range of semantically equivalent contexts, testing invariance to surface form and relational direction. For each question $q$q, the authors generate multiple responses and use an LLM grader to compute the question grade as the accuracy over gradable responses:

$$g(q) = \frac{c_q}{c_q + i_q}$$g(q)=cq​+iq​cq​​

where $c_q$cq​ and $i_q$iq​ denote the number of correct and incorrect responses, respectively.

As shown in the figure below:

Based on these definitions, facts are characterized along three dimensions: whether they are encoded, known without thinking, and known only with thinking. This interaction yields five distinct knowledge profiles. Encoding Failure occurs when a fact is neither encoded nor known, indicating limitations in capacity or data coverage. Recall Failure describes facts that are encoded but inaccessible even with thinking, pointing to post-training issues. Direct Recall represents facts that are encoded and readily accessible. Recall with Thinking indicates facts that require additional inference-time computation to be retrieved. Finally, Inference without Encoding covers facts known via reasoning despite not being directly encoded.

The distribution of these profiles across models is illustrated in the figure below:

To support this profiling, the authors introduce the WIKIPROFILE benchmark, constructed via a fully automated pipeline. The process begins with fact extraction from Wikipedia documents. The pipeline performs named entity recognition and selects candidate object entities based on strict criteria, ensuring they represent non-trivial, time-independent completions of their left context.

The question generation module creates direct and reverse questions through generation, refinement, and filtering. Direct questions are initially high-verbatim to test encoding, then rephrased to test robustness. Reverse questions are derived by swapping subject and object roles. All questions undergo strict filtering grounded in Google Search to ensure unique answers; if any question for a fact is rejected, the entire fact is discarded. The pipeline also generates contextual questions by appending direct questions to the left context, and multiple-choice variants with plausible distractors.

The complete architecture of the benchmark creation pipeline is shown in the figure below:

Experiment

The evaluation introduces the WIKIPROFILE benchmark, which assesses both knowledge encoding and recall through a set of extracted Wikipedia facts and varied question formats. Across evaluations of multiple large language models, the central finding is that factual errors primarily stem from a recall bottleneck rather than missing knowledge; models have encoded the information but often fail to access it, especially for rare facts or when query direction reverses the training order. The analysis shows that thinking mechanisms can partially recover this inaccessible knowledge, narrowing performance gaps and suggesting that the challenge has shifted from acquisition to retrieval. These patterns mirror human memory phenomena like the tip-of-the-tongue state, indicating a functional parallel between artificial and human recall systems.

Frontier language models encode nearly all facts, with encoding failures falling to low levels, while smaller models still miss a notable share. Even the largest models fail to directly recall many encoded facts, leaving a substantial fraction that require thinking or remain inaccessible. Thinking serves as an effective recovery mechanism, converting most otherwise lost facts into correct answers, though a small residual of lost facts persists. Scaling reduces encoding failures: frontier models encode over 85% of facts, compared to under 80% for smaller models. Direct recall remains limited even at scale: in frontier models, only about two-thirds of facts are known directly, with the rest requiring thinking or remaining lost. Thinking recovers most encoded-but-not-directly-known facts in frontier models, slashing the lost category to a few percent. Smaller models rely more on verification: GPT-5 mini knows over a quarter of facts only in multiple-choice settings, a pattern nearly absent in frontier models.

The experiments evaluate how language models encode and retrieve factual knowledge, comparing direct recall with reasoning-augmented generation. Frontier models encode almost all facts, yet direct recall remains limited, leaving many facts accessible only through thinking, which recovers most of them and reduces the lost fraction to a few percent. Smaller models show more encoding failures and depend on verification, with a notable share of facts known only in multiple-choice settings, a pattern that nearly disappears at scale.