One-sentence Summary

Meta AI introduces GAMUT, a benchmark for factual completeness in long-form generation that employs a two-level meta-rubric framework—a structured meta-rubric capturing content organization and importance, mechanically compiled into binary, machine-gradable checklists—to evaluate open-ended responses, proving challenging for 14 frontier models (best score 58.7% by Gemini 3.1 Pro) across 1,813 multimodal and text-only questions.

Key Contributions

• The paper introduces a two-level meta-rubric framework that separates the structured representation of a complete answer (open-ended sets, ordered processes, relationships, importance tiers) from a mechanically compiled flat binary checklist, resolving the trade-off between expressiveness and reliable automated grading.
• The framework is instantiated as Gamut, a benchmark for factual completeness with 1,813 multimodal questions grounded in real wearable imagery across 10 domains, each paired with an evidence-backed rubric verified by expert annotators; evaluation shows the best model, Gemini 3.1 Pro, achieves only 58.7%.
• The work releases a modality-agnostic text-only variant, and empirical analysis demonstrates that the benchmark discriminatively separates models with stable rankings across different LLM judges, and that performance differences largely reflect knowledge completeness rather than perception alone.

Introduction

Long-form generation evaluation has largely centered on precision, with decompose-search-verify pipelines that check each claim independently, while factual recall and completeness receive far less attention. Both approaches share a deeper limitation: they reduce factuality to a flat list of independent boolean fact-checks, which cannot represent open-ended sets of acceptable answers, ordered processes, or importance-weighted groupings. The authors introduce Gamut, a benchmark and two-level rubric framework that resolves this tension. A structured meta-rubric captures the organization, ordering, and importance of required content, and is then mechanically compiled into a flat, binary checklist so that grading remains reliable and low-variance. Gamut instantiates this design on everyday deep-research questions grounded in real wearable imagery, demonstrating that factual completeness remains a major unsolved challenge for current models.

Dataset

The authors construct Gamut, a benchmark for evaluating everyday deep research questions grounded in wearable imagery. The dataset is used as an evaluation set, not for training, and centers on a fine-grained scoring protocol that measures answer completeness.

• Dataset composition and sources

  • 1,813 questions, each paired with one egocentric image from the CRAG-MM benchmark.
  • Images are real-world photographs taken with smart glasses, where the object of interest is often small, rotated, occluded, or poorly lit.
  • Entities span 10 everyday domains: plants, food, animals, vehicles, local places, everyday objects, consumer products, and more.
  • Every question is accompanied by a human-verified binary rubric (average 15.2 checks) and a structured meta-rubric that captures lists, processes, and relationships.

• Question creation process

  • A frontier multimodal LLM proposes candidate open-ended questions that require multi-step research, refer to the image entity only through generic terms, and pass a “stranger test” (cannot presuppose entity identity).
  • Expert annotators review, revise, or discard candidates over multiple rounds. A second LLM pass rejects leading questions, selects the most research-intensive and diverse candidate, and seeds new diverse suggestions when needed.
  • The pipeline yields one accepted question per image, with 1,843 questions before final rubric cleanup.

• Rubric creation and refinement

  • An LLM drafts a structured meta-rubric from web-sourced evidence and compiles it into a binary rubric.
  • A self-refinement stage re-verifies all snippets and retrieves missing evidence, then rebuilds the rubric.
  • A small in-house team revises the binary rubric directly, using the meta-rubric as a scaffold, over two human rounds.
  • Upstream errors (wrong entity, ambiguous questions) are hand-corrected in 12 cases, and 30 questions are dropped when no sound rubric could be produced, leaving 1,813 final examples.

• Rubric statistics and structure

  • Rubrics average 15.2 binary checks: ~5.9 Answer-Critical, ~5.8 Valuable, ~3.5 Context checks.
  • 98% of questions include at least one structured component beyond isolated facts—flexible lists (86%), strict lists (49%), ordered processes (17%), and explicit relationships (6%).
  • Flexible lists produce threshold checks; processes produce sequence checks that grade ordering.
  • The meta-rubric enables the benchmark to express open-ended coverage requirements and partial-credit logic that a flat checklist cannot.

• How the data is used

  • The benchmark is used to evaluate model-generated answers. Each answer is scored against the binary rubric, producing a Gamut score that measures completeness at a fine grain.
  • No training split is reported; the dataset serves exclusively as an evaluation benchmark for deep research capabilities grounded in real-world images.

Method

The authors propose a two-level evaluation framework that separates the expressive description of a good answer from the reliable, low-variance checks a judge can apply. The upper level is a structured meta-rubric that captures coverage, ordering, and importance; the lower level is a flat binary rubric derived from it by deterministic rules. This design keeps the rich structure where answers are described and the mechanical, auditable checks where scoring happens.

Meta-Rubric Structure. The meta-rubric models answer quality as a set of items, each belonging to one of three importance tiers: Answer-Critical (must be present), Valuable (meaningfully improves the answer), or Context (helpful background). Items carry a type that defines their internal structure:

• Simple Knowledge: a single discrete fact.
• Strict List: a finite set where every element is required.
• Flexible List: a pool of valid options for which sufficient coverage, not a specific choice, is required; a baseline threshold specifies the minimum number of items that must be mentioned.
• Process: an ordered sequence of steps where order matters, with both required and optional steps.
• Relationship: a connection or contrast between entities.

Beyond individual items, a list or process may carry a meta-insight—a synthesis across items, such as an overarching trend, that no single fact captures. Every item can be grounded in cited web snippets, making the rubric evidence-based rather than reliant on parametric memory.

Conversion to Binary Rubrics. To avoid the holistic judgment that a complex rubric would reintroduce, the meta-rubric is compiled into a flat list of pass-or-fail checks by fixed, mechanical rules. Simple knowledge and strict-list items become straightforward checks. Flexible lists produce two kinds of check: a baseline coverage check (e.g., “mention at least N of the following: …”) and additional credit either as per-item checks for small pools or as higher-tier coverage thresholds for long pools. Processes generate presence checks for each step plus sequence checks that verify ordering of the required steps and, optionally, the full order. Relationships and meta-insights each become one or a few checks. Across all types, checks that constitute the genuine requirement keep their parent item’s importance, while secondary checks (optional items, higher coverage thresholds, optional steps, and full ordering) drop one level. The result is a binary rubric whose checks are still individually gradable but can also ask whether enough of an open set is covered and whether facts appear in the right order.

Scoring. Given a response and its binary rubric, scoring proceeds in three steps. First, an LLM judge evaluates each check independently, returning one of four verdicts: meets, partially meets, missing, or contradicts. Distinguishing missing from contradiction is essential for factual reliability; a contradiction is penalized more heavily. The per-tier score is computed as

$$s = \frac{M + \lambda P + \mu C}{M + P + S + |\mu| C},$$s=M+P+S+∣μ∣CM+λP+μC​,

where $M$M, $P$P, $S$S, and $C$C are the counts of meets, partially meets, missing, and contradicts verdicts, with $\lambda = 0.5$λ=0.5 and $\mu = -2$μ=−2. Because $\mu$μ enters both numerator and denominator, a contradiction both removes credit and amplifies the penalty, so a cautious but incomplete answer outranks a confident but wrong one. Finally, the overall Gamut score is a weighted average of the three tier scores, using global weights $w_{\text{AC}} = 0.6$wAC​=0.6, $w_{\text{V}} = 0.3$wV​=0.3, and $w_{\text{C}} = 0.1$wC​=0.1. These weights are fixed across questions, preventing a question with many minor Context checks from drowning out its few Answer-Critical ones. When a tier has no checks, its weight is redistributed to the remaining tiers.

Dataset Construction Pipeline. The framework is applied to build the Gamut benchmark through a two-stage process that combines LLM generation with human verification. First, questions are created from egocentric imagery: a frontier multimodal LLM proposes open-ended questions that require multi-step research and refer to the subject only through a pronoun or generic term (e.g., “this car”). Expert annotators review, revise, and discard candidates over multiple rounds, and an automatic pass further filters leading questions and diversifies the set. For each accepted question, a structured meta-rubric is generated by prompting the LLM to gather web evidence as cited snippets, organize the content into the five item types, and compile the binary rubric via the deterministic rules. The rubric undergoes an LLM self-refinement phase that audits snippet grounding and re-researches missing evidence, after which human annotators revise the binary rubric directly, using the meta-rubric as a scaffold to spot structural errors. The cycle repeats until every question has a human-verified rubric, yielding a dataset whose evaluation logic is fully auditable and grounded in the structured representation.

Experiment

The evaluation uses the Gamut benchmark, scoring models with an LLM judge (Gemini 3.1 Pro) and verifying robustness through two additional judges who reproduce the same ranking. The main results show that Gamut is challenging and highly discriminative, correctly ordering models by known quality factors, with omission rather than contradiction being the dominant failure mode and completeness of answers largely separating stronger from weaker models. A text-only variant reveals that removing the image yields a uniform score increase across all models, indicating that visual identification is a near-constant tax rather than a key differentiator, and the multimodal ranking remains stable.

The best model achieves a Gamut score of 58.7% and meets just over half of rubric elements, showing the benchmark is far from saturated. Rankings align with known model properties—larger, newer, proprietary models outperform smaller, older, open-weight ones—and the dominant failure mode is omission, with missing elements persisting even for top performers. The strongest model meets only 55% of rubric elements, leaving substantial headroom for improvement. Missing verdicts are the largest error category, making up over a quarter of elements for the top model and rising sharply for weaker ones, so completeness drives the score differences.

Three different LLM judges scored all 14 models on the same set of questions. Gemini 3.1 Pro and Claude Opus 4.8 produced identical rankings with scores within 1.8 points of each other and no self-preference, while the smaller Qwen3-VL 235B judge was consistently more lenient but maintained the same broad ordering, with only minor adjacent swaps. This confirms the robustness of the Gamut score rankings to the choice of judge. Gemini 3.1 Pro and Claude Opus 4.8 yield identical model rankings, with scores differing by at most 1.8 points. Qwen3-VL 235B, a weaker open-weight judge, is uniformly more lenient by 4 to 11 points but preserves the same broad ordering, with only adjacent swaps among closely spaced models.

Removing the image from questions yields a uniform score increase across all models, with most gaining between 10 and 20 points. The relative ranking of models remains broadly unchanged, indicating that visual identification acts as a consistent difficulty tax rather than a differentiator of model quality. The text-only gap reflects genuine answer incompleteness separate from perception. Every model scores higher on the text-only variant, with gains falling mostly in the 10–20 point range. The model ranking is largely preserved when images are removed, suggesting the multimodal ordering captures underlying answer quality. The improvement is nearly independent of the original multimodal score, making visual identification a roughly constant challenge across models. Even the strongest model gains as much as most others, so its remaining gap in the text-only setting stems from incomplete reasoning rather than visual errors.

On a multimodal chemistry benchmark with rubric-based evaluation, the best model reaches just 58.7% Gamut, with omission of required elements as the dominant failure mode and substantial headroom for improvement. Rankings are robust to the choice of LLM judge, and removing images reveals that visual identification acts as a constant difficulty tax, while remaining gaps stem from incomplete reasoning rather than perception.