One-sentence Summary

The Nilekani Centre at AI4Bharat and IIT Madras introduce FBI, a framework that introduces 2400 targeted perturbations across 22 categories to stress-test Evaluator LLMs on factual accuracy, instruction following, coherence, and reasoning, revealing that current evaluators fail to detect quality drops in over 50% of cases on average.

Key Contributions

• The paper introduces FBI, a human-crafted meta-evaluation framework that perturbs candidate answers to test evaluator LLMs on factual accuracy, instruction following, long-form writing, and reasoning. It includes 2,400 perturbed answers across 22 categories for fine-grained diagnosis of evaluator weaknesses.

• The work compares five prominent evaluator LLMs across single-answer scoring, pairwise comparison, and reference-based scoring to identify evaluation protocols that expose reliability gaps.

• Experiments show that current evaluator LLMs fail to identify quality drops in over 50% of cases on average, while reference-based evaluation performs better than single-answer and pairwise evaluation. These findings indicate that current LLM judges remain unreliable for many practical assessment settings.

Introduction

The authors examine LLM-as-evaluator systems, which are increasingly used to score model outputs and maintain AI leaderboards because they are cheaper and faster than human evaluation. Prior work often reports correlations with human judgments, but it may miss fine-grained weaknesses such as failure to detect factual errors, broken instructions, incoherent writing, or flawed reasoning. To address this, the authors introduce FBI, a framework that uses interpretable checklists and targeted answer perturbations across factual accuracy, instruction following, long-form coherence, and reasoning. Using three evaluation settings, they show that current Evaluator LLMs have major blind spots and often fail to adjust scores when answer quality drops.

Dataset

• The authors introduce FBI, a meta-evaluation dataset for testing evaluator LLMs. Each example has the schema (I, A_gold, A_perturb), where I is the instruction, A_gold is the correct answer, and A_perturb is a deliberately perturbed answer.

• The benchmark covers four task abilities: factual accuracy, reasoning, instruction following, and long-form writing. Prompts are manually labeled into these categories.

• The initial prompt pool comes from six English test sets: WizardLM, MT Bench, UltraChat, LIMA, LLMBar, and IFEval. These contain 1809 prompts focused on long-form generation, open-ended tasks, creativity, and instruction following.

• From this pool, the authors sample 100 questions for each ability. Reasoning prompts are supplemented with GSM8k and MATH, and 200 additional instruction-following prompts are created for specific perturbation categories.

• Gold answers are generated with GPT-4-Turbo. The authors manually check a 25% sample from each category for correctness, though exact answer quality is less central than whether perturbed answers score lower than clean answers.

• Perturbed answers are generated by prompting GPT-4-Turbo with category-specific instructions and asking it to explain each perturbation. Instructions are iteratively refined after manual review of 25% of perturbations per category.

• All generated perturbations undergo manual vetting. Annotators compare each perturbed answer with the gold answer and mark it valid, invalid, or score invariant. A perturbation is kept as valid only if it should logically cause a scoring penalty.

• Score-invariant perturbations are modifications that should not reduce the score. The dataset contains 516 such examples, collected from human labeling and from GEMINI-1.5-PRO paraphrases that preserve the original facts and details.

• The data is used as an evaluation benchmark, not a training corpus. Evaluator LLMs receive the instruction, gold answer, and perturbed answer, and the paper examines whether their scores correctly reflect the presence of clear errors.

• No cropping strategy is reported. The main processing steps are prompt categorization, LLM-based answer and perturbation generation, manual verification, perturbation validity filtering, and construction of a score-invariant negative set.

Method

The proposed method introduces FBI, a meta-evaluation framework for auditing the ability of LLM evaluators to detect errors in generated answers. Rather than testing the correctness of the evaluator model itself, the authors leverage this framework to examine whether an evaluator LLM can assign consistently lower judgments to answers that contain targeted flaws. The design centers on paired answer comparisons: for each instruction, a gold answer is provided together with a perturbed answer that introduces a specific error type. The evaluator is then prompted to score, compare, or otherwise judge these answers under controlled conditions.

At the core of the framework, each instance is represented as a tuple $(I, A_{gold}, A_{perturb})$(I,Agold​,Aperturb​), where $I$I is the instruction given to the model, $A_{gold}$Agold​ is the reference answer, and $A_{perturb}$Aperturb​ is a modified version of the reference answer. The framework is organized around four task abilities: factual accuracy, reasoning ability, instruction following, and long-form writing proficiency. The goal is not necessarily to certify absolute score values, but to measure directional score changes, that is, whether $A_{perturb}$Aperturb​ receives a lower evaluation than $A_{gold}$Agold​ when a clear error has been introduced.

The benchmark construction pipeline begins with prompt selection. The authors leverage prompts drawn from instruction datasets covering long-form generation, open-ended creativity, reasoning, and explicit instruction following. Prompts are manually categorized into four ability types: long-form writing, factual question answering, instruction following, and reasoning. Gold answers are generated using GPT-4-TURBO and then partially verified by human reviewers to ensure high reliability. The authors emphasize that the absolute quality of the gold answers is less important than the consistency of the error introduction process, since the main evaluation signal comes from comparing each perturbed answer against its corresponding gold answer.

A key module of the method is the perturbation taxonomy. The authors define error categories across the four task abilities, grounded in common failure modes of current LLMs. These include factual inaccuracies, flawed reasoning, omitted or violated constraints, degraded long-form coherence, and other answer-level defects. In addition to error-inducing perturbations, the framework also includes score-invariant perturbations, which are modifications that should not reduce the score of a response if they preserve all essential content. This dual design allows the benchmark to test not only whether evaluators penalize defective answers, but also whether they unnecessarily penalize harmless paraphrases.

Perturbed answers are generated through a controlled prompting workflow using GPT-4-TURBO. For each perturbation category, the model is given instructions specifying the desired error type and is asked both to produce a perturbed answer and to explain why the modification constitutes an error. The instructions are iteratively refined by manually reviewing generated samples. This creates a feedback loop in which the model’s perturbation behavior is monitored, corrected, and improved until the generated edits reliably match the intended category.

Because automated generation can fail in subtle ways, the authors incorporate a human-in-the-loop validation module. In this stage, each generated perturbed answer is compared against the corresponding gold answer. Human reviewers classify the perturbation as valid, invalid, or score invariant. A valid perturbation must logically justify a scoring penalty, meaning that a reasonable evaluator should detect the introduced defect. The reviewers are trained in LLM failure modes and use a dedicated review tool to maintain consistency. This step is central to the methodological design, ensuring that the benchmark tests evaluator capability rather than artifacts from imperfect perturbation generation.

Once the audit set is constructed, the framework defines several evaluator strategies. These strategies are not separate trained models, but different prompting protocols through which an evaluator LLM is queried. The authors group them into three paradigms: single-answer scoring, pairwise comparison, and reference-guided scoring. Each paradigm exposes the evaluator to a different amount of context and therefore tests different aspects of meta-evaluation competence.

In the single-answer scoring paradigm, the evaluator must assess a response without direct access to a corrected reference. The simplest variant presents only the instruction and the model response, requiring the evaluator to output a score. Formally, this can be written as

$$f(P_{eval}, I, A_{model})(score)$$f(Peval​,I,Amodel​)(score)

where $f$f denotes the evaluator LLM, $P_{eval}$Peval​ is the evaluation prompt, $I$I is the instruction, and $A_{model}$Amodel​ is the candidate answer. A stricter variant asks the evaluator to provide both an explanation and a score:

$$f(P_{eval}, I, A_{model})(exp, score)$$f(Peval​,I,Amodel​)(exp,score)

More guided variants enrich the prompt with additional evaluation scaffolding. In the rubric strategy, the evaluator receives a grading rubric $R$R together with the instruction and response, and is prompted to generate an explanation followed by a score:

$$f(P_{eval}, R, I, A_{model})(exp, score)$$f(Peval​,R,I,Amodel​)(exp,score)

In the axis strategy, the evaluator is directed to assess the answer along a specific evaluation axis $Ax$Ax, such as factual correctness, reasoning quality, instruction compliance, or long-form coherence. This is expressed as

$$f(P_{eval}, Ax, I, A_{model})(exp, score)$$f(Peval​,Ax,I,Amodel​)(exp,score)

The authors further combine these cues into a richer protocol, where the evaluator is given both an axis and a rubric:

$$f(P_{eval}, Ax, R, I, A_{model})(exp, score)$$f(Peval​,Ax,R,I,Amodel​)(exp,score)

This progression from minimal to fully scaffolded prompting allows the method to test whether evaluators improve when provided with explicit evaluation criteria.

In the pairwise comparison paradigm, the evaluator receives two candidate responses and must decide which one is better, or whether they are equivalent. In the simplest form, the evaluator sees only the instruction and two responses $A_1$A1​ and $A_2$A2​:

$$f(P_{eval}, I, A_1, A_2)(verdict)$$f(Peval​,I,A1​,A2​)(verdict)

A more interpretable variant also requires an explanation:

$$f(P_{eval}, I, A_1, A_2)(exp, verdict)$$f(Peval​,I,A1​,A2​)(exp,verdict)

The method further introduces rule-based and axis-based variants. In the rules strategy, the evaluator receives detailed comparison rules $R$R:

$$f(P_{eval}, R, I, A_1, A_2)(exp, verdict)$$f(Peval​,R,I,A1​,A2​)(exp,verdict)

In the axis variant, the evaluator compares the two answers along a specific axis $Ax$Ax:

$$f(P_{eval}, Ax, I, A_1, A_2)(exp, verdict)$$f(Peval​,Ax,I,A1​,A2​)(exp,verdict)

A combined strategy supplies both an axis and detailed rules:

$$f(P_{eval}, Ax, R, I, A_1, A_2)(exp, verdict)$$f(Peval​,Ax,R,I,A1​,A2​)(exp,verdict)

The authors structure these variants to isolate whether evaluation prompts, rubrics, and explicit axes improve the evaluator’s ability to notice errors.

The final paradigm is reference-guided single-answer scoring. Here, the evaluator is given the instruction, the candidate answer, and a gold reference answer, and must judge the candidate against the reference:

$$f(P_{eval}, I, A_{gold}, A_{model})(exp, score)$$f(Peval​,I,Agold​,Amodel​)(exp,score)

This strategy is more constrained than the unassisted variants and is especially relevant for tasks where a reliable reference can be constructed. By comparing performance across all these protocols, the method evaluates whether an evaluator model benefits from external guidance, whether it can reason about errors explicitly, and whether its judgments remain consistent when the same underlying failure is tested through different evaluation interfaces.

Overall, the authors leverage a tightly coupled pipeline of curated instructions, controlled perturbations, human validation, and structured evaluator prompting to create a diagnostic system for evaluator LLMs. The framework treats evaluation as a measurable, testable capability, with explicit error categories, paired reference and perturbed answers, and multiple prompting regimes designed to reveal where evaluator models succeed or fail.

Experiment

The FBI benchmark evaluates Evaluator LLMs across factual, reasoning, instruction-following, and long-form writing by testing single-answer scoring, pairwise comparison, and reference-guided evaluation on gold answers and human-vetted perturbations, with score-invariant perturbations as controls. These experiments validate whether evaluators penalize clear errors, prefer gold answers, avoid penalizing harmless changes, benefit from explanations, and improve with broader scoring ranges, but show that current models often fail to detect perturbations, especially outside reasoning. GPT-4-Turbo is stronger than other tested models in reference-free settings yet remains unreliable, while reference-based evaluation can overpenalize correct answers and wider scoring ranges provide only slight improvement.

The presented statistics describe human-created and manually vetted perturbations used to test evaluator LLMs across four task abilities. The shown rows break down long-form writing perturbations into categories such as grammar, spelling, consistency, chronology, coherence, and comprehensiveness. Among these displayed long-form categories, grammar and spelling appear more represented than chronology. The benchmark evaluates evaluator LLMs by testing whether they can detect controlled answer errors across task abilities. Long-form writing perturbations are divided into multiple human-defined categories covering language form and discourse quality. Among the listed long-form perturbation categories, grammar and spelling are more prominent while chronology is less represented.

The perturbation catalog organizes checks for evaluator LLMs around specific text generation abilities, including grammar, spelling, consistency, chronology, coherence, comprehensiveness, and contextual factual errors. The categories are designed to isolate distinct failure modes, so evaluator scores should change only when a perturbation affects the targeted ability, while score-invariant changes should leave scores consistent. Human-curated examples and vetting make the perturbations interpretable for diagnosing blind spots in model-based evaluation. Grammar and spelling perturbations test whether evaluators notice surface language quality. Coherence, chronology, consistency, and comprehensiveness perturbations target long-form writing ability. Contextual fact replacement tests factual accuracy by substituting incorrect but similar facts. Score-invariant perturbations provide a control for evaluators that should preserve scores.

GPT-4-Turbo is a weak evaluator for perturbation sensitivity, since it often keeps the same score or verdict after perturbations except on reasoning tasks. For single answer scoring, simpler evaluation prompts are more reliable at penalizing error-inducing changes, while rubric and axis prompts trade that sensitivity for more stability on score-invariant changes. For pairwise comparisons, more detailed rules help, and adding a reference answer improves detection markedly, though many failures remain. Reasoning tasks are the clearest exception, with more reliable detection of perturbations. Explicit rubrics and evaluation axes can add thoroughness without improving sensitivity to subtle errors in single answer scoring. Pairwise evaluation benefits more than single answer scoring from detailed evaluation rules. Reference-based evaluation is stronger than reference-free evaluation but still misses many perturbed answers.

Across the best observed evaluation strategies, GPT-4-TURBO is the strongest general evaluator in reference-less paradigms, while LLAMA-3-70B-INSTRUCT appears strongest in reference-based scoring. However, LLAMA-3-70B-INSTRUCT is highly stringent and often does not assign perfect scores, which may indicate strong reference dependence rather than reliable robustness. Claude-3-OPUS and a trained evaluator perform worse than several general-purpose models, and score-invariant perturbations reveal remaining weaknesses. GPT-4-TURBO consistently leads in reference-less evaluation. LLAMA-3-70B-INSTRUCT excels with references but is unusually strict and may over rely on them. Score-invariant perturbations show that many models still penalize acceptable responses.

Extending the evaluator score range from 1-3 to 1-5 generally reduces the percentage of perturbed answers whose scores remain unaffected, indicating slightly better sensitivity to perturbations. Axis+Rubrics shows the clearest gains in LF, F, and IF, while Rubrics also improves, except for a marginal decline in R. A wider 1-5 range usually helps both Rubrics and Axis+Rubrics reflect perturbations more often than the narrower 1-3 range. Axis+Rubrics achieves the largest improvements in LF, while Rubrics remains slightly weaker on R with the wider range.

The experiments evaluate evaluator LLMs using human-curated, manually vetted perturbations across writing, factual, and reasoning-style abilities, with controlled error-inducing changes and score-invariant controls. The results show that evaluator sensitivity is highly task- and strategy-dependent: simpler prompts and wider scoring ranges improve error detection for single-answer scoring, while detailed rules and references help pairwise evaluation but still miss many perturbations. GPT-4-Turbo performs best in reference-free settings but is often insensitive to perturbations outside reasoning, whereas Llama-3-70B-Instruct performs better with references yet may be overstrict, and score-invariant tests reveal persistent weaknesses in current evaluators.