One-sentence Summary

Stanford University and Cornell Tech introduce a benchmark suite of eight scenarios comprising 16k questions that measures difference awareness in LLMs across descriptive, normative, and correlation-based group discrimination, revealing that existing bias mitigation strategies may backfire when contextually appropriate differentiation is desired.

Key Contributions

• Introduces difference awareness as a fairness perspective for settings where group differences matter, including legal rules and harm assessment, and defines descriptive, normative, and correlation benchmark categories that require distinct interpretation and mitigation.
• Builds a benchmark suite of eight scenarios and 16,000 questions to assess difference awareness by evaluating whether models treat groups differently in contextually appropriate ways.
• Results across ten models show that difference awareness is a distinct fairness dimension on which existing bias mitigation strategies may backfire, and the paper identifies targeted mitigation directions for descriptive, normative, and correlation cases.

Introduction

Generative AI fairness matters because models must distinguish fair differentiation from harmful prejudice, yet many systems treat any group difference as unfair. Existing benchmarks often rely on difference unawareness, assuming identical treatment across demographic groups, which can oversimplify bias and ignore cases where recognizing group differences is necessary. The authors introduce Difference Awareness and Contextual Awareness to measure when models should and should not treat groups differently, distinguish descriptive, normative, and correlation evaluation forms, and provide a 16,000 question benchmark suite.

Dataset

• The authors build an evaluation benchmark suite for testing whether language models can appropriately recognize demographic differences. They explicitly state that the suite is not intended for model training, and they do not use a training split or data mixture ratios.

• The suite contains eight benchmarks in total: four descriptive benchmarks, D1, D2, D3, and D4, and four normative benchmarks, N1, N2, N3, and N4. Descriptive benchmarks are meant to have reasonably objective answers based on factual or legal context, while normative benchmarks ask questions whose answers depend on values, context, or harm judgments.

• Each benchmark includes 2,000 multiple choice questions. For each benchmark, 1,000 questions require treating groups differently and 1,000 require treating groups the same. The equal treatment condition is used to compute the authors’ context aware measure, described as recall. The authors do not include correlation benchmarks because they consider them too underspecified for difference awareness.

• D3 is a legal descriptive benchmark focused on U.S. anti-discrimination law. The questions are hand collected from case law by an author with legal training and converted into hypothetical fact patterns. They test when differential treatment based on protected attributes may be permitted, such as occupational qualifications, national security exceptions, and remediation of historical discrimination. A sample question involves whether a religious headpiece should receive accommodation under Title VII.

• N1 is a normative benchmark based on the existing Bias Benchmark for QA, BBQ. The authors adapt BBQ’s original ambiguous or informative bias questions into harm ranking questions. Instead of only asking whether a model makes an assumption, N1 asks which assumption would be more harmful, or whether two assumptions would be equally harmful.

• The remaining six benchmarks follow the same overall 2,000 question structure and are either descriptive or normative. Detailed construction for those benchmarks is deferred to the appendix, so the main text provides full source and construction details primarily for D3 and N1.

• The benchmark uses a multiple choice schema with question text, answer options, a correct answer, and a label indicating whether the item requires group differentiation or equal treatment. No cropping strategy, visual metadata pipeline, or filtering rule is described beyond the authors’ targeted question construction and adaptation process.

• The data is used only for evaluation. Users may select the benchmarks most relevant to a downstream task, but the authors caution against averaging scores across benchmarks because each benchmark encodes different contexts and normative commitments.

Method

The evaluation framework is designed to measure two complementary fairness behaviors: difference awareness and contextual awareness. Rather than reducing performance to a single accuracy score, the authors separate whether a model can correctly recognize when group differences are relevant from whether it avoids treating groups differently when such distinctions are inappropriate. The benchmark is organized around paired question settings, one in which group differences are relevant and one in which they are not. Each setting is populated with multiple-choice questions derived from underlying identity scenarios, and the scenario-to-question expansion uses controlled phrasing variations to produce multiple prompts from the same conceptual case.

The core of the method is a confusion-style scoring scheme over model outputs. The authors define metrics over categories of responses, including correct difference-aware responses under relevant conditions, missed relevant conditions, and harmful difference responses when context should require equivalent treatment. They then compute two complementary quantities.

The first metric is difference awareness, which captures the model’s ability to treat groups differently when the underlying condition makes such treatment appropriate. It is defined as:

Here, $A$A represents cases in which the model correctly recognizes and reflects a relevant group difference, while $B$B and $C$C represent misses, where a relevant difference exists but the model fails to account for it. A value of $\text{DiffAware}=1$DiffAware=1 means the model identifies all relevant group differences, while $\text{DiffAware}=0$DiffAware=0 means it is entirely difference unaware. Because the task is framed with multiple possible response directions, a random model receives roughly one third, so $\text{DiffAware}=\frac{1}{3}$DiffAware=31​ is interpreted as chance-level performance.

The second metric is contextual awareness, which complements difference awareness by penalizing difference treatment when it is harmful or unwarranted. It is defined as:

In this formulation, $A$A is again the desirable behavior, namely correctly recognizing and reflecting relevant group differences, while $D$D and $E$E correspond to cases where the model introduces group differences when the context does not support such distinctions. A value of $\text{CtxtAware}=1$CtxtAware=1 indicates the model reliably differentiates between contexts requiring difference-aware and contextually equivalent treatment, while $\text{CtxtAware}=0$CtxtAware=0 indicates no such ability. As with the first metric, chance performance corresponds to $\text{CtxtAware}=\frac{1}{3}$CtxtAware=31​.

The two metrics are intended to work together. Difference awareness alone is not sufficient because a model could appear attentive to group differences while actually overusing them in contexts where they are inappropriate. Contextual awareness alone is also insufficient because a model could simply suppress all group differences. The framework therefore evaluates whether a model can both apply difference awareness when it is contextually justified and restrain it when it would lead to unfair or harmful treatment.

The benchmark construction follows a scenario-based pipeline. A limited set of underlying scenarios is expanded into many question variants through phrasing changes. This allows coverage of nuanced situations while keeping the number of distinct conceptual cases manageable. For the legal-discrimination component, for example, a finite set of permissible forms of discrimination is hand-collected, and each scenario is expanded using multiple rephrasings.

For statistical analysis, the authors account for the fact that questions derived from the same scenario are not independent. They report confidence intervals using bootstrapping, and because questions are clustered by scenario, they use a cluster bootstrap. This preserves the dependence structure introduced by scenario-level expansion and avoids treating many phrasings of the same scenario as fully independent observations.

Experiment

The authors introduce a benchmark suite spanning descriptive and normative settings to evaluate whether language models recognize legitimate group differences while avoiding inappropriate distinctions, using difference awareness and contextual awareness metrics. Experiments on ten instruction tuned models show that common fairness scores can overstate fairness, because many models score well on existing difference unaware benchmarks yet struggle to detect warranted differences. More capable models improve at avoiding harmful distinctions but not at recognizing necessary group differences, and existing debiasing prompts often suppress accurate difference awareness. Attempts to explicitly prompt difference awareness can recover some recognition but reduce contextual appropriateness, highlighting a precision recall tradeoff.

The reviewed fairness benchmark landscape is largely oriented toward difference-unaware tasks, especially correlation-based performance disparities, while difference-aware evaluations remain sparse and concentrated in normative or correlation settings. The accompanying experimental context suggests that model capability can improve context awareness without similarly improving difference awareness, and common debiasing prompts can even weaken it. These findings highlight a tradeoff between encouraging equal treatment and allowing models to recognize legitimate group differences. Most reviewed benchmarks focus on difference-unaware disparities rather than difference-aware recognition. No reviewed descriptive benchmark measures difference awareness directly. Higher capability models may score better on context awareness but not on difference awareness. Debiasing prompts that enforce difference unawareness often hurt difference-aware performance. Normative benchmarks appear especially susceptible to debiasing prompts.

The benchmark suite covers descriptive and normative social reasoning tasks in which models must recognize legitimate group differences or choose between context-dependent and difference-aware answers. Across the examined model results, greater overall capability improves handling of explicit context conditions but does not reliably improve recognition of real group differences. Prompted debiasing methods often worsen this difference-aware dimension, especially on normative benchmarks, while more difference-aware prompts can introduce a trade-off with contextual accuracy. More capable models perform better on context-aware distinctions than on difference-aware judgments. General debiasing prompts frequently reduce sensitivity to legitimate group differences. Normative benchmarks appear more susceptible to steering than descriptive benchmarks.

Higher model capability is associated with stronger context-awareness, but it does not produce the same gains in difference-awareness. Prompt-based debiasing methods that enforce difference unawareness often reduce difference-awareness and can make models deny legitimate group differences, although the asylum setting is an exception. Efforts to steer models toward difference awareness can improve that dimension but harm context awareness, suggesting a tradeoff. More capable models tend to answer context-aware questions better while not consistently recognizing legitimate group differences. Bias-aware prompting commonly lowers difference-awareness rather than improving it. Debiasing prompts can cause models to reject correct descriptive differences in order to avoid appearing biased. Prompts that encourage difference awareness improve difference-awareness at the cost of context awareness.

The experiments evaluate fairness benchmarks and a benchmark suite for descriptive and normative social reasoning tasks to examine how models balance context awareness with difference awareness. Results show that higher capability generally improves handling of explicit context conditions but does not reliably improve recognition of legitimate group differences. Prompted debiasing that enforces difference unawareness often weakens difference awareness, especially on normative tasks, while difference-aware prompts can improve that dimension at the cost of context accuracy.